Electronic map updating method, computer device and storage medium

Abstract

The invention discloses an electronic map updating method, a computer device and a storage medium, which can avoid comprehensive semantic division of an original point cloud data set covering the whole scene and can significantly reduce the calculation overhead. Good real-time performance can be maintained when the electronic map is updated in scenes such as complex and dynamically changing construction sites; by performing semantic annotation and optimization of the region of interest on the original point cloud data set, the obtained priori semantic map carries accurate semantic annotation information, so that when updating is performed according to the real-time image, the region possibly needing to be updated in the priori semantic map can be quickly determined according to the semantic information of the real-time image, and the updating efficiency is improved. According to the method, the probability that global search needs to be carried out on a priori semantic map is reduced, the calculation overhead can also be remarkably reduced, the electronic map updating speed and real-time performance are improved, and the navigation efficiency and use safety of an execution subject in a scene are improved. The method is widely applied to the technical field of electronic maps.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic maps, and in particular to an electronic map updating method, a computer device and a storage medium. Background Art

[0002] SLAM (Simultaneous Localization And Mapping) is a technology that can perform positioning and map construction at the same time. SLAM can realize movement from an unknown position in an unknown environment, and locate itself according to the position offset and map data during the movement. At the same time, it builds an incremental map based on its own positioning, and realizes autonomous positioning and navigation of autonomous driving vehicles, which can be applied to construction site inspection, autonomous driving, scene mapping and virtual reality.

[0003] The key to SLAM technology is to build a map of the surrounding area while relying on moving objects such as autonomous vehicles equipped with sensors. However, the current implementation of SLAM technology requires a lot of computing resources, and the real-time performance of the system is often affected. Especially when processing high-density point cloud data, there are still significant challenges in ensuring real-time performance and accuracy. Summary of the invention

[0004] In view of the technical problems such as the poor real-time performance of the current SLAM technology, the object of the present invention is to provide an electronic map updating method, a computer device and a storage medium.

[0005] In one aspect, an embodiment of the present invention includes a method for updating an electronic map, the method comprising the following steps:

[0006] In a first time period, an original point cloud data set and image information of the scene are obtained;

[0007] According to the image information, semantically annotating and optimizing the region of interest of the original point cloud data set to obtain a priori semantic map;

[0008] Acquire a real-time image of the scene within a second time period; the second time period is a time period after the first time period;

[0009] The priori semantic map is updated according to the real-time image.

[0010] Furthermore, the semantic annotation and optimization of the region of interest of the original point cloud data set are performed according to the image information to obtain a priori semantic map, including:

[0011] Performing target detection on the original point cloud data set to determine at least one region of interest;

[0012] The point cloud subset within the region of interest is semantically annotated, so as to determine a corresponding local semantic space in the original point cloud data set; the local semantic space includes the point cloud subset within the region of interest and a semantic label obtained by the semantic annotation.

[0013] Furthermore, the semantic annotation and optimization of the region of interest of the original point cloud data set are performed according to the image information to obtain a priori semantic map, and further includes:

[0014] According to the semantic category, the local semantic space with the corresponding semantic label is cut out from the original point cloud data set to obtain the priori semantic map.

[0015] Further, updating the prior semantic map according to the real-time image includes:

[0016] registering the prior semantic map;

[0017] Determine the target semantics;

[0018] According to the target semantics, detecting the real-time image and the registered prior semantic map;

[0019] When no portion corresponding to the target semantics is detected from the real-time image, and no point cloud corresponding to the target semantics is detected from the priori semantic map, keeping the priori semantic map unchanged;

[0020] When a portion corresponding to the target semantics is not detected from the real-time image, and a point cloud corresponding to the target semantics is detected from the priori semantic map, the point cloud detected in the priori semantic map is deleted.

[0021] Furthermore, the updating of the prior semantic map according to the real-time point cloud further includes:

[0022] In the second time period, a real-time point cloud of the scene is obtained;

[0023] According to the target semantics, detecting the real-time point cloud and the prior semantic map;

[0024] When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is not detected from the prior semantic map, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame;

[0025] When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is detected from the prior semantic map, a position change between the point cloud detected from the real-time point cloud and the point cloud detected from the prior semantic map is detected; when the position change is detected, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame; when the position change is not detected, the prior semantic map is kept unchanged.

[0026] Furthermore, the detecting of the real-time point cloud and the prior semantic map according to the target semantics includes:

[0027] Performing three-dimensional target detection on the prior semantic map according to the target semantics to determine a directed bounding box;

[0028] Performing target detection on the real-time image according to the target semantics to determine the target area;

[0029] Projecting the target area onto the real-time point cloud;

[0030] Clustering the projected points in the real-time point cloud to obtain at least one point cloud cluster;

[0031] Screening each of the point cloud clusters;

[0032] The screened point cloud cluster is used as the point cloud detected from the real-time point cloud.

[0033] Furthermore, the screening of each of the point cloud clusters includes:

[0034] Respectively obtain the confidence of each of the point cloud clusters;

[0035] The point cloud clusters whose corresponding confidence levels are higher than a confidence threshold are screened out.

[0036] Furthermore, the obtaining of the confidence of each of the point cloud clusters respectively includes:

[0037] For any of the point cloud clusters, according to the formula

[0038]

[0039] Calculate, where C represents the confidence of the point cloud cluster, N points is the number of points in the point cloud cluster, d mean is the average distance between the point cloud cluster and the center of the first detection component, V is the volume of the point cloud cluster, and α and β are weight parameters.

[0040] On the other hand, an embodiment of the present invention further includes a computer device, including a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load at least one program to execute the electronic map updating method in the embodiment.

[0041] On the other hand, an embodiment of the present invention further includes a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the electronic map updating method in the embodiment.

[0042] The beneficial effects of the present invention are as follows: the electronic map updating method in the embodiment, wherein the prior semantic map is obtained by semantically annotating and optimizing the area of ​​interest of the original point cloud data set, can avoid comprehensive semantic division of the original point cloud data set covering the entire scene, thereby significantly reducing the computational overhead, and is conducive to maintaining good real-time performance when performing electronic map updates in complex and dynamically changing scenes such as construction sites; by semantically annotating and optimizing the area of ​​interest of the original point cloud data set, the obtained prior semantic map has accurate semantic annotation information, and can quickly determine the area in the prior semantic map that may need to be updated according to the semantic information of the real-time image when updating according to the real-time image, thereby reducing the possibility of needing to perform a global search of the prior semantic map, and can also significantly reduce the computational overhead, improve the speed and real-time performance of electronic map updates, and improve the navigation efficiency and use safety of execution entities such as robots or cars in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the steps of the electronic map updating method in the embodiment;

[0044] Figure 2 is a schematic diagram of a system to which the electronic map updating method can be applied in an embodiment;

[0045] Figure 3 Schematic diagram of the principle of the electronic map updating method in the embodiment. DETAILED DESCRIPTION

[0046] Terminology explanation:

[0047] 1. Laser SLAM (Simultaneous Localization and Mapping)

[0048] Definition: Laser SLAM can be divided into 2D-SLAM and 3D-SLAM according to the dimension. LiDAR can provide high-precision distance data, making the positioning and mapping of laser SLAM in complex environments more accurate and stable.

[0049] 2. Dynamic SLAM

[0050] Definition: Dynamic SLAM refers to a technology that enables robots to perceive, identify and track dynamic objects (such as pedestrians, vehicles, etc.) in real time in an environment where there are dynamic objects, and update and optimize maps accordingly.

[0051] 3. Semantic Information

[0052] Definition: Refers to information with meaning or function such as various objects, roads, and traffic rules in the environment. This information can be used to explain the meaning and role of these objects in the real world. For example, in a set of information x, y, z, r, g, b, label describing a point cloud, x, y, z represent the coordinates of the point cloud, r, g, b represent the color of the point cloud in the RGB channel, and label is semantic information, which is used to indicate what object the point cloud belongs to.

[0053] 4. Prior semantic map

[0054] Definition: A semantic map is a map representation that integrates geographic information and semantic information. A priori semantic map refers to a map containing semantic information obtained by semantically annotating the target area through a 3D target detection model when constructing the initial global map.

[0055] Purpose: Serves as a benchmark and reference for subsequent map updates and path planning in the SLAM system.

[0056] 5. Semantic Segmentation

[0057] Definition: Semantic segmentation is a technique in computer vision that aims to classify each pixel in an image or video into one of the predefined categories, such as road, vehicle, pedestrian, etc.

[0058] Purpose: In SLAM systems, semantic segmentation can be used to distinguish dynamic objects from static environments, thereby improving the robustness and accuracy of SLAM algorithms in dynamic environments.

[0059] Current SLAM technology usually adopts a global reconstruction strategy when updating its electronic map. This process not only leads to excessive consumption of computing resources, but also significantly increases the communication burden, which has a negative impact on the overall operating efficiency, resulting in low overall operating efficiency and further reducing the real-time performance of electronic map updates.

[0060] Based on the above principle, in this embodiment, a method for updating an electronic map is provided. Figure 1 , the electronic map updating method comprises the following steps:

[0061] S1. Acquire the original point cloud data set and image information of the scene within the first time period;

[0062] S2. According to the image information, semantic annotation and optimization of the region of interest are performed on the original point cloud dataset to obtain a priori semantic map;

[0063] S3. During the second time period, obtaining a real-time image of the scene;

[0064] S4. Update the prior semantic map according to the real-time image.

[0065] The electronic map updating method in this embodiment can be applied to Figure 2 In the system shown, the system may be a car or a robot. In this embodiment, a robot that transports materials on a construction site is used as an example. Figure 2 This is illustrated with the example of the system in .

[0066] Reference Figure 2 The system includes a control module, a first detection component, and a second detection component. In this embodiment, the first detection component may be a laser radar, and the second detection component may be a camera that shoots through visible light. The control module may call the first detection component or the second detection component to detect the external environment, wherein the data detected by the first detection component is point cloud data, and the data detected by the second detection component is a visible light image. Specifically, the control module may execute each step in the electronic map update method.

[0067] The principle of the electronic map updating method in this embodiment is as follows: Figure 3 shown.

[0068] In step S1, the control module calls the first detection component, i.e., the laser radar, to detect the scene where the robot is located, i.e., the construction site, within the first time period T1, thereby obtaining an original point cloud data set P = {p1, p2, ..., p n}, where p i It represents the data in the i-th spatial position obtained by the first detection component through laser detection, and n such data constitute a point cloud.

[0069] In step S1, the control module calls the second detection component, namely the camera, to photograph the scene where the robot is located, namely the construction site, within the first time period T1, so as to obtain image information image.

[0070] In step S2, the control module performs semantic annotation and optimization of the region of interest on the original point cloud dataset P according to the image information image, and obtains a priori semantic map P semanteme . Prior semantic map P semanteme It is also in the form of point cloud data, some of which are semantically annotated, so it contains both point cloud information and semantic information.

[0071] In step S3, in a second time period T2 after the first time period T1, the control module can call the second detection component to detect the scene, thereby obtaining a real-time image image realtime The control module can also detect the real-time point cloud P realtime At the same time, that is, in the second time period T2, the first detection component, specifically the laser radar, is called to detect the scene, so as to obtain the real-time point cloud P realtime .

[0072] The control module then executes step S4, according to the image image realtime Determine the prior semantic map P semanteme Whether the target represented by the relevant part in has changed (for example, the location where the target object existed in the first time period T1, but now does not exist in the second time period T2, or the location where the target object did not exist in the first time period T1, but now exists in the second time period T2). If so, the prior semantic map P is updated. semanteme If there is no change, the prior semantic map P is maintained. semanteme unchanged, so that the prior semantic map P semanteme The points at each position in represent the existence of the object, which is consistent with the latest detected result in the second time period T2, thereby realizing the update of the prior semantic map.

[0073] In this embodiment, the principle of executing steps S1-S4 is that the prior semantic map is obtained by semantically annotating and optimizing the area of ​​interest of the original point cloud dataset, which can avoid comprehensive semantic division of the original point cloud dataset covering the entire scene, thereby significantly reducing the computing overhead, which is conducive to maintaining good real-time performance when performing electronic map updates in complex and dynamically changing scenes such as construction sites; by semantically annotating and optimizing the area of ​​interest of the original point cloud dataset, the obtained prior semantic map has accurate semantic annotation information, and can quickly determine the area in the prior semantic map that may need to be updated according to the semantic information of the real-time image when updating according to the real-time image, thereby reducing the possibility of needing to perform a global search of the prior semantic map, and can also significantly reduce the computing overhead, improve the speed and real-time performance of electronic map updates, and improve the navigation efficiency and use safety of execution entities such as robots or cars in the scene.

[0074] In this embodiment, when executing step S2, that is, performing semantic annotation and optimization of the region of interest on the original point cloud data set according to the image information to obtain a priori semantic map, the following steps may be specifically performed:

[0075] S201. Perform target detection on the original point cloud data set to determine at least one region of interest;

[0076] S202. semantically annotating a point cloud subset within the region of interest, thereby determining a corresponding local semantic space in the original point cloud dataset;

[0077] S203. According to the semantic category, a local semantic space with corresponding semantic labels is cut out from the original point cloud dataset to obtain a priori semantic map.

[0078] In step S201, the control module can run a 3D object detection model (such as PointRCNN, etc.) on the original point cloud dataset P = {p1, P2, ..., p n} Perform target detection and determine at least one region of interest. Each region of interest has a corresponding bounding box. For example, the bounding box of the jth region of interest is B j , the original point cloud dataset P is in the bounding box B j The point cloud in forms a point cloud subset P B ={p i ∈P|p i ∈B j}.

[0079] In step S202, the control module can automatically B Perform semantic recognition, or manually input the point cloud subset P B The semantics of the point cloud, so that the point cloud subset P B Perform semantic annotation so that the point cloud subset P B Get a corresponding semantic label l i . Semantic tags i Represents the point cloud subset P B Point cloud subset P B and the corresponding semantic label l i Form a local semantic space S B , that is, S B ={(p i ,l i )|p i ∈P B}.

[0080] In step S203, the control module may define a screening function f(d i ,l j ), define if the semantic category d i With semantic tags i Corresponding (can be the same, for example, semantic category d i and semantic label L iThe contents of both are "stools"), then f(d i ,l j )=1, if the semantic category d i With semantic tags i Not corresponding (e.g. different content, such as semantic category d i The content is "stool", the semantic tag l i The content of is "table"), then f(d i ,l j )=0. Traversing all i and j, the semantic label l that can make the value of the screening function f 1 i Filter out and filter out the semantic tags l i The corresponding local semantic space S B Cut out from the original point cloud dataset P and mark the cut out local semantic space as

[0081] In this embodiment, for each local semantic space cut out A certain extended area buffer can be set for it in the local semantic space Based on this, we extend the expansion area buffer outward to obtain That is In this embodiment, Indicates an expansion operation, buffer can be 0, which means no local semantic space is used. The entire local semantic space cut out and its extended area constitute the prior semantic map.

[0082] In this embodiment, before executing step S4 (specifically, before executing steps S1-S4, or after executing S1-S3 and before executing step S4), the following steps can be performed to construct a nonlinear optimization problem based on constraints such as the relative position and direction between the first detection component (lidar) and the second detection component (camera), and solve the external calibration parameters, so as to align the first detection component (lidar) and the second detection component (camera):

[0083] (1) The point detected by the first detection component (lidar) is converted to the coordinate system of the second detection component (camera) to satisfy:

[0084]

[0085] in is the i-th point detected by the first detection component (laser radar), For the general The point obtained by converting to the coordinate system of the second detection component (camera), (R, T) is the transformation matrix from the first detection component (lidar) to the second detection component (camera);

[0086] (2) For The point represented by the second detection component (camera) can be directly photographed to obtain the corresponding pixel point is still a point in three-dimensional space. The second detection component (camera) can be calibrated with internal parameters, or its internal camera matrix K can be obtained from the product parameters of the second detection component (camera), where the internal camera matrix K can be expressed as

[0087]

[0088] The form shown;

[0089] (3) Using the internal camera matrix K, Projected onto the image plane,

[0090]

[0091] Can Normalize to get Specifically, Expressed as

[0092]

[0093] The form shown, where u, v and w represent , T represents the transpose of the matrix. According to the formula

[0094]

[0095] Calculated

[0096] (4) After calculating After that, the calculation is performed using the following formula:

[0097]

[0098] in, Indicates that it is calculated based on (R, T) before optimization represents the reprojection error between the first detection component and the second detection component, and N represents the number of points detected by the first detection component (lidar) and the second detection component (camera);

[0099] (5) Using nonlinear least squares method Levenberg-Marquardt and other optimization algorithms, (R, T) is gradually optimized. In the first step of optimization, an initial value R0 can be estimated for R, and an initial value T0 can be estimated for T.

[0100] According to the matrix T and the external calibration parameter R obtained by optimizing the above steps (1)-(5), the transformation matrix [R, T] can be determined. The transformation matrix [R, T] can convert the coordinates of the point detected by the first detection component (lidar) into the coordinates of the same point detected by the second detection component (camera). Similarly, [R, T] can be obtained according to the transformation matrix [R, T] -1 , using [R,T] -1 The coordinates of the point detected by the second detection component (camera) are converted into coordinates equivalent to the same point detected by the first detection component (lidar).

[0101] In this embodiment, when executing step S4, that is, updating the prior semantic map according to the real-time image, the following steps may be specifically performed:

[0102] S401. Registering the prior semantic map;

[0103] S402. Determine target semantics;

[0104] S403. Detecting the real-time image and the registered prior semantic map according to the target semantics;

[0105] S404. When no portion corresponding to the target semantics is detected from the real-time image, and no point cloud corresponding to the target semantics is detected from the prior semantic map, the prior semantic map is kept unchanged;

[0106] S405. When no portion corresponding to the target semantics is detected from the real-time image, and a point cloud corresponding to the target semantics is detected from the prior semantic map, the point cloud detected in the prior semantic map is deleted;

[0107] S406. Detecting the real-time point cloud and the prior semantic map according to the target semantics;

[0108] S407. When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is not detected from the prior semantic map, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame;

[0109] S408. When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is detected from the prior semantic map, a position change between the point cloud detected from the real-time point cloud and the point cloud detected from the prior semantic map is detected;

[0110] S409. When a position change is detected, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame;

[0111] S410. When no position change is detected, keep the prior semantic map unchanged.

[0112] In S401, a rigid transformation matrix T can be used * The prior semantic map obtained by executing step S2 is registered. In this embodiment, the rigid transformation matrix T can be determined by the following steps: * Specific values:

[0113] (A) Using the rigid transformation matrix T * The local semantic space cut out after executing steps S201-S203 is expanded Perform registration;

[0114] (B) The local semantic space obtained after expansion The matching error between the point cloud data after registration and before registration is minimized as the optimization goal, and the optimization equation is established; specifically, the optimization equation is as follows:

[0115]

[0116] Among them, T * ·p k Represents the use of rigid transformation matrix T * The local semantic space cut out Point p in k Perform registration, |T * ·p k -p k | 2 Represents the local semantic space cut out Point p in k The value T after registration * ·p k Compared with the value p before registration k The matching error of

[0117] (C) Solve the optimization equation in step (B) and calculate the rigid transformation matrix T * Perform optimization solution.

[0118] By performing steps (A)-(C), the rigid transformation matrix T can be determined * The specific value of . Use the rigid transformation matrix T * Register the points in the prior semantic map.

[0119] In step S402, the control module may use the original point cloud data set P obtained in step S1 and the target object present in the image information image, such as "stool", to obtain the semantics of the target object, thereby obtaining the target semantics.

[0120] In step S403, the control module detects the real-time image and the prior semantic map respectively, taking "whether there is a point cloud (or part of the image) corresponding to the target semantics (such as a stool)" as the detection target.

[0121] By executing step S403, possible detection results obtained include:

[0122] ① The part corresponding to the target semantics is not detected from the real-time image, and the point cloud corresponding to the target semantics is not detected from the prior semantic map;

[0123] ② The part corresponding to the target semantics is not detected from the real-time image, and the point cloud corresponding to the target semantics is detected from the prior semantic map.

[0124] If the detection result of step S403 is ①, for example, the real-time image image detected in the second time period T2 realtime does not contain the target semantic part of "stool", and the prior semantic map P detected in the first time period T1 semanteme does not contain a point cloud with the target semantics of "stool", which means that in the scene where the robot is located, there is no target object "stool" from the first time period T1 to the second time period T2, then step S404 is executed, and it is considered that the scene has not changed, and the prior semantic map P is maintained. semanteme constant.

[0125] If the detection result of step S403 is ②, for example, the real-time image image detected in the second time period T2 realtime does not contain the target semantic part of "stool", but the prior semantic map P detected in the first time period T1 semanteme contains a point cloud with the target semantics of "stool", which means that in the scene where the robot is located, there is a target object "stool" in the first time period T1, but there is no target object "stool" in the second time period T2, then step S405 is executed, and it is considered that the target object "stool" in the scene is removed, and the prior semantic map P is deleted. semanteme The detected point cloud.

[0126] In this embodiment, steps S403-S405 are equivalent to the steps of roughly detecting and updating the prior semantic map, and steps S404-S405 require relatively less data processing, so that the prior semantic map can be quickly updated.

[0127] In this embodiment, by executing step S403, the detection result that may be obtained may still be "a part corresponding to the target semantics is detected from the real-time image", and step S406 can be executed in this case.

[0128] In this embodiment, before executing step S406, the real-time point cloud P realtime Points in the real-time point cloud P are further filtered out of non-ground points and ground points. When the subsequent steps S406-S410 are executed, only the real-time point cloud P realtime The non-ground points in the

[0129] Specifically, the RANSAC algorithm can be combined with the region growing algorithm to analyze the real-time point cloud P realtime The points in the are fitted to determine the real-time point cloud P realtime The point corresponding to the ground plane in the real-time point cloud P realtime If the coordinates of a point in the real-time point cloud P satisfy the equation of the ground plane, then the point is determined to be on the ground plane, that is, the point belongs to the ground point; if the real-time point cloud P realtime If the coordinates of a point in the real-time point cloud P do not satisfy the equation of the ground plane, then it is determined that the point is not on the ground plane, that is, the point is a non-ground point. realtime In this embodiment, the real-time point cloud that only retains non-ground points can still be marked as P realtime .

[0130] Since the ground points are less likely to contain valid semantic information, we delete the real-time point cloud P realtime The ground points in the map can reduce the amount of data processing required for map updates without affecting the map update effect.

[0131] In step S406, similar to step S403, the control module detects the real-time point cloud and the prior semantic map respectively, with "whether there is a point cloud corresponding to the target semantics (such as a stool)" as the detection target.

[0132] By executing step S406, possible detection results obtained include:

[0133] ③ A point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is not detected from the prior semantic map;

[0134] ④ A point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is detected from the prior semantic map.

[0135] If the detection result of step S406 is ③, for example, the real-time point cloud P detected in the second time period T2 realtime contains the point cloud belonging to the target object “stool”, and the prior semantic map P detected in the first time period T1 semanteme does not contain the point cloud belonging to the target object "stool", which means that in the scene where the robot is located, there is no target object "stool" in the first time period T1, but the target object "stool" appears in the second time period T2, then step S407 is executed, and it is considered that the target object "stool" appears in the scene, and the control module performs the control based on the point cloud P from the real-time point cloud P. realtime The detected point cloud (that is, the point cloud corresponding to the target object "stool") generates a 3D detection frame, and the prior semantic map P is updated according to the 3D detection frame. semanteme , for example, the control module can map the 3D detection box to the prior semantic map P semanteme And the prior semantic map P semanteme The mapped part in the image is labeled with semantic information such as "stool" to represent the prior semantic map P semanteme The target object "stool" and its location in the game.

[0136] If the detection result of step S406 is ④, for example, the real-time point cloud P detected in the second time period T2 realtime contains the point cloud belonging to the target object “stool”, and the prior semantic map P detected in the first time period T1 semanteme , which also contains the point cloud belonging to the target object "stool", indicating that the target object "stool" has always existed in the scene where the robot is located from the first time period T1 to the second time period T2, then step S408 is executed to determine whether the target object "stool" is present in the scene from the real-time point cloud P realtime The point cloud of the target object “stool” detected in the image is consistent with the point cloud of the target object “stool” detected in the image. semanteme Whether there is any position change between the point clouds belonging to the target object "stool" detected in the image.

[0137] Specifically, in step S408, a 3D object recognition algorithm such as PointRCNN is used to obtain the prior semantic map P semanteme The point cloud belonging to the target object "stool" detected in the image is a directed bounding box (OBB). The point cloud inside the directed bounding box OBB is the point of the target object "stool". On the other hand, target detection algorithms such as YOLOv5 can be used to detect the target object from the real-time image. realtime Target objects such as stools and tables are identified in the real-time image image realtimeThe area in is the target area. In this embodiment, each target area has a bounding box. Then according to the transformation matrix [R, T] -1 , the real-time image image realtime The target area identified in the project is projected onto the real-time point cloud P realtime The coordinate system where the real-time point cloud P realtime Some points will be projected to the real-time point cloud P. Then, the control module can run clustering algorithms such as DBSCAN to realtime The projected parts are clustered to obtain multiple point cloud clusters. Multiple point cloud clusters clustered into the same category correspond to the same target object (for example, they all belong to the points detected by the first detection component, i.e., the laser radar, for the target object "stool"). Point cloud clusters clustered into different categories correspond to different target objects. realtime Any point cloud cluster formed by clustering can be obtained according to the formula

[0138]

[0139] Calculate the confidence C of this point cloud cluster. Among them, N points is the number of points in the point cloud cluster, d mean is the average distance between all points in the point cloud cluster and the center of the first detection component (lidar), and V is the volume of the point cloud cluster. α and β are weight parameters, which can balance the influence of point density and volume in the point cloud cluster on confidence.

[0140] After calculating the confidence of each point cloud cluster, a fixed confidence threshold can be set, and the confidence above the confidence threshold is judged as high confidence, and the confidence below or equal to the confidence threshold is judged as low confidence. The point cloud clusters with high confidence are screened out to form the point cloud belonging to the target object "stool". In this way, the real-time point cloud P realtime The whole is simplified into a real-time point cloud P realtime The point cloud of the target object "stool" in the image is not required for the real-time point cloud P realtime It is beneficial to improve data processing efficiency by processing other parts in the data.

[0141] In step S408, the control module can realtime The point cloud of the target object “stool” obtained by DBSCAN clustering is compared with the prior semantic map P semanteme By comparing the oriented bounding box (OBB) in the real-time point cloud P realtime The change is evaluated by whether a certain point q in the point cloud belonging to the target object "stool" obtained by clustering is located inside the OBB, and the points q located inside the OBB are counted, accounting for the real-time point cloud P realtimeThe ratio of all points in the point cloud of the target object "stool" obtained by clustering. Comparing this ratio with the threshold, there are two specific cases:

[0142] ④a. If the ratio is greater than the threshold, it can be determined that the target object "stool" has changed position, then step S409 is executed, and the control module changes the position of the target object according to the real-time point cloud P realtime The clustered point cloud (that is, the point cloud corresponding to the target object "stool") generates a 3D detection frame, and the prior semantic map P is updated according to the 3D detection frame. semanteme , for example, the control module can map the 3D detection box to the prior semantic map P semanteme And the prior semantic map P semanteme The mapped part in the image is labeled with semantic information such as "stool" to represent the prior semantic map P semanteme The target object "stool" and its location in the image are removed, and the prior semantic map P is deleted. semanteme The point cloud detected in the image is used to indicate that the target object “stool” no longer exists in the original position.

[0143] ④b. If the ratio is less than or equal to the threshold, it can be determined that the target object "stool" has no position change, then step S410 is executed, and it is considered that the scene has not changed, and the prior semantic map P is maintained. semanteme constant.

[0144] Steps S406-S410 are equivalent to the steps of performing detailed detection and updating of the prior semantic map. By dividing the rough detection and updating (steps S403-S405) and the detailed detection and updating (steps S406-S410), it is possible to implement the steps of performing only the rough detection and updating in certain situations, for example, only performing the prior semantic map P semanteme Partial deletion or maintaining unchanged eliminates the need to perform detailed detection and update steps, reducing the amount of data processing required; and the execution speed of rough detection and update steps is generally faster than the execution speed of detailed detection and update steps, so the processing speed can be improved.

[0145] In this embodiment, steps S403-S410 respectively modify the prior semantic map P by using the strategies of “addition, deletion, and modification”. semanteme Dynamic update can ensure the prior semantic map P semanteme accuracy and timeliness.

[0146] In this embodiment, the prior semantic map P semanteme After updating, the prior semantic map P semantemeThe target objects in the navigation system are divided into three categories: static objects, semi-static objects, and dynamic objects, and corresponding treatment measures and safety measures are formulated in path planning. For example, static objects, semi-static objects, and dynamic objects can be managed in different levels, and objects of the corresponding level can be selected for path planning according to tasks with different danger levels, thereby improving the safety and efficiency of navigation.

[0147] The electronic map updating method in this embodiment significantly reduces the computational complexity by limiting the semantic annotation process to the target area of ​​interest when semantically marking the map, avoiding semantic annotation processing that fully covers the entire scene; by comparing the target objects appearing in the real-time point cloud with the information in the prior semantic map, the current position of the robot is located, and the semantic information is fully utilized in combination with the image recognition results, and the computational efficiency and accuracy of the map update are optimized by narrowing the search space and targeted area screening. The effects of the electronic map updating method in this embodiment include:

[0148] High efficiency: The semantic map construction method based on 3D object detection greatly reduces the amount of point cloud data to be processed, significantly reducing the computational burden. At the same time, the semantic-driven multi-stage map dynamic update strategy effectively improves the efficiency of the system, especially in dealing with dynamically changing building environments.

[0149] Higher safety: The system manages objects in the map in different levels based on object recognition results, and selects objects of corresponding levels for path planning according to tasks with different danger levels.

[0150] Resource consumption is significantly reduced and real-time performance is improved: the system only updates the changed parts. After obtaining the prior semantic map, we combine the semantic-guided point cloud efficient segmentation of target detection with the semantic-driven multi-stage map dynamic update strategy to update the map in real time. Through this method, we have significantly reduced the computing overhead and improved the real-time performance of the system.

[0151] Steps S1-S4 and other steps in this embodiment are executed dynamically in real time. For example, in a construction site scenario, a material transport robot can execute steps S1-S2 in the first time period T1 (such as the time period when the project is about to start on that day) to obtain a priori semantic map, and the priori semantic map can be used as the latest global map in the scenario on that day; in the second time period T2, such as when the robot is transporting materials, high real-time performance is required at this time, and steps S3-S4 can be executed to combine the semantically guided efficient segmentation of point clouds for target detection with the semantically driven multi-stage map dynamic update strategy, so as to update the map in real time based on the priori semantic map, thereby significantly reducing the robot's computing overhead and improving the robot's real-time performance, thereby effectively improving the robot's navigation and work efficiency in the construction scene.

[0152] A computer program that executes the electronic map updating method in this embodiment can be written and written into a computer device or storage medium. When the computer program is read out and run, the electronic map updating method in this embodiment is executed, thereby achieving the same technical effect as the electronic map updating method in the embodiment.

[0153] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by technicians in this technical field. The terms used in the specification of this embodiment are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0154] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0155] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0156] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0157] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0158] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0159] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. An electronic map updating method, characterized in that: The electronic map updating method comprises: In a first time period, an original point cloud data set and image information of the scene are obtained; According to the image information, semantically annotating and optimizing the region of interest of the original point cloud data set to obtain a priori semantic map; Acquire a real-time image of the scene within a second time period; the second time period is a time period after the first time period; The priori semantic map is updated according to the real-time image.

2. The electronic map updating method according to claim 1, characterized in that: The step of semantically annotating and optimizing the region of interest of the original point cloud data set according to the image information to obtain a priori semantic map includes: Performing target detection on the original point cloud data set to determine at least one region of interest; The point cloud subset within the region of interest is semantically annotated, so as to determine a corresponding local semantic space in the original point cloud data set; the local semantic space includes the point cloud subset within the region of interest and a semantic label obtained by the semantic annotation.

3. The electronic map updating method according to claim 2, characterized in that: The step of semantically annotating and optimizing the region of interest of the original point cloud data set according to the image information to obtain a priori semantic map further includes: According to the semantic category, the local semantic space with the corresponding semantic label is cut out from the original point cloud data set to obtain the priori semantic map.

4. The electronic map updating method according to any one of claims 1 to 3, characterized in that: The updating of the prior semantic map according to the real-time image includes: registering the prior semantic map; Determine the target semantics; According to the target semantics, detecting the real-time image and the registered prior semantic map; When no portion corresponding to the target semantics is detected from the real-time image, and no point cloud corresponding to the target semantics is detected from the priori semantic map, keeping the priori semantic map unchanged; When a portion corresponding to the target semantics is not detected from the real-time image, and a point cloud corresponding to the target semantics is detected from the priori semantic map, the point cloud detected in the priori semantic map is deleted.

5. The electronic map updating method according to claim 4, characterized in that: The updating of the prior semantic map according to the real-time point cloud further includes: In the second time period, a real-time point cloud of the scene is obtained; According to the target semantics, detecting the real-time point cloud and the prior semantic map; When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is not detected from the prior semantic map, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame; When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is detected from the prior semantic map, a position change between the point cloud detected from the real-time point cloud and the point cloud detected from the prior semantic map is detected; when the position change is detected, a 3D detection frame is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection frame; when the position change is not detected, the prior semantic map is kept unchanged.

6. The electronic map updating method according to claim 5, characterized in that: The detecting the real-time point cloud and the prior semantic map according to the target semantics includes: Performing three-dimensional target detection on the prior semantic map according to the target semantics to determine a directed bounding box; Performing target detection on the real-time image according to the target semantics to determine the target area; Projecting the target area onto the real-time point cloud; Clustering the projected points in the real-time point cloud to obtain at least one point cloud cluster; Screening each of the point cloud clusters; The screened point cloud cluster is used as the point cloud detected from the real-time point cloud.

7. The electronic map updating method according to claim 6, characterized in that: The screening of each of the point cloud clusters comprises: Respectively obtain the confidence of each of the point cloud clusters; The point cloud clusters whose corresponding confidence levels are higher than a confidence threshold are screened out.

8. The electronic map updating method according to claim 7, characterized in that: The obtaining the confidence of each of the point cloud clusters respectively includes: For any of the point cloud clusters, according to the formula Calculate, where C represents the confidence of the point cloud cluster, N points is the number of points in the point cloud cluster, d mean is the average distance between the point cloud cluster and the center of the first detection component, C is the volume of the point cloud cluster, and α and β are weight parameters.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the electronic map updating method according to any one of claims 1-8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the electronic map updating method described in any one of claims 1 to 8 when executed by the processor.

Images