Map positioning and maintaining method and device, electronic equipment and storage medium
Through the combination of deep learning object detection and semantic information, a joint optimization function is built for robot positioning and the map is updated in real time, which solves the problems of poor positioning effect and insufficient map maintenance in the existing technology, and improves positioning accuracy and stability.
Patent Information
- Application Number
- CN202411238705.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
Existing robot positioning methods are prone to failure in complex or degraded scenarios, resulting in poor positioning effect, map maintenance is not considered, and it cannot adapt to most practical application scenarios.
The combination of object detection and semantic information based on deep learning is adopted, and the positioning information is obtained through real-time environmental image object detection, a joint optimization function is constructed for iterative optimization, the optimal positioning information is output, and the semantic map and laser point cloud map are updated when the conditions are met.
It improves the positioning accuracy and stability of the robot in complex scenarios, expands the application scenarios, and realizes real-time maintenance of maps, avoiding the problem of laser point cloud positioning failure.
Smart Images

Figure CN119022918A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of laser point cloud technology, and in particular to a map positioning and maintenance method, device, electronic equipment and storage medium. Background Art
[0002] In recent years, laser point clouds and point cloud maps have been widely used as the main source of sensor information for positioning in low-speed unmanned driving products. Taking the low-speed unmanned driving robot in the service robot as an example, the robot's 6-DOF positioning information is currently obtained mainly by aligning the laser point cloud frame with the original point cloud map. Among them, the point cloud registration methods mainly include ICP (Iterative Closest Point, an algorithm widely used in point cloud registration), NICP (Non-Rigid Iterative Closest Point, an algorithm for non-rigid registration) and NDT (Normal Distributions Transform, an algorithm for point cloud registration). These methods are based on the geometric constraints of the point cloud, and the optimal solution is solved by the least squares method to calculate the 6-DOF positioning information.
[0003] For robots, 6-DOF positioning information is also commonly referred to as position and posture, that is, position and posture. Generally speaking, the above-mentioned registration methods can meet the effectiveness and positioning accuracy required by robots. However, in complex or degraded scenarios, these positioning methods based on geometric constraints are prone to failure, resulting in poor positioning results. For example, places with dense traffic or parks with missing geometric features. In actual situations, places with sufficient and stable geometric constraints are rare in real usage scenarios, but other unfavorable scenarios account for the majority. For example, sweepers are used in parks, sidewalks, parks, etc. And most of the current positioning methods do not consider map maintenance. Summary of the invention
[0004] The present invention provides a map positioning and maintenance method, device, electronic device and storage medium, which are used to solve or partially solve the technical problems of poor positioning effect and failure to consider map maintenance in existing robot-related positioning methods.
[0005] The present invention provides a map positioning and maintenance method, which is applied to a service robot, and the method comprises:
[0006] Get the original location information of the visited area;
[0007] Collecting a real-time environment image of the visited area, performing target detection on the real-time environment image to extract real-time positioning information, and performing positioning verification based on the original positioning information and the real-time positioning information;
[0008] When the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information;
[0009] When a preset update condition is met, updating the semantic map of the visited area in combination with the real-time positioning information;
[0010] When the service robot completes the task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information.
[0011] Optionally, the original positioning information includes a laser point cloud map and a three-dimensional point cloud model of the visited area, and the target detection is performed on the real-time environment image to extract the real-time positioning information, and the positioning verification is performed according to the original positioning information and the real-time positioning information, including:
[0012] Extracting first geometric feature information and first semantic information of each identifiable target in the access area from the three-dimensional point cloud model;
[0013] Recognize and segment the real-time environment image through target detection based on a deep learning algorithm, and extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area;
[0014] Performing affine transformation on each of the second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information having the same semantic information in the laser point cloud map, thereby obtaining initial position information of the service robot;
[0015] Projecting each of the second geometric feature information onto the laser point cloud map according to the initial pose information;
[0016] Based on the feature correspondence, the first geometric feature information and the second geometric feature information are matched to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.
[0017] Optionally, when the positioning check passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output optimal positioning information, including:
[0018] When the positioning test is passed, it indicates that each of the second geometric feature information can be matched to the corresponding first geometric feature information;
[0019] Constructing geometric constraints, image processing constraints and semantic constraints, and combining the geometric constraints, the image processing constraints and the semantic constraints to construct a joint optimization function;
[0020] The joint optimization function is used to iteratively optimize the initial posture information of the service robot through the least squares method to calculate the optimal posture that satisfies the geometric constraints, the image processing constraints and the semantic constraints at the same time as the optimal positioning information.
[0021] Optionally, the method further comprises:
[0022] If the positioning check fails, it means that the second geometric feature information cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning is invalid. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.
[0023] Optionally, after each positioning check is passed, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary cache area; when a preset update condition is met, the semantic map of the access area is updated in combination with the real-time positioning information, including:
[0024] During the task execution, the number of visits to the visit area is obtained in real time;
[0025] When it is determined that the number of accesses is greater than or equal to a preset access threshold, the image acquisition time and the light intensity at the last positioning up to now are obtained;
[0026] When it is determined that the illumination intensity exceeds the preset illumination intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning check passed.
[0027] Optionally, after each positioning check is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-constructed temporary cache area of point cloud information; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information, including:
[0028] When the service robot completes the task, the laser point cloud information is reconstructed using the multiple sets of second geometric feature information cached during the task execution, and a new laser point cloud map is reconstructed based on the laser point cloud information;
[0029] If the map reconstruction fails, canceling the updating of the laser point cloud map of the visited area;
[0030] If the map reconstruction is successful, determine whether the newly created laser point cloud map is dense;
[0031] If so, the newly created laser point cloud map is compared with the original laser point cloud map, and based on the comparison result, the original laser point cloud map is supplemented or deleted to update the laser point cloud map of the visited area.
[0032] Optionally, before the service robot performs a task, the method further includes:
[0033] Collecting laser point cloud data of the access area and surrounding environment images through information collection equipment;
[0034] Using the laser point cloud data, constructing a laser point cloud map of the visited area;
[0035] Extracting first semantic information of each identifiable target in the access area from the surrounding environment image, and first segmenting and then clustering the laser point cloud map according to each of the first semantic information to obtain laser point clusters corresponding to each of the identifiable targets;
[0036] Offline three-dimensional reconstruction is performed based on each group of laser point clusters, and finally combined into a three-dimensional point cloud model of the access area.
[0037] The present invention also provides a map positioning and maintenance device, which is applied to a service robot, and the device comprises:
[0038] An information acquisition module is used to obtain the original positioning information of the access area;
[0039] A positioning verification module, used to collect a real-time environment image of the visited area, perform target detection on the real-time environment image to extract real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information;
[0040] An information output module is used to construct a joint optimization function in combination with multi-condition positioning constraints when the positioning test passes, and iteratively optimize the real-time positioning information based on the joint optimization function to output the optimal positioning information;
[0041] A semantic map updating module, configured to update the semantic map of the visited area in combination with the real-time positioning information when a preset update condition is met;
[0042] The point cloud map updating module is used to update the laser point cloud map of the visited area offline in combination with the real-time positioning information when the service robot completes the task.
[0043] The present invention also provides an electronic device, the device comprising a processor and a memory:
[0044] The memory is used to store program code and transmit the program code to the processor;
[0045] The processor is used to execute the map positioning and maintenance method as described in any one of the above items according to the instructions in the program code.
[0046] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the map positioning and maintenance method as described in any one of the above items.
[0047] It can be seen from the above technical solutions that the present invention has the following advantages:
[0048] A map positioning and maintenance method for a service robot is provided. First, the original positioning information of the access area is obtained; then, the real-time environment image of the access area is collected, and the target detection is performed on the real-time environment image to extract the real-time positioning information, and the positioning test is performed based on the original positioning information and the real-time positioning information; when the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information. Therefore, more accurate and effective positioning information can be obtained through positioning test and joint optimization function based on multi-condition positioning constraints. When the preset update conditions are met, the semantic map of the access area is updated in combination with the real-time positioning information; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information. Therefore, considering the maintenance of the laser point cloud map and the semantic map at the same time, different update mechanisms are set respectively, and the existing map is updated and maintained in combination with the real-time collected information. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 A flowchart of the steps of a map positioning and maintenance method;
[0051] Figure 2 It is a schematic diagram of the overall process of a map positioning and maintenance method;
[0052] Figure 3 The present invention is a structural block diagram of a map positioning and maintenance device. DETAILED DESCRIPTION
[0053] The embodiments of the present invention provide a map positioning and maintenance method, device, electronic device and storage medium, which are used to solve or partially solve the poor positioning effect existing in the existing robot-related positioning methods and the technical problem that map maintenance is not taken into consideration.
[0054] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] As an example, taking a low-speed unmanned robot in a service robot as an example, the current main method is to obtain the robot's 6-DOF positioning information by registering the laser point cloud frame with the original point cloud map. Among them, the main point cloud registration methods include ICP, NICP and NDT. These methods are based on the geometric constraints of the point cloud, and use the least squares method to solve the optimal solution to calculate the 6-DOF positioning information.
[0056] Generally speaking, the above-mentioned registration methods can meet the effectiveness and positioning accuracy required by the robot. However, in complex or degraded scenarios, these positioning methods based on geometric constraints are prone to failure, resulting in poor positioning results. For example, places with dense traffic, or parks with missing geometric features. In reality, places with sufficient and stable geometric constraints are rare in real usage scenarios, but other unfavorable scenarios account for the majority. For example, sweepers are used in parks, sidewalks, parks, etc. And most of the current positioning methods do not consider map maintenance.
[0057] High-precision semantic map refers to the process of assigning target categories in visual images to corresponding laser point clusters to form labels in a three-dimensional map composed of high-precision laser point clouds through deep learning and other means. Target detection based on deep learning includes category detection and corresponding image region segmentation. The segmented pixels can be projected onto the map through geometric relationships. Currently, there is no relevant literature proposing the use of high-precision semantic maps in low-speed unmanned robots and related means for maintaining existing maps.
[0058] Therefore, one of the core inventive points of the embodiment of the present invention is: for robot positioning scenarios, a map positioning and maintenance method for service robots is provided. On the one hand, a semantic map based on high-precision laser point cloud is used in a service robot (such as a low-speed unmanned robot), so as to increase the information redundancy of the laser point cloud during positioning through semantic information; real-time detection of identifiable targets in the surrounding environment, combining target categories with point cloud computing positioning, and through positioning verification and joint iterative optimization, visual verification is added in scenarios that are conducive to laser positioning, and stable positioning is provided as a backup in scenarios where laser positioning fails, thereby further improving the stability and accuracy of positioning and avoiding the problem of laser point cloud positioning failure. On the other hand, the existing point cloud map and semantic information are maintained through the laser and visual data collected by the service robot each time it performs a task, thereby improving the stability and accuracy of the robot positioning, expanding the application scenarios, and filling the gap in the maintenance of existing maps that is not considered in the current robot positioning method.
[0059] Specifically, the original positioning information of the access area is first obtained; then the real-time environment image of the access area is collected, and the target detection is performed on the real-time environment image to extract the real-time positioning information, and the positioning test is performed based on the original positioning information and the real-time positioning information; when the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information. Therefore, more accurate and effective positioning information can be obtained through positioning verification and joint optimization functions based on multi-condition positioning constraints. When the preset update conditions are met, the semantic map of the access area is updated in combination with the real-time positioning information; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information. Therefore, considering the maintenance of the laser point cloud map and the semantic map at the same time, different update mechanisms are set respectively, and the existing map is updated and maintained in combination with the real-time collected information.
[0060] Reference Figure 1 , shows a flowchart of the steps of a map positioning and maintenance method provided by an embodiment of the present invention, the method is applied to a service robot, and the method may specifically include the following steps:
[0061] Step 101, obtaining original positioning information of the access area;
[0062] In the embodiment of the present invention, a low-speed unmanned robot in a service robot is used as an example for explanation. For a certain area (hereinafter referred to as the access area), before the low-speed unmanned robot needs to perform related tasks in the access area, the surrounding environment image can be collected by a camera, a mobile phone terminal and other information collection devices, and the initial laser point cloud map and three-dimensional point cloud model of the access area can be constructed based on the laser point cloud technology combined with the image processing technology, so that in the subsequent process, the positioning information obtained by real-time processing during the execution of the task can be combined with the pre-constructed laser point cloud map and three-dimensional point cloud model to perform positioning inspection, thereby avoiding the problem of positioning failure.
[0063] In a specific implementation, before a low-speed unmanned robot performs a task, laser point cloud data of the access area can be collected through information collection devices such as cameras and mobile terminals. While collecting laser point cloud data, the surrounding environment image can also be sampled to extract relevant semantic information. Then, the laser point cloud data is used to construct a laser point cloud map of the access area.
[0064] After completing the construction of the high-precision laser point cloud map, the first semantic information of each identifiable target (such as buildings, road signs, railings, etc.) in the visited area is extracted from the surrounding environment image (to distinguish it from the semantic information obtained during the subsequent real-time positioning, the semantic information corresponding to this process is defined as the first semantic information, and the semantic information corresponding to the subsequent real-time positioning process is defined as the second semantic information).
[0065] Then, according to each first semantic information, the laser point cloud map is first segmented and then clustered to obtain laser point clusters corresponding to each identifiable target. Then, offline 3D reconstruction is performed based on each group of laser point clusters, and finally combined into a 3D point cloud model of the visited area.
[0066] In the subsequent processing flow, based on the 3D point cloud model, effective first geometric feature information can be extracted (similarly, in order to distinguish it from the geometric feature information obtained in the subsequent real-time positioning, the geometric feature information corresponding to this process is defined as the first geometric feature information, and the geometric feature information corresponding to the subsequent real-time positioning process is defined as the second geometric feature information). For example, plane information, plane intersection line information, describable surface information, etc.
[0067] Based on the 3D point cloud model, the first semantic information can also be extracted, which can also be understood as typical indication information, such as road arrows, traffic signs, LOGO, etc. The first geometric feature information and the first semantic information will be used in the subsequent positioning and inspection process.
[0068] Step 102, collecting a real-time environment image of the visited area, performing target detection on the real-time environment image to extract real-time positioning information, and performing positioning verification based on the original positioning information and the real-time positioning information;
[0069] It is understandable that, although both step 101 and step 102 involve the collection of the environmental image of the access area, what is collected in step 101 is equivalent to the environmental information before the robot actually works. And this information is collected once by the collection device, and is mainly used to construct the laser point cloud map. The environmental image in step 102 is the environmental information collected in real time by the robot when performing the task. In actual situations, since the environment may change, the environmental information collected in real time may be different from the environmental information collected once in the previous step 101. For example, during the time period from the first collection to the current collection, a road sign was removed, or a new railing was added, etc. By comparing the information collected in real time with the information collected once, it can be determined whether the map has changed.
[0070] Combined with the above discussion, it can be known that based on the construction of laser point cloud, the original positioning information can mainly include the laser point cloud map and the three-dimensional point cloud model of the access area. Specifically, the process of performing target detection on the real-time environment image to extract the real-time positioning information and performing positioning verification based on the original positioning information and the real-time positioning information can be implemented by executing the following sub-steps S01 to S05:
[0071] Step S01: extracting first geometric feature information and first semantic information of each identifiable target in the access area from the three-dimensional point cloud model;
[0072] Step S02: Recognize and segment the real-time environment image through target detection based on a deep learning algorithm, and extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area;
[0073] Step S03: performing affine transformation on each second semantic information, so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map, and obtaining the initial position and posture information of the service robot;
[0074] Step S04: projecting each second geometric feature information onto a laser point cloud map according to the initial posture information;
[0075] Step S05: Based on the feature correspondence, the first geometric feature information and the second geometric feature information are matched to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.
[0076] Step 103: when the positioning check passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output optimal positioning information;
[0077] The previous step 102 mainly describes the one-time pose calculation process of the low-speed unmanned robot. Through the mutual verification between the original acquisition information and the real-time acquisition information of the laser point cloud positioning, it can be determined whether the laser positioning has failed. When both the laser positioning result and the semantic positioning result are verified, or even if there is a slight error, but the difference between the two meets the requirements (the error is small and does not affect the overall positioning effect), the geometric constraints, image processing constraints and semantic constraints of the laser point cloud can be combined together to construct a joint optimization function to calculate the optimal pose that satisfies the relevant constraints at the same time.
[0078] Specifically, when the positioning check passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information. The process can be implemented by executing the following sub-steps S11 to S13:
[0079] Step S11: when the positioning test is passed, it indicates that each second geometric feature information can be matched to the corresponding first geometric feature information;
[0080] Step S12: constructing geometric constraints, image processing constraints and semantic constraints, and combining the geometric constraints, image processing constraints and semantic constraints to construct a joint optimization function;
[0081] Furthermore, geometric constraints can mainly include plane constraints, plane intersection line constraints, and surface constraints. Image processing constraints mainly include RGB constraints (Red-Green-Blue, a color model) (RGB Constraint) and grayscale constraints (Grayscale Constraint).
[0082] Among them, plane constraints refer to restricting one or more elements (such as points, lines, and surfaces) to be located on a specific plane. Plane intersection constraints refer to the constraints on the line formed at the intersection of two planes. Surface constraints involve ensuring the relationship between surfaces in surface modeling, such as tangency, smooth transition, alignment, etc.
[0083] RGB constraint refers to the imposition of certain conditions or restrictions on the values of the three color channels of red (R), green (G), and blue (B) of a pixel when processing a color image. Grayscale constraint is similar to RGB constraint, but is for grayscale images, that is, images with only brightness values and no color information.
[0084] Semantic constraints refer to limiting or guiding the behavior of positioning, recognition, or understanding algorithms based on semantic information in the environment (such as the category, function, and meaning of objects).
[0085] The above-mentioned constraints are conventional settings, and the specific settings are not the focus of the technical solution of the present invention. Those skilled in the art can flexibly set them according to actual conditions, and the present invention does not limit this.
[0086] After constructing the above constraints, we can further construct a joint optimization function, so that more accurate and effective positioning information can be obtained by optimizing the cost functions that consider geometric constraints, image processing constraints, and semantic constraints at the same time.
[0087] Step S13: adopting the joint optimization function, iteratively optimizing the initial posture information of the service robot through the least square method, so as to calculate the optimal posture that satisfies the geometric constraints, image processing constraints and semantic constraints at the same time, as the optimal positioning information.
[0088] Finally, a joint optimization function can be used to iteratively optimize the initial posture information of the service robot through the least squares method (that is, equivalent to iterating the initial posture of the robot) to calculate the optimal posture that satisfies the geometric constraints, image processing constraints, and semantic constraints at the same time as the optimal positioning information.
[0089] In another case, the positioning test may fail (i.e., the test fails). When the positioning test fails, it indicates that the second geometric feature information in the real-time positioning information cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning fails. In this case, the real-time positioning information has the problem of inaccurate positioning. It is known that the first semantic information in the map is collected in advance, so there may be deviations from the second semantic information currently detected in real time. For example, other semantic information is added or reduced, and the information is incomplete due to different viewing angles. In order to obtain stable positioning information in the scenario where laser positioning fails, the geometric feature information and semantic information in the original positioning information collected in advance can be used as the positioning result output of this positioning. Specifically, each first geometric feature information can be used as the target geometric feature information for this positioning, and each first semantic information can be used as the target semantic information for this positioning.
[0090] Step 104, when a preset update condition is met, updating the semantic map of the visited area in combination with the real-time positioning information;
[0091] Before the low-speed unmanned driving robot starts to perform tasks each time, a temporary semantic information cache area (temporary semantic information warehouse) can be built first. After each qualified semantic positioning (i.e., each positioning test passes), the second semantic information extracted in real time is temporarily stored in the temporary semantic information cache area.
[0092] Furthermore, for the update and maintenance of the semantic map, a semantic information update mechanism can be pre-set to update the semantic information in the map when the update conditions are met. The specific update content can mainly include: RGB color, grayscale, contour, indication information, and information positioning stabilization program.
[0093] Specifically, the robot may visit the same area multiple times during the execution of a task, and collect semantic information of the area multiple times. For the visited area, during the execution of a task, each successful positioning test is counted once, and the corresponding second semantic information is cached in the temporary cache area of the semantic information. Then, the current cumulative number of visits is judged in real time. When the number of visits to the visited area is less than a certain threshold (such as less than 5 times or 10 times, the threshold of the number of visits can be set according to the actual situation, and the present invention is not limited to this), the semantic information of the visited area is not updated. When the cumulative number reaches the above-mentioned preset threshold of the number of visits (such as 5 times or 10 times), the current semantic map is updated based on the second semantic information cached after the last positioning test.
[0094] The RGB and grayscale information in the semantic information are both related to the light intensity at that time. When the light is strong, the grayscale intensity is high, and the RGB is closer to the true value. Therefore, when updating the RGB and grayscale values, it is necessary to combine the time and light intensity when the image was acquired to build a judgment criterion to decide whether to update. For example, if the light intensity is within a certain range, no update is performed. When it is outside the range, the update is determined. The indicative information in the semantic information is very stable, so it can be supplemented or added directly based on the existing information. The contour information in the semantic information is the key to constructing geometric constraints. When updating, it can be supplemented multiple times by fitting the optimal boundary through geometric methods.
[0095] Among them, geometric method to fit the optimal boundary is a common task in computer vision, image processing and machine learning, which is mainly used to identify and segment the boundaries of objects in images. Achieving this goal usually relies on image feature extraction, edge detection, curve fitting and optimization technology. Since this processing flow is an existing conventional means and is not the focus of the technical solution of the present invention, it will not be described here.
[0096] In a specific implementation, when the preset update conditions are met, the semantic map of the access area is updated in combination with the real-time positioning information, which can be as follows: during the execution of the task, the number of visits to the access area is obtained in real time; when the number of visits is judged to be greater than or equal to the preset access threshold, the image acquisition time and light intensity at the time of the last positioning are obtained; when the light intensity is judged to exceed the preset light intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning check passed.
[0097] By combining the number of visits to the visited area during the execution of the task, it is judged in real time whether the environmental information of the current location has changed. Based on the judgment result, it is further determined whether to update the semantic information (semantic map). When the change condition is met (such as combining the acquisition time and the light intensity to judge that the current environmental information has changed), the information is updated. If it is not met, the semantic information is not updated temporarily. In this way, the real-time update and maintenance of the semantic map is realized.
[0098] Step 105, when the service robot completes the task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information.
[0099] Similar to the above, before the low-speed unmanned driving robot starts to perform tasks each time, it can also build a temporary point cloud information cache (temporary laser point cloud information warehouse). Therefore, even if the laser point cloud positioning may fail, when the positioning is normal (that is, each positioning test passes), the second geometric feature information collected in real time can be temporarily stored in the temporary point cloud information cache.
[0100] When the robot is performing a task, its moving trajectory is generally relatively stable. This is significantly different from when collecting maps. Therefore, considering this actual scenario, the laser point cloud collected by the robot each time it performs a task will not be updated to the map in real time. Instead, the update process is run offline when the robot returns to the station after completing the task.
[0101] Specifically, when the robot completes a task, the laser point cloud collected during the task will first be used to execute the laser mapping process to build a new laser point cloud map. However, in actual situations, there may be situations where the construction fails due to insufficient laser point cloud data collected or other abnormal conditions. When the construction cannot be completed, the update is canceled. After the map is successfully built, it is necessary to first evaluate whether the map is dense enough (such as whether the amount of information is sufficient, or whether the feature coverage is high enough, etc.), otherwise it cannot be compared and merged with the original map.
[0102] In actual operation, when collecting high-precision laser point cloud maps, the map will be divided into multiple local areas (this process can be combined with manual judgment to ensure the rationality of the area division). At this time, the constructed updated map can be imported into the copied local area, and these local areas can be further divided into three-dimensional grids. Then the original map is compared with the updated map to evaluate whether there are missing point clouds in each three-dimensional grid. Finally, the updated map is used to supplement or delete the three-dimensional grid.
[0103] In a specific implementation, when the service robot completes a task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information. This can be done as follows: when the service robot completes a task, the laser point cloud information is reconstructed using multiple sets of second geometric feature information cached during the execution of the task, and a new laser point cloud map is reconstructed based on the laser point cloud information; if the map reconstruction fails, the update of the laser point cloud map of the visited area is canceled; if the map reconstruction is successful, it is determined whether the newly built laser point cloud map is dense; if so, the newly built laser point cloud map is compared with the original laser point cloud map, and based on the comparison result, the original laser point cloud map is supplemented or deleted to update the laser point cloud map of the visited area.
[0104] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, some technical features are distinguished by the first and the second in the embodiments of the present invention. The first and the second are only used for data distinction and have no other special meanings. It can be understood that the present invention is not limited to this.
[0105] In an embodiment of the present invention, a map positioning and maintenance method for a service robot is provided. First, the original positioning information of the access area is obtained; then, the real-time environment image of the access area is collected, and target detection is performed on the real-time environment image to extract the real-time positioning information, and a positioning test is performed based on the original positioning information and the real-time positioning information; when the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information. Thus, more accurate and effective positioning information can be obtained through positioning verification and a joint optimization function based on multi-condition positioning constraints. When the preset update conditions are met, the semantic map of the access area is updated in combination with the real-time positioning information; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information. Thus, the maintenance of the laser point cloud map and the semantic map is taken into consideration at the same time, and different update mechanisms are set respectively, and the existing map is updated and maintained in combination with the real-time collected information.
[0106] For better explanation, refer to Figure 2, showing a schematic diagram of the overall process of a map positioning and maintenance method provided by an embodiment of the present invention. It should be pointed out that this embodiment only briefly describes the general process of map positioning and maintenance, and the specific implementation process of each step can be understood by referring to the relevant content in the aforementioned embodiment, which will not be repeated here. It can be understood that the present invention does not limit this.
[0107] Collect laser point cloud data of the visited area and surrounding environment images, and further construct a laser point cloud map and a three-dimensional point cloud model of the visited area for laser point cloud positioning.
[0108] The first geometric feature information and the first semantic information of each identifiable target in the access area are extracted from the three-dimensional point cloud model.
[0109] A real-time environment image of the access area is collected, and target detection is performed on the real-time environment image to extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area.
[0110] An affine transformation is performed on each piece of second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map, thereby obtaining the initial posture information of the service robot.
[0111] According to the initial posture information, each second geometric feature information is projected onto the laser point cloud map.
[0112] Based on the feature correspondence, the first geometric feature information and the second geometric feature information are matched to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.
[0113] When the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information.
[0114] After each positioning test is passed, the second semantic information extracted in real time is temporarily stored in a pre-built temporary cache area for semantic information, and the second geometric feature information extracted in real time is temporarily stored in a pre-built temporary cache area for point cloud information.
[0115] When the preset update conditions are met, the semantic map of the access area is updated in combination with the real-time positioning information.
[0116] When the service robot completes the task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information.
[0117] Reference Figure 3, shows a structural block diagram of a map positioning and maintenance device provided by an embodiment of the present invention, the device is applied to a service robot, and the device may specifically include:
[0118] The information acquisition module 301 is used to acquire the original positioning information of the access area;
[0119] A positioning verification module 302 is used to collect a real-time environment image of the visited area, perform target detection on the real-time environment image to extract real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information;
[0120] The information output module 303 is used to construct a joint optimization function in combination with multi-condition positioning constraints when the positioning test passes, and iteratively optimize the real-time positioning information based on the joint optimization function to output the optimal positioning information;
[0121] A semantic map updating module 304, configured to update the semantic map of the visited area in combination with the real-time positioning information when a preset update condition is met;
[0122] The point cloud map updating module 305 is used to update the laser point cloud map of the visited area offline in combination with the real-time positioning information when the service robot completes the task.
[0123] In an optional embodiment, the original positioning information includes a laser point cloud map and a three-dimensional point cloud model of the access area, and the positioning verification module 302 includes:
[0124] An information extraction module, used for extracting first geometric feature information and first semantic information of each identifiable target in the access area from the three-dimensional point cloud model;
[0125] A recognition and segmentation module, configured to recognize and segment the real-time environment image through target detection based on a deep learning algorithm, and extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area;
[0126] An affine transformation module, used for performing affine transformation on each of the second semantic information, so that the transformed second semantic information corresponds to and matches the first semantic information having the same semantic information in the laser point cloud map, and obtaining the initial posture information of the service robot;
[0127] An information projection module, used for projecting each of the second geometric feature information onto the laser point cloud map according to the initial posture information;
[0128] The feature information matching module is used to match the first geometric feature information with the second geometric feature information based on the feature correspondence relationship, so as to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.
[0129] In an optional embodiment, the information output module 303 includes:
[0130] A positioning inspection passing module, used for indicating that each of the second geometric feature information can be matched to the corresponding first geometric feature information when the positioning inspection passes;
[0131] A joint optimization function construction module, used to construct geometric constraints, image processing constraints and semantic constraints, and to combine the geometric constraints, the image processing constraints and the semantic constraints to construct a joint optimization function;
[0132] An iterative optimization module is used to adopt the joint optimization function to iteratively optimize the initial posture information of the service robot through the least squares method to calculate the optimal posture that simultaneously satisfies the geometric constraints, the image processing constraints and the semantic constraints as the optimal positioning information.
[0133] In an optional embodiment, the device further includes:
[0134] The positioning check failure module is used to indicate that when the positioning check fails, the second geometric feature information cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning is invalid. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.
[0135] In an optional embodiment, after each positioning check is passed, the second semantic information extracted in real time is temporarily stored in a pre-built semantic information temporary buffer area; the semantic map updating module 304 includes:
[0136] A visit count acquisition module, used to acquire the visit count for the visit area in real time during the execution of the task;
[0137] An access number determination module, for obtaining the image acquisition time and light intensity of the last positioning up to the present when it is determined that the access number is greater than or equal to a preset access threshold;
[0138] The semantic map updating submodule is used to update the semantic map of the access area based on the second semantic information cached when the last positioning check passed when it is determined that the illumination intensity exceeds the preset illumination intensity range corresponding to the image acquisition time.
[0139] In an optional embodiment, after each positioning check is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-constructed point cloud information temporary buffer area; the point cloud map updating module 305 includes:
[0140] A laser point cloud map reconstruction module, configured to reconstruct the laser point cloud information using the plurality of sets of second geometric feature information cached during the execution of the task when the service robot completes the task, and to reconstruct a new laser point cloud map based on the laser point cloud information;
[0141] A cancel update module, used for canceling the update of the laser point cloud map of the access area when the map reconstruction fails;
[0142] The dense judgment module is used to judge whether the newly created laser point cloud map is dense when the map reconstruction is successful;
[0143] The point cloud map updating submodule is used to compare the newly created laser point cloud map with the original laser point cloud map, and based on the comparison result, supplement or delete the original laser point cloud map to update the laser point cloud map of the visited area.
[0144] In an optional embodiment, the device further includes:
[0145] An information collection module, used to collect laser point cloud data of the access area and surrounding environment images through information collection equipment;
[0146] A laser point cloud map construction module, used to construct a laser point cloud map of the visited area using the laser point cloud data;
[0147] A point cloud segmentation and clustering module is used to extract the first semantic information of each identifiable target in the access area from the surrounding environment image, and first segment and then cluster the laser point cloud map according to each of the first semantic information to obtain laser point clusters corresponding to each of the identifiable targets;
[0148] The offline three-dimensional reconstruction module is used to perform offline three-dimensional reconstruction based on each group of laser point clusters, and finally combine them into a three-dimensional point cloud model of the access area.
[0149] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.
[0150] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:
[0151] The memory is used to store the program code and transmit the program code to the processor;
[0152] The processor is used to execute the map positioning and maintenance method of any embodiment of the present invention according to the instructions in the program code.
[0153] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the map positioning and maintenance method of any embodiment of the present invention.
[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0155] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0157] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A map positioning and maintenance method, characterized in that: Applied to a service robot, the method comprises: Get the original location information of the visited area; Collecting a real-time environment image of the visited area, performing target detection on the real-time environment image to extract real-time positioning information, and performing positioning verification based on the original positioning information and the real-time positioning information; When the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information; When a preset update condition is met, updating the semantic map of the visited area in combination with the real-time positioning information; When the service robot completes the task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information.
2. The map positioning and maintenance method according to claim 1, characterized in that: The original positioning information includes a laser point cloud map and a three-dimensional point cloud model of the visited area, and the target detection is performed on the real-time environment image to extract the real-time positioning information, and the positioning verification is performed according to the original positioning information and the real-time positioning information, including: Extracting first geometric feature information and first semantic information of each identifiable target in the access area from the three-dimensional point cloud model; Recognize and segment the real-time environment image through target detection based on a deep learning algorithm, and extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area; Performing affine transformation on each of the second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information having the same semantic information in the laser point cloud map, thereby obtaining initial position information of the service robot; Projecting each of the second geometric feature information onto the laser point cloud map according to the initial pose information; Based on the feature correspondence, the first geometric feature information and the second geometric feature information are matched to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.
3. The map positioning and maintenance method according to claim 2, characterized in that: When the positioning check passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output optimal positioning information, including: When the positioning test is passed, it indicates that each of the second geometric feature information can be matched to the corresponding first geometric feature information; Constructing geometric constraints, image processing constraints and semantic constraints, and combining the geometric constraints, the image processing constraints and the semantic constraints to construct a joint optimization function; The joint optimization function is used to iteratively optimize the initial posture information of the service robot through the least squares method to calculate the optimal posture that satisfies the geometric constraints, the image processing constraints and the semantic constraints at the same time as the optimal positioning information.
4. The map positioning and maintenance method according to claim 2, characterized in that: Also includes: If the positioning check fails, it means that the second geometric feature information cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning is invalid. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.
5. The map positioning and maintenance method according to claim 2, characterized in that: After each positioning check is passed, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary cache area; when the preset update condition is met, the semantic map of the access area is updated in combination with the real-time positioning information, including: During the task execution, the number of visits to the visit area is obtained in real time; When it is determined that the number of accesses is greater than or equal to a preset access threshold, the image acquisition time and the light intensity at the last positioning up to now are obtained; When it is determined that the illumination intensity exceeds the preset illumination intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning check passed.
6. The map positioning and maintenance method according to claim 2, characterized in that: After each positioning check is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-built temporary cache area of point cloud information; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information, including: When the service robot completes the task, the laser point cloud information is reconstructed using the multiple sets of second geometric feature information cached during the task execution, and a new laser point cloud map is reconstructed based on the laser point cloud information; If the map reconstruction fails, canceling the updating of the laser point cloud map of the visited area; If the map reconstruction is successful, determine whether the newly created laser point cloud map is dense; If so, the newly created laser point cloud map is compared with the original laser point cloud map, and based on the comparison result, the original laser point cloud map is supplemented or deleted to update the laser point cloud map of the visited area.
7. The map positioning and maintenance method according to any one of claims 2 to 6, characterized in that: Before performing a task by the service robot, the method further includes: Collecting laser point cloud data of the access area and surrounding environment images through information collection equipment; Using the laser point cloud data, constructing a laser point cloud map of the visited area; Extracting first semantic information of each identifiable target in the access area from the surrounding environment image, and first segmenting and then clustering the laser point cloud map according to each of the first semantic information to obtain laser point clusters corresponding to each of the identifiable targets; Offline three-dimensional reconstruction is performed based on each group of laser point clusters, and finally combined into a three-dimensional point cloud model of the access area.
8. A map positioning and maintenance device, characterized in that: Applied to a service robot, the device comprises: An information acquisition module is used to obtain the original positioning information of the access area; A positioning verification module, used to collect a real-time environment image of the visited area, perform target detection on the real-time environment image to extract real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information; An information output module is used to construct a joint optimization function in combination with multi-condition positioning constraints when the positioning test passes, and iteratively optimize the real-time positioning information based on the joint optimization function to output the optimal positioning information; A semantic map updating module, configured to update the semantic map of the visited area in combination with the real-time positioning information when a preset update condition is met; The point cloud map updating module is used to update the laser point cloud map of the visited area offline in combination with the real-time positioning information when the service robot completes the task.
9. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the map positioning and maintenance method according to any one of claims 1-7 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the map positioning and maintenance method described in any one of claims 1-7.
Citation Information
Patent Citations
A mobile robot positioning method based on an improved ORB algorithm
CN109903338A
Deep learning-based method for constructing three-dimensional semantic map of indoor environment
CN110243370A
Autonomous positioning method for unmanned vehicle in long-term scene
CN112925322A
Mobile robot extension mapping method and device based on grid map and medium
CN114326699A
Semantic information-based positioning method and device and terminal equipment
CN115131363A
