Map positioning and maintenance method, device, electronic device and storage medium

Abstract

The present invention discloses a method, device, electronic device and storage medium for map positioning and maintenance, which are used to solve the technical problems of poor positioning effect in existing robot-related positioning methods and the failure to consider map maintenance. The method includes: obtaining the original positioning information of the accessed area; collecting the real-time environmental images of the accessed area, performing target detection on the real-time environmental images to extract the real-time positioning information, and performing positioning verification according to the original positioning information and the real-time positioning information; when the positioning verification passes, constructing a joint optimization function in combination with multi-condition positioning constraints, and iteratively optimizing the real-time positioning information based on the joint optimization function to output more accurate and effective positioning information; when the preset update condition is satisfied, updating the semantic map of the accessed area in combination with the real-time positioning information; when the service robot completes the task, offline updating the laser point cloud map of the accessed area in combination with the real-time positioning information to realize the maintenance of the existing map.

Description

Technical Field

[0001] The present invention relates to the technical field of laser point cloud, and in particular to a map positioning and maintenance method, device, electronic device and storage medium. Background Art

[0002] In recent years, laser point clouds and point cloud maps have been widely used as the main sensing information sources for positioning in low-speed unmanned products. Taking the low-speed unmanned robot in a service robot as an example, currently, the 6-degree-of-freedom positioning information of the robot is mainly obtained by registering the laser point cloud frame with the original point cloud map. Among them, the point cloud registration methods mainly include ICP (Iterative Closest Point, an algorithm widely used in point cloud registration), NICP (Non-Rigid Iterative Closest Point, an algorithm for non-rigid registration), and NDT (Normal Distributions Transform, an algorithm for point cloud registration), etc. These methods are all based on the geometric constraints of the point cloud, and the optimal solution is solved by the least squares method to calculate the 6-degree-of-freedom positioning information.

[0003] For a robot, the 6-degree-of-freedom positioning information is usually also called pose, that is, position and attitude. Usually, the above-mentioned several registration methods can meet the required timeliness and positioning accuracy of the robot. However, in complex or degraded scenarios, these geometric constraint-based positioning methods are prone to failure problems, resulting in poor positioning effects. For example, places with dense crowds, or parks lacking geometric features. In actual situations, places with sufficient and stable geometric constraints are rare in real usage scenarios, and instead, other adverse scenarios account for the majority. For example, a sweeper is applied in parks, sidewalks, industrial parks, etc. And most of the current positioning methods do not consider map maintenance either. Summary of the Invention

[0004] The present invention provides a map positioning and maintenance method, device, electronic device and storage medium, which are used to solve or partially solve the technical problems of poor positioning effects and lack of consideration for map maintenance in existing robot-related positioning methods.

[0005] The present invention provides a map positioning and maintenance method, which is applied to a service robot. The method includes:

[0006] Obtain the original positioning information of the access area;

[0007] Collect the real-time environmental images of the access area, perform target detection on the real-time environmental images to extract the real-time positioning information, and perform positioning verification according to the original positioning information and the real-time positioning information;

[0008] When the positioning inspection passes, a joint optimization function is constructed by combining multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output optimal positioning information;

[0009] When a preset update condition is satisfied, the semantic map of the access area is updated in combination with the real-time positioning information;

[0010] When the service robot completes a task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information.

[0011] Optionally, the original positioning information includes the laser point cloud map and the three-dimensional point cloud model of the access area. The target detection of the real-time environmental image is performed to extract real-time positioning information, and the positioning inspection based on the original positioning information and the real-time positioning information includes:

[0012] Extract the first geometric feature information and the first semantic information of each recognizable target in the access area from the three-dimensional point cloud model;

[0013] The real-time environmental image is recognized and segmented by target detection based on a deep learning algorithm to extract the second geometric feature information and the second semantic information of at least one currently recognizable target in the access area;

[0014] An affine transformation is performed on each of the second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map to obtain the initial pose information of the service robot;

[0015] According to the initial pose information, each of the second geometric feature information is projected onto the laser point cloud map;

[0016] Based on the feature correspondence relationship, the first geometric feature information and the second geometric feature information are matched to perform a positioning inspection on the recognizable target corresponding to the first geometric feature information and the currently recognizable target corresponding to the second geometric feature information.

[0017] Optionally, the step of, when the positioning inspection passes, constructing a joint optimization function by combining multi-condition positioning constraints and iteratively optimizing the real-time positioning information based on the joint optimization function to output optimal positioning information includes:

[0018] When the positioning inspection passes, it indicates that each of the second geometric feature information can be matched to the corresponding first geometric feature information;

[0019] Construct geometric constraints, image processing constraints, and semantic constraints, and jointly construct a joint optimization function by combining the geometric constraints, the image processing constraints, and the semantic constraints;

[0020] Use the joint optimization function to iteratively optimize the initial pose information of the service robot by the least squares method to calculate the optimal pose that simultaneously satisfies the geometric constraints, the image processing constraints, and the semantic constraints as the optimal positioning information.

[0021] Optionally, the method further includes:

[0022] If the positioning check fails, it indicates that there is a second geometric feature information that cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning fails. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.

[0023] Optionally, after each positioning check passes, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary buffer; when the preset update condition is met, the semantic map of the access area is updated in combination with the real-time positioning information, including:

[0024] During the task execution process, the access times for the access area are obtained in real time;

[0025] When it is determined that the access times are greater than or equal to the preset access threshold, the image acquisition time and the light intensity at the last positioning up to now are obtained;

[0026] When it is determined that the light intensity exceeds the preset light intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning check passed.

[0027] Optionally, after each positioning check passes, the second geometric feature information extracted in real time is temporarily stored in a pre-constructed point cloud information temporary buffer; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information, including:

[0028] When the service robot completes the task, the laser point cloud information is reconstructed using multiple groups of second geometric feature information cached during the current task execution, and a new laser point cloud map is reconstructed based on the laser point cloud information;

[0029] If the map reconstruction fails, the update of the laser point cloud map of the access area is cancelled;

[0030] If the map reconstruction is successful, it is determined whether the newly created laser point cloud map is dense;

[0031] If so, compare the newly created lidar point cloud map with the original lidar point cloud map, and based on the comparison result, supplement or delete the original lidar point cloud map to update the lidar point cloud map of the access area.

[0032] Optionally, before the service robot executes a task, the method further includes:

[0033] Collect the lidar point cloud data of the access area and the surrounding environment images through an information collection device;

[0034] Use the lidar point cloud data to construct a lidar point cloud map of the access area;

[0035] Extract the first semantic information of each recognizable target in the access area from the surrounding environment images, and first segment and then cluster the lidar point cloud map according to each piece of the first semantic information to obtain lidar point clusters corresponding to each recognizable target;

[0036] Perform offline 3D reconstruction based on each group of the lidar point clusters respectively, and finally combine them into a 3D point cloud model of the access area.

[0037] The present invention also provides a map positioning and maintenance device, which is applied to a service robot. The device includes:

[0038] An information acquisition module, configured to acquire the original positioning information of the access area;

[0039] A positioning inspection module, configured to collect the real-time environment images of the access area, perform target detection on the real-time environment images to extract real-time positioning information, and perform positioning inspection according to the original positioning information and the real-time positioning information;

[0040] An information output module, configured to, when the positioning inspection passes, construct a joint optimization function by combining multi-condition positioning constraints, and iteratively optimize the real-time positioning information based on the joint optimization function, and output the optimal positioning information;

[0041] A semantic map update module, configured to update the semantic map of the access area by combining the real-time positioning information when a preset update condition is satisfied;

[0042] A point cloud map update module, configured to offline update the lidar point cloud map of the access area by combining the real-time positioning information when the service robot completes the task.

[0043] The present invention also provides an electronic device, which includes a processor and a memory:

[0044] The memory is used to store program code and transfer the program code to the processor;

[0045] The processor is used to execute the map positioning and maintenance method described in any one of the above according to the instructions in the program code.

[0046] The present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the map positioning and maintenance method described in any one of the above.

[0047] It can be seen from the above technical solutions that the present invention has the following advantages:

[0048] A map positioning and maintenance method applied to a service robot is provided. First, obtain the original positioning information of the access area; then collect the real-time environment image of the access area, perform object detection on the real-time environment image to extract the real-time positioning information, and perform positioning verification according to the original positioning information and the real-time positioning information; when the positioning verification passes, construct a joint optimization function in combination with multi-condition positioning constraints, and perform iterative optimization on the real-time positioning information based on the joint optimization function to output the optimal positioning information. Thus, through positioning verification and the joint optimization function based on multi-condition positioning constraints, more accurate and effective positioning information can be obtained. When the preset update condition is satisfied, update the semantic map of the access area in combination with the real-time positioning information; when the service robot completes the task, offline update the laser point cloud map of the access area in combination with the real-time positioning information. Thus, considering the maintenance of both the laser point cloud map and the semantic map, different update mechanisms are set respectively, and the existing maps are updated and maintained in combination with the real-time collected information. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 It is a flowchart of the steps of a map positioning and maintenance method;

[0051] Figure 2 It is a schematic diagram of the overall process of a map positioning and maintenance method;

[0052] Figure 3 It is a structural block diagram of a map positioning and maintenance device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] Embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for map positioning and maintenance, which are used to solve or partially solve the technical problems of poor positioning effect and lack of consideration for map maintenance in existing robot-related positioning methods.

[0054] To make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the following described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0055] As an example, taking a low-speed driverless robot in a service robot as an example, currently, the 6-degree-of-freedom positioning information of the robot is mainly obtained by registering the laser point cloud frame with the original point cloud map. Among them, the point cloud registration methods mainly include ICP, NICP, and NDT, etc. These methods are all based on the geometric constraints of the point cloud and solve the optimal solution through the least squares method to calculate the 6-degree-of-freedom positioning information.

[0056] Generally, the above-mentioned several registration methods can meet the timeliness and positioning accuracy required by the robot. However, in complex or degraded scenarios, these geometric constraint-based positioning methods are prone to failure problems, resulting in poor positioning effects. For example, places with dense crowds, or parks lacking geometric features, etc. In actual situations, places with sufficient and stable geometric constraints are not common in real usage scenarios, but other adverse scenarios account for the majority. For example, a cleaning vehicle is applied in parks, sidewalks, industrial parks, etc. And most of the currently adopted positioning methods do not consider maintaining the map either.

[0057] A high-precision semantic map refers to a three-dimensional map composed of high-precision laser point clouds, in which the target categories in the visual pictures are assigned to the corresponding laser point clusters to form labels through means such as deep learning. Object detection based on deep learning includes category detection and corresponding picture area segmentation. The segmented pixels can be projected onto the map through geometric relationships. Currently, no relevant literature has proposed means of adopting a high-precision semantic map in a low-speed driverless robot and maintaining the existing map.

[0058] Therefore, one of the core inventive points of the embodiments of the present invention lies in: for the robot positioning scenario, providing a map positioning and maintenance method applied to service robots. On the one hand, in service robots (such as low-speed driverless robots), a semantic map based on high-precision laser point clouds is used, so as to increase the information redundancy of the laser point clouds during positioning through semantic information; the recognizable targets in the surrounding environment are detected in real time, the positioning is calculated by combining the target category and the point cloud, and through positioning verification and joint iterative optimization, visual verification is increased in the laser positioning scenario, and stable positioning is provided as a backup in the laser positioning failure scenario, thereby further improving the stability and accuracy of positioning and avoiding the problem of laser point cloud positioning failure. On the other hand, the existing point cloud map and semantic information are maintained through the laser and visual data collected by the service robot each time it performs a task, improving the stability and accuracy of robot positioning, expanding the application scenarios, and filling the gap in the current robot positioning method that does not consider maintaining the existing map.

[0059] Specifically, first, obtain the original positioning information of the access area; then collect the real-time environment image of the access area, perform target detection on the real-time environment image to extract the real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information; when the positioning verification passes, construct a joint optimization function in combination with multi-condition positioning constraints, and perform iterative optimization on the real-time positioning information based on the joint optimization function to output the optimal positioning information. Thus, through positioning verification and the joint optimization function based on multi-condition positioning constraints, more accurate and effective positioning information can be obtained. When the preset update condition is met, update the semantic map of the access area in combination with the real-time positioning information; when the service robot completes the task, offline update the laser point cloud map of the access area in combination with the real-time positioning information. Thus, considering the maintenance of both the laser point cloud map and the semantic map, different update mechanisms are set respectively, and the existing map is updated and maintained in combination with the real-time collected information.

[0060] Refer to Figure 1 , which shows the step flowchart of a map positioning and maintenance method provided by the embodiments of the present invention. The method is applied to a service robot, and the method may specifically include the following steps:

[0061] Step 101, obtain the original positioning information of the access area;

[0062] In an embodiment of the present invention, a low-speed driverless robot in a service robot is taken as an example for illustration. For a certain area (hereinafter all referred to as the access area), before a low-speed driverless robot needs to perform related tasks in the access area, surrounding environment images can be collected through information collection devices such as cameras and mobile terminals, and an initial laser point cloud map and a three-dimensional point cloud model of the access area can be constructed based on the laser point cloud technology in combination with the image processing technology, so that in the subsequent process, positioning verification can be performed based on the pre-constructed laser point cloud map and three-dimensional point cloud model, in combination with the positioning information obtained by real-time processing during the task execution process, to avoid the problem of positioning failure.

[0063] In a specific implementation, before a low-speed driverless robot executes a task, laser point cloud data of the access area can be collected through information collection devices such as cameras and mobile terminals. While collecting the laser point cloud data, surrounding environment images can be sampled at the same time for extracting relevant semantic information. Then, the laser point cloud data is used to construct a laser point cloud map of the access area.

[0064] After the construction of the high-precision laser point cloud map is completed, the first semantic information of each recognizable target (such as buildings, road signs, railings, etc.) in the access area is extracted from the surrounding environment images (in order to distinguish from the semantic information obtained during subsequent real-time positioning, the semantic information corresponding to this process is defined as the first semantic information, and the semantic information corresponding to the subsequent real-time positioning process is defined as the second semantic information).

[0065] Then, according to each first semantic information, the laser point cloud map is first segmented and then clustered to obtain laser point clusters corresponding to each recognizable target. Then, offline three-dimensional reconstruction is performed based on each group of laser point clusters respectively, and finally, they are combined into a three-dimensional point cloud model of the access area.

[0066] In the subsequent processing flow, based on the three-dimensional point cloud model, effective first geometric feature information can be extracted (similarly, in order to distinguish from the geometric feature information obtained during subsequent real-time positioning, the geometric feature information corresponding to this process is defined as the first geometric feature information, and the geometric feature information corresponding to the subsequent real-time positioning process is defined as the second geometric feature information). Such as plane information, plane intersection line information, describable surface information, etc.

[0067] Based on the three-dimensional point cloud model, the first semantic information can also be extracted, which can also be understood as typical indication information. Such as indication arrows on the road surface, traffic signs, LOGO, etc. These two types of information, namely the first geometric feature information and the first semantic information, will be used in the subsequent positioning and verification processes.

[0068] Step 102: Collect the real-time environmental image of the access area, perform object detection on the real-time environmental image to extract real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information.

[0069] It can be understood that although the collection of the environmental image of the access area is involved in both Step 101 and Step 102, the environmental information collected in Step 101 is equivalent to the environmental information before the robot actually works. And this information is collected once by the collection device and is mainly used to construct a laser point cloud map. The environmental image in Step 102 is the environmental information collected in real time when the robot is performing tasks. In actual situations, due to possible environmental changes, the environmental information collected in real time may be different from the environmental information collected once in the previous Step 101. For example, within the time period from the first collection to the current collection, a road sign is removed, or a new railing is added, etc. By comparing the information collected in real time with the information collected once, it can be determined whether the map has changed.

[0070] Combined with the previous discussion, based on the construction of the laser point cloud, the original positioning information mainly can include the laser point cloud map of the access area and the three-dimensional point cloud model. Specifically, the process of performing object detection on the real-time environmental image to extract real-time positioning information and performing positioning verification based on the original positioning information and the real-time positioning information can be achieved by executing the following sub-steps S01 to S05:

[0071] Step S01: Extract the first geometric feature information and the first semantic information of each recognizable object in the access area from the three-dimensional point cloud model.

[0072] Step S02: Perform recognition and segmentation on the real-time environmental image through object detection based on a deep learning algorithm, and extract the second geometric feature information and the second semantic information of at least one currently recognizable object in the access area.

[0073] Step S03: Perform an affine transformation on each second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map, and obtain the initial pose information of the service robot.

[0074] Step S04: Project each second geometric feature information onto the laser point cloud map according to the initial pose information.

[0075] Step S05: Based on the feature correspondence relationship, match the first geometric feature information and the second geometric feature information to perform positioning verification on the recognizable object corresponding to the first geometric feature information and the currently recognizable object corresponding to the second geometric feature information.

[0076] Step 103: When the positioning verification passes, construct a joint optimization function by combining multi-condition positioning constraints, and iteratively optimize the real-time positioning information based on the joint optimization function to output the optimal positioning information;

[0077] In the previous Step 102, the main description was about the process of calculating the pose of a low-speed unmanned robot once. Through the mutual verification between the original acquisition information and the real-time acquisition information by laser point cloud positioning, it can be judged whether the laser positioning fails. When both the laser positioning result and the semantic positioning result pass the verification, or even if there are slight errors but the difference between them meets the requirements (the error is small and does not affect the overall positioning effect), the geometric constraints, image processing constraints, and semantic constraints of the laser point cloud can be combined to construct a joint optimization function to calculate the optimal pose that simultaneously meets the relevant constraint conditions.

[0078] Specifically, when the positioning verification passes, the process of constructing a joint optimization function by combining multi-condition positioning constraints and iteratively optimizing the real-time positioning information based on the joint optimization function to output the optimal positioning information can be implemented by performing the following sub-steps S11 to S13:

[0079] Step S11: When the positioning verification passes, it means that each second geometric feature information can be matched to the corresponding first geometric feature information;

[0080] Step S12: Construct geometric constraints, image processing constraints, and semantic constraints, and combine the geometric constraints, image processing constraints, and semantic constraints to construct a joint optimization function;

[0081] Furthermore, the geometric constraints mainly can include plane constraint (Plane Constraint), intersection line constraint (Intersection Line Constraint), and surface constraint (Surface Constraint). The image processing constraints mainly include RGB constraint (Red-Green-Blue, a color model) (RGB Constraint) and grayscale constraint (Grayscale Constraint).

[0082] Among them, the plane constraint means restricting one or more elements (such as points, lines, planes) to be located on a specific plane. The intersection line constraint means the constraint of the line formed at the intersection of two planes. The surface constraint involves ensuring the relationships between surfaces in surface modeling, such as tangency, smooth transition, alignment, etc.

[0083] RGB constraints refer to imposing certain conditions or restrictions on the values of the red (R), green (G), and blue (B) color channels of pixels when processing color images. Grayscale constraints are similar to RGB constraints, but are for grayscale images, that is, images with only luminance values and no color information.

[0084] Semantic Constraints refer to restricting or guiding the behavior of localization, recognition, or understanding algorithms based on semantic information in the environment (such as the category, function, and meaning of objects, etc.).

[0085] The above several constraint conditions are conventional settings, and the specific setting situations are not the key points emphasized by the technical solution of the present invention. Those skilled in the art can set them flexibly according to the actual situation, and the present invention does not limit this.

[0086] After constructing the above several constraints, a joint optimization function can be further constructed. Thus, by simultaneously considering the cost functions of geometric constraints, image processing constraints, and semantic constraints for optimization calculation, more accurate and effective localization information can be obtained.

[0087] Step S13: Adopt the joint optimization function to iteratively optimize the initial pose information of the service robot by the least squares method, so as to calculate the optimal pose that simultaneously satisfies geometric constraints, image processing constraints, and semantic constraints as the optimal localization information.

[0088] Finally, the joint optimization function can be adopted to iteratively optimize the initial pose information of the service robot (that is, equivalent to iterating the initial pose of the robot) by the least squares method, so as to calculate the optimal pose that simultaneously satisfies these several constraints of geometric constraints, image processing constraints, and semantic constraints as the optimal localization information.

[0089] In another case, a situation where the localization check fails (i.e., the check fails) may occur. When the localization check fails, it indicates that there is a second geometric feature information in the real-time localization information that cannot be matched to the corresponding first geometric feature information, and the laser point cloud localization fails. In this case, there is a problem of inaccurate real-time localization information. It can be known that the first semantic information in the map is collected in advance, so there may be a deviation from the second semantic information detected in real time currently. For example, other semantic information is increased or decreased, or the information is incomplete due to different perspectives, etc. In order to obtain stable localization information in the laser localization failure scenario, at this time, the geometric feature information and semantic information in the originally collected localization information can be used as the localization result output for this localization. Specifically, each first geometric feature information can be used as the target geometric feature information for this localization, and each first semantic information can be used as the target semantic information for this localization.

[0090] Step 104: When the preset update condition is satisfied, update the semantic map of the access area in combination with the real-time positioning information.

[0091] Before each task execution of the low-speed driverless robot, a temporary buffer area for semantic information (a temporary semantic information repository) can be constructed first. After each qualified semantic positioning (i.e., each positioning inspection passes), the second semantic information extracted in real time is temporarily stored in this temporary buffer area for semantic information.

[0092] Furthermore, for the update and maintenance of the semantic map, a semantic information update mechanism can be preset to update the semantic information in the map when the update condition is met. Among them, the specific update content mainly can include: RGB color, grayscale, contour, indication information, and the positioning stability program of the information.

[0093] Specifically, during the task execution of the robot, the same area may be visited multiple times, and the semantic information of this area is collected multiple times. For the accessed area, during one task execution, each time a successful positioning inspection is completed, it is counted once, and the corresponding second semantic information is cached in the temporary buffer area for semantic information. Then, the current cumulative number of visits is judged in real time. When the number of visits to this accessed area is less than a certain threshold (such as less than 5 times or 10 times, the access number threshold can be set according to the actual situation, and the present invention does not limit this), the semantic information of this accessed area is not updated. When the cumulative number reaches the above preset access number threshold (such as reaching 5 times or 10 times), the current semantic map is updated based on the second semantic information cached after the last positioning inspection passes.

[0094] Both the RGB and grayscale information in the semantic information are related to the light intensity at that time. When the light is strong, the grayscale intensity is high, and the RGB is also closer to the true value. Therefore, when updating the RGB and grayscale values, it is necessary to combine the time and light intensity when collecting the image to construct a judgment criterion to determine whether to update. For example, when it is judged that the light intensity is within a certain range, no update is performed, and when it is outside this range, it is determined to update. The indication information in the semantic information is very stable, so information can be directly supplemented or added on the basis of the existing information. The contour information in the semantic information is the key to constructing geometric constraints, and the optimal boundary can be fitted multiple times by geometric methods for supplementation during the update.

[0095] Among them, fitting the optimal boundary by geometric methods is a common task in computer vision, image processing, and machine learning, mainly used to identify and segment the object boundaries in images. Achieving this goal usually depends on image feature extraction, edge detection, curve fitting, and optimization techniques. Since this processing flow is an existing conventional means and is not the key content to be discussed in the technical solution of the present invention, it will not be elaborated here.

[0096] In a specific implementation, when a preset update condition is met, the semantic map of the access area is updated in combination with real-time positioning information, which can be: during the execution of a task, the access times for the access area are obtained in real time; when it is determined that the access times are greater than or equal to a preset access threshold, the image acquisition time and the light intensity at the last positioning up to now are obtained; when it is determined that the light intensity exceeds the preset light intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning passed the inspection.

[0097] By combining the access times for the accessed area during the execution of a task, it is determined in real time whether the environmental information of the current positioning has changed. According to the judgment result, it is further determined whether to update the semantic information (semantic map). When the change condition is met (such as determining that the current environmental information has changed by combining the acquisition time and the light intensity), the information is updated. When it is not met, the semantic information is not updated temporarily. Thus, the real-time update and maintenance of the semantic map are realized.

[0098] Step 105, when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information.

[0099] Similarly to the previous case, before each start of the execution of a task by a low-speed driverless robot, a point cloud information temporary buffer (temporary laser point cloud information warehouse) can also be constructed first. Thus, even if laser point cloud positioning may fail, when the positioning is normal (i.e., each positioning inspection passes), the second geometric feature information collected in real time can be temporarily stored in this point cloud information temporary buffer.

[0100] During the execution of a task by the robot, the moving trajectory is generally relatively stable. There are obvious differences compared with when collecting the map. Therefore, considering this actual scenario, the laser point cloud collected by the robot each time it executes a task is not updated to the map in real time. Instead, the offline update process is run when the robot returns to the stationary point after completing the task.

[0101] Specifically, when the robot completes the task, the laser point cloud collected during the execution of the task is first used to execute the laser mapping process to construct a new laser point cloud map. However, in actual situations, there may be cases where the construction fails due to insufficient laser point cloud data collected or other abnormal situations. When the construction cannot be completed, the current update is cancelled. After successfully constructing the map, it is also necessary to first evaluate whether the map is dense enough (such as judging whether the amount of information is sufficient, or whether the feature coverage rate is high enough, etc.), otherwise it cannot be compared and merged with the original map.

[0102] In actual operation, when collecting a high-precision laser point cloud map, the map is divided into multiple local regions (this process can be combined with manual judgment to ensure the rationality of the region division). At this time, the constructed updated map can be imported into the copied local regions, and these local regions are further divided into three-dimensional grids. Then, the original map is compared with the updated map to evaluate whether there are missing point clouds in each three-dimensional grid. Finally, the updated map is used to supplement or delete the three-dimensional grid.

[0103] In a specific implementation, when the service robot completes a task, the laser point cloud map of the access area can be updated offline by combining real-time positioning information, as follows: when the service robot completes a task, multiple groups of second geometric feature information cached during the execution of the task are used to reconstruct the laser point cloud information, and a new laser point cloud map is reconstructed based on the laser point cloud information; if the map reconstruction fails, the update of the laser point cloud map of the access area is cancelled; if the map reconstruction is successful, it is determined whether the newly built laser point cloud map is dense; if so, the newly built laser point cloud map is compared with the original laser point cloud map, and based on the comparison result, the original laser point cloud map is supplemented or deleted to update the laser point cloud map of the access area.

[0104] It should be noted that in order to enable those skilled in the art to better distinguish data of the same type but with different actual pointing meanings, in the embodiments of the present invention, some technical features are distinguished and described using first and second. First and second are only used for data distinction and have no other special meanings. It can be understood that the present invention does not limit this.

[0105] In the embodiments of the present invention, a map positioning and maintenance method applied to a service robot is provided. First, the original positioning information of the access area is obtained; then, the real-time environmental image of the access area is collected, and object detection is performed on the real-time environmental image to extract the real-time positioning information, and positioning verification is performed based on the original positioning information and the real-time positioning information; when the positioning verification passes, a joint optimization function is constructed by combining multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information. Thus, through the positioning verification and the joint optimization function based on multi-condition positioning constraints, more accurate and effective positioning information can be obtained. When the preset update condition is met, the semantic map of the access area is updated by combining the real-time positioning information; when the service robot completes a task, the laser point cloud map of the access area is updated offline by combining the real-time positioning information. Thus, considering the maintenance of both the laser point cloud map and the semantic map, different update mechanisms are set respectively, and the existing map is updated and maintained by combining the real-time collected information.

[0106] For better illustration, refer to Figure 2, which shows a schematic diagram of the overall process of a map positioning and maintenance method provided by an embodiment of the present invention. It should be noted that this embodiment only briefly describes the general process of map positioning and maintenance. The specific implementation process of each step can be understood by referring to the relevant content in the foregoing embodiments, and will not be elaborated here. It can be understood that the present invention places no restrictions thereon.

[0107] Collect the laser point cloud data of the access area and the surrounding environment images, and further construct the laser point cloud map and the three-dimensional point cloud model of the access area for laser point cloud positioning.

[0108] Extract the first geometric feature information and the first semantic information of each recognizable target in the access area from the three-dimensional point cloud model.

[0109] Collect the real-time environment images of the access area, perform target detection on the real-time environment images to extract the second geometric feature information and the second semantic information of at least one currently recognizable target in the access area.

[0110] Perform an affine transformation on each second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map, and obtain the initial pose information of the service robot.

[0111] Project each second geometric feature information onto the laser point cloud map according to the initial pose information.

[0112] Based on the feature correspondence relationship, match the first geometric feature information and the second geometric feature information to perform a positioning check on the recognizable target corresponding to the first geometric feature information and the currently recognizable target corresponding to the second geometric feature information.

[0113] When the positioning check passes, construct a joint optimization function in combination with multi-condition positioning constraints, and iteratively optimize the real-time positioning information based on the joint optimization function, and output the optimal positioning information.

[0114] After each positioning check passes, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary buffer area, and the second geometric feature information extracted in real time is temporarily stored in a pre-constructed point cloud information temporary buffer area.

[0115] When the preset update condition is met, update the semantic map of the access area in combination with the real-time positioning information.

[0116] When the service robot completes the task, update the laser point cloud map of the access area offline in combination with the real-time positioning information.

[0117] Refer to Figure 3, which shows a structural block diagram of a map positioning and maintenance device provided by an embodiment of the present invention. The device is applied to a service robot and specifically may include:

[0118] An information acquisition module 301, configured to acquire the original positioning information of the access area;

[0119] A positioning verification module 302, configured to collect the real-time environmental image of the access area, perform target detection on the real-time environmental image to extract the real-time positioning information, and perform positioning verification according to the original positioning information and the real-time positioning information;

[0120] An information output module 303, configured to, when the positioning verification passes, construct a joint optimization function by combining multi-condition positioning constraints, and iteratively optimize the real-time positioning information based on the joint optimization function, and output the optimal positioning information;

[0121] A semantic map update module 304, configured to, when a preset update condition is satisfied, update the semantic map of the access area by combining the real-time positioning information;

[0122] A point cloud map update module 305, configured to, when the service robot completes the task, offline update the laser point cloud map of the access area by combining the real-time positioning information.

[0123] In an optional embodiment, the original positioning information includes the laser point cloud map and the three-dimensional point cloud model of the access area, and the positioning verification module 302 includes:

[0124] An information extraction module, configured to extract the first geometric feature information and the first semantic information of each recognizable target in the access area from the three-dimensional point cloud model;

[0125] A recognition and segmentation module, configured to perform recognition and segmentation on the real-time environmental image through target detection based on a deep learning algorithm, and extract the second geometric feature information and the second semantic information of at least one currently recognizable target in the access area;

[0126] An affine transformation module, configured to perform an affine transformation on each of the second semantic information, so that the transformed second semantic information corresponds to and matches the first semantic information with the same semantic information in the laser point cloud map, and obtain the initial pose information of the service robot;

[0127] An information projection module, configured to project each of the second geometric feature information onto the laser point cloud map according to the initial pose information;

[0128] A feature information matching module, which is used to match the first geometric feature information and the second geometric feature information based on the feature correspondence relationship, so as to perform positioning inspection on the recognizable target corresponding to the first geometric feature information and the current recognizable target corresponding to the second geometric feature information.

[0129] In an optional embodiment, the information output module 303 includes:

[0130] A positioning inspection passing module, which is used to represent that each of the second geometric feature information can match the corresponding first geometric feature information when the positioning inspection passes;

[0131] A joint optimization function construction module, which is used to construct geometric constraints, image processing constraints, and semantic constraints, and jointly construct a joint optimization function with the geometric constraints, the image processing constraints, and the semantic constraints;

[0132] An iterative optimization module, which is used to adopt the joint optimization function and iteratively optimize the initial pose information of the service robot by the least squares method to calculate the optimal pose that simultaneously satisfies the geometric constraints, the image processing constraints, and the semantic constraints as the optimal positioning information.

[0133] In an optional embodiment, the device further includes:

[0134] A positioning inspection failure module, which is used to represent that there is second geometric feature information that cannot match the corresponding first geometric feature information and the laser point cloud positioning fails when the positioning inspection fails. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.

[0135] In an optional embodiment, after each positioning inspection passes, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary buffer; the semantic map update module 304 includes:

[0136] An access times acquisition module, which is used to acquire the access times for the access area in real time during the task execution process;

[0137] An access times judgment module, which is used to acquire the image acquisition time and the light intensity at the last positioning so far when it is judged that the access times is greater than or equal to a preset access threshold;

[0138] A semantic map update sub-module, which is used to update the semantic map of the access area based on the second semantic information cached when the last positioning inspection passes when it is judged that the light intensity exceeds the preset light intensity range corresponding to the image acquisition time.

[0139] In an alternative embodiment, after each positioning inspection is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-constructed temporary buffer for point cloud information; the point cloud map update module 305 includes:

[0140] A laser point cloud map reconstruction module, configured to, when the service robot completes a task, reconstruct laser point cloud information using multiple sets of second geometric feature information cached during the execution of the task, and reconstruct a new laser point cloud map based on the laser point cloud information;

[0141] A cancellation update module, configured to cancel the update of the laser point cloud map of the accessed area when the map reconstruction fails;

[0142] A density judgment module, configured to judge whether the newly built laser point cloud map is dense when the map reconstruction is successful;

[0143] A point cloud map update sub-module, configured to compare the newly built laser point cloud map with the original laser point cloud map, and supplement or delete the original laser point cloud map based on the comparison result to update the laser point cloud map of the accessed area.

[0144] In an alternative embodiment, the device further includes:

[0145] An information acquisition module, configured to acquire laser point cloud data of the accessed area and surrounding environment images through an information acquisition device;

[0146] A laser point cloud map construction module, configured to construct a laser point cloud map of the accessed area using the laser point cloud data;

[0147] A point cloud segmentation and clustering module, configured to extract first semantic information of each recognizable target in the accessed area from the surrounding environment images, and segment and cluster the laser point cloud map according to each piece of the first semantic information to obtain laser point clusters corresponding to each of the recognizable targets;

[0148] An offline 3D reconstruction module, configured to perform offline 3D reconstruction based on each group of the laser point clusters respectively, and finally combine them into a 3D point cloud model of the accessed area.

[0149] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, please refer to the partial description of the foregoing method embodiment.

[0150] The embodiment of the present invention further provides an electronic device, the device including a processor and a memory:

[0151] The memory is used to store program code and transmit the program code to the processor;

[0152] The processor is used to execute the map positioning and maintenance method according to any embodiment of the present invention according to the instructions in the program code.

[0153] An embodiment of the present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the map positioning and maintenance method according to any embodiment of the present invention.

[0154] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0155] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0158] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A map positioning and maintenance method, characterized in that: Applied to a service robot, the method comprises: Get the original location information of the visited area; Collecting a real-time environment image of the visited area, performing target detection on the real-time environment image to extract real-time positioning information, and performing positioning verification based on the original positioning information and the real-time positioning information; When the positioning test passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output the optimal positioning information; When a preset update condition is met, updating the semantic map of the visited area in combination with the real-time positioning information; When the service robot completes the task, the laser point cloud map of the visited area is updated offline in combination with the real-time positioning information; The real-time positioning information includes the second geometric feature information of at least one currently identifiable target in the access area; after each positioning check is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-built point cloud information temporary cache area; when the service robot completes the task, the laser point cloud map of the access area is updated offline in combination with the real-time positioning information, including: When the service robot completes the task, the laser point cloud information is reconstructed using the multiple sets of second geometric feature information cached during the task execution, and a new laser point cloud map is reconstructed based on the laser point cloud information; If the map reconstruction fails, canceling the updating of the laser point cloud map of the visited area; If the map reconstruction is successful, determine whether the newly created laser point cloud map is dense; If so, the newly created laser point cloud map is compared with the original laser point cloud map, and based on the comparison result, the original laser point cloud map is supplemented or deleted to update the laser point cloud map of the visited area.

2. The map positioning and maintenance method according to claim 1, characterized in that: The original positioning information includes a laser point cloud map and a three-dimensional point cloud model of the visited area, and the target detection is performed on the real-time environment image to extract the real-time positioning information, and the positioning verification is performed according to the original positioning information and the real-time positioning information, including: Extracting first geometric feature information and first semantic information of each identifiable target in the access area from the three-dimensional point cloud model; Recognize and segment the real-time environment image through target detection based on a deep learning algorithm, and extract second geometric feature information and second semantic information of at least one currently identifiable target in the access area; Performing affine transformation on each of the second semantic information so that the transformed second semantic information corresponds to and matches the first semantic information having the same semantic information in the laser point cloud map, thereby obtaining initial position information of the service robot; Projecting each of the second geometric feature information onto the laser point cloud map according to the initial pose information; Based on the feature correspondence, the first geometric feature information and the second geometric feature information are matched to perform positioning verification on the identifiable target corresponding to the first geometric feature information and the current identifiable target corresponding to the second geometric feature information.

3. The map positioning and maintenance method according to claim 2, characterized in that: When the positioning check passes, a joint optimization function is constructed in combination with multi-condition positioning constraints, and the real-time positioning information is iteratively optimized based on the joint optimization function to output optimal positioning information, including: When the positioning test is passed, it indicates that each of the second geometric feature information can be matched to the corresponding first geometric feature information; Constructing geometric constraints, image processing constraints and semantic constraints, and combining the geometric constraints, the image processing constraints and the semantic constraints to construct a joint optimization function; The joint optimization function is used to iteratively optimize the initial posture information of the service robot through the least squares method to calculate the optimal posture that satisfies the geometric constraints, the image processing constraints and the semantic constraints at the same time as the optimal positioning information.

4. The map positioning and maintenance method according to claim 2, characterized in that: Also includes: If the positioning check fails, it means that the second geometric feature information cannot be matched to the corresponding first geometric feature information, and the laser point cloud positioning is invalid. At this time, each of the first geometric feature information is used as the target geometric feature information for this positioning, and each of the first semantic information is used as the target semantic information for this positioning.

5. The map positioning and maintenance method according to claim 2, characterized in that: After each positioning check is passed, the second semantic information extracted in real time is temporarily stored in a pre-constructed semantic information temporary cache area; when the preset update condition is met, the semantic map of the access area is updated in combination with the real-time positioning information, including: During the task execution, the number of visits to the visit area is obtained in real time; When it is determined that the number of accesses is greater than or equal to a preset access threshold, the image acquisition time and the light intensity at the last positioning up to now are obtained; When it is determined that the illumination intensity exceeds the preset illumination intensity range corresponding to the image acquisition time, the semantic map of the access area is updated based on the second semantic information cached when the last positioning check passed.

6. The map positioning and maintenance method according to any one of claims 2 to 5, characterized in that: Before performing a task by the service robot, the method further comprises: Collecting laser point cloud data of the access area and surrounding environment images through information collection equipment; Using the laser point cloud data, constructing a laser point cloud map of the visited area; Extracting first semantic information of each identifiable target in the access area from the surrounding environment image, and first segmenting and then clustering the laser point cloud map according to each of the first semantic information to obtain laser point clusters corresponding to each of the identifiable targets; Offline three-dimensional reconstruction is performed based on each group of laser point clusters, and finally combined into a three-dimensional point cloud model of the access area.

7. A map positioning and maintenance device, characterized in that: Applied to a service robot, the device comprises: An information acquisition module is used to obtain the original positioning information of the access area; A positioning verification module, used to collect a real-time environment image of the visited area, perform target detection on the real-time environment image to extract real-time positioning information, and perform positioning verification based on the original positioning information and the real-time positioning information; An information output module is used to construct a joint optimization function in combination with multi-condition positioning constraints when the positioning test passes, and iteratively optimize the real-time positioning information based on the joint optimization function to output the optimal positioning information; A semantic map updating module, configured to update the semantic map of the visited area in combination with the real-time positioning information when a preset update condition is met; A point cloud map updating module, used for offline updating the laser point cloud map of the visited area in combination with the real-time positioning information when the service robot completes the task; The real-time positioning information includes the second geometric feature information of at least one currently identifiable target in the access area; after each positioning check is passed, the second geometric feature information extracted in real time is temporarily stored in a pre-built point cloud information temporary cache area; the point cloud map update module includes: A laser point cloud map reconstruction module, configured to reconstruct the laser point cloud information using the plurality of sets of second geometric feature information cached during the execution of the task when the service robot completes the task, and to reconstruct a new laser point cloud map based on the laser point cloud information; A cancel update module, used for canceling the update of the laser point cloud map of the access area when the map reconstruction fails; The dense judgment module is used to judge whether the newly created laser point cloud map is dense when the map reconstruction is successful; The point cloud map updating submodule is used to compare the newly created laser point cloud map with the original laser point cloud map, and based on the comparison result, supplement or delete the original laser point cloud map to update the laser point cloud map of the visited area.

8. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the map positioning and maintenance method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the map positioning and maintenance method described in any one of claims 1-6.

Images