Multi-sensor based repositioning method, apparatus, device, and storage medium

Abstract

The application relates to a multi-sensor-based repositioning method, device, equipment and storage medium. The method comprises the following steps: acquiring environment images collected by respective sensors; performing global matching on the respective environment images and images in a prior environment image library to obtain respective robot poses before updating corresponding to the respective sensors; performing local matching on the respective environment images and respective surrounding environment images corresponding to the respective robot poses before updating in the prior environment image library to obtain score values of respective robot poses after updating; and determining a repositioning result of a current robot pose according to the score values of the respective robot poses after updating. By performing global matching and local matching on the environment images collected by different types of sensors and the images in the prior environment image library, the repositioning result of the current robot pose can be ensured to have accuracy and reliability.

Description

Technical Field

[0001] This application relates to the field of robot localization technology, and in particular to a relocalization method, apparatus, device and storage medium based on multiple sensors. Background Technology

[0002] With the rapid development of machine learning and perception technologies, mobile robots with autonomous navigation capabilities have entered people's daily lives at an unprecedented pace. Mobile robots primarily use pre-installed sensors such as LiDAR and cameras to perceive environmental information and their own state, performing simultaneous localization and mapping to achieve autonomous movement towards a target in obstacle-filled environments—that is, autonomous navigation. The foundation for autonomous navigation is obtaining its initial position in a priori map. However, when the mobile robot system is shut down, loses power, or is moved, its position and posture change. Upon restarting, the robot cannot pinpoint its location on the map. In this case, it needs to find its position on the map again and use this position as the initial position to restart real-time localization—this is the technology of relocalization.

[0003] In existing technologies, robot relocalization typically employs preset start-up points or relies solely on LiDAR or vision sensors. However, pre-setting start-up points may lead to errors in similar areas outside of those points, while relying on a single sensor can result in relocalization failure due to the sensor's susceptibility to environmental conditions such as lighting, weather, and season. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-sensor-based repositioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can ensure the accuracy and reliability of the repositioning result of the robot's current pose, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a multi-sensor-based relocalization method applied to a robot, wherein the robot includes at least two different types of sensors. The method includes:

[0006] Acquire environmental images obtained by each sensor separately;

[0007] Each environmental image is globally matched with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0008] Local matching is performed between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update, to obtain the score value of each robot pose after the update.

[0009] Based on the scores of each updated robot pose, the relocalization result of the robot's current pose is determined.

[0010] Secondly, this application also provides a multi-sensor-based repositioning device for use in a robot, wherein the robot includes at least two different types of sensors. The device includes:

[0011] The acquisition module is used to acquire environmental images collected by each sensor.

[0012] The global matching module is used to perform global matching between each environmental image and the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0013] The local matching module is used to perform local matching between each environmental image and the surrounding environmental image corresponding to each robot pose before the update in the prior environment image library, so as to obtain the score value of each robot pose after the update.

[0014] The determination module is used to determine the relocalization result of the robot's current pose based on the score values ​​of each updated robot pose.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] Acquire environmental images obtained by each sensor separately;

[0017] Each environmental image is globally matched with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0018] Local matching is performed between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update, to obtain the score value of each robot pose after the update.

[0019] Based on the scores of each updated robot pose, the relocalization result of the robot's current pose is determined.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] Acquire environmental images obtained by each sensor separately;

[0022] Each environmental image is globally matched with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0023] Local matching is performed between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update, to obtain the score value of each robot pose after the update.

[0024] Based on the scores of each updated robot pose, the relocalization result of the robot's current pose is determined.

[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0026] Acquire environmental images obtained by each sensor separately;

[0027] Each environmental image is globally matched with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0028] Local matching is performed between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update, to obtain the score value of each robot pose after the update.

[0029] Based on the scores of each updated robot pose, the relocalization result of the robot's current pose is determined.

[0030] The aforementioned multi-sensor-based relocalization method, apparatus, computer equipment, storage medium, and computer program product acquire environmental images collected by each sensor; perform global matching of each environmental image with images in a prior environmental image library to obtain the robot pose before the update corresponding to each sensor; perform local matching of each environmental image with the surrounding environmental image corresponding to each robot pose before the update in the prior environmental image library to obtain the score value of each robot pose after the update; and determine the relocalization result of the robot's current pose based on the score value of each robot pose after the update. By using the method provided in this application embodiment, and performing global and local matching of environmental images collected by different types of sensors with images in the prior environmental image library, the inaccuracy of relocalization results due to the performance limitations of a single sensor is avoided, thereby ensuring the accuracy and reliability of the obtained relocalization result of the robot's current pose. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of a multi-sensor-based relocation method in one embodiment;

[0032] Figure 2This is a flowchart illustrating a multi-sensor-based relocation method in one embodiment;

[0033] Figure 3 This is a flowchart illustrating a multi-sensor-based relocation method in another embodiment;

[0034] Figure 4 This is a structural block diagram of a multi-sensor-based relocation device in one embodiment;

[0035] Figure 5 This is a structural block diagram of a multi-sensor-based relocation device in another embodiment;

[0036] Figure 6 This is an internal structural diagram of a computer device in one embodiment;

[0037] Figure 7 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] It should be noted that in the following description, the terms "first," "second," and "third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application can be implemented in an order other than that shown or described.

[0040] The multi-sensor-based relocation method provided in this application can be applied to, for example... Figure 1In the application environment shown, robot 102 includes a sensor array 104, which comprises at least two different types of sensors (sensor 104a, sensor 104b, sensor 104c, etc.). Robot 102 acquires environmental images collected by each sensor; it performs global matching between the environmental images collected by each sensor and images in a prior environment image library to obtain the robot pose before the update corresponding to each sensor; it performs local matching between each environmental image and the surrounding environment image corresponding to each robot pose before the update in the prior environment image library to obtain a score value for each updated robot pose; and it determines the relocalization result of the robot's current pose based on the score values ​​of each updated robot pose. Robot 102 includes, but is not limited to, various industrial robots requiring autonomous movement (such as handling robots, palletizing robots, painting robots, etc.), service robots (such as cleaning robots, delivery robots, lawnmower robots, etc.), or special robots (firefighting robots, underwater robots, security robots, etc.). The robot 102 can be connected to the workstation via Bluetooth, USB (Universal Serial Bus) or network, and this application does not impose any restrictions on this.

[0041] In one embodiment, such as Figure 2 As shown, a multi-sensor-based relocalization method is provided and applied to a robot. The robot includes at least two different types of sensors, and the method is based on... Figure 1 Taking robot 102 as an example, the following steps S202 to S208 are explained:

[0042] S202, acquire environmental images collected by each sensor.

[0043] The robot used in this application embodiment can be any terminal device that can move autonomously in the working environment. The robot includes a memory and a processor, and the processor executes the above-described multi-sensor-based relocation method.

[0044] The robot includes at least two different types of sensors for collecting environmental information about the surrounding environment during robot movement. The sensors may include at least two of the following: RGB (Red, Green, Blue) sensors, depth sensors, LiDAR, or other types of sensors; this application embodiment does not limit this. Specifically, the sensors may be mounted on the robot body in the form of lenses, cameras, or laser heads. For example, if the sensor is an RGB sensor, the robot may include an RGB camera; for example, if the sensor is a depth sensor, the robot may include at least one of various commonly used active depth cameras (structured light cameras, ToF cameras, light field cameras, etc.) and passive depth cameras (binocular cameras, etc.). The environmental information collected by RGB cameras and depth cameras is typically represented as images, while the environmental information collected by LiDAR is typically represented as laser point clouds; for convenience, these are collectively referred to as environmental images in this application.

[0045] Specifically, when a robot moves in its environment, it perceives the environment through various sensors at different poses to acquire environmental images collected by each sensor. Pose refers to the robot's position and orientation when acquiring these images; position represents the robot's coordinates in space, and orientation represents its facing direction. In essence, as the robot moves, it acquires environmental images at a certain frequency using sensors. Each frame corresponds to a pose, and different images correspond to different poses. Robot localization or relocalization can be achieved through frame-to-frame matching or pose-to-pose matching. However, regardless of the matching method, pose information is indispensable for the robot's ultimate localization or relocalization.

[0046] Specifically, before acquiring the environmental images collected by each sensor, the user can initiate a relocation task on the robot, thereby enabling the robot to respond to the relocation task and acquire the environmental images collected by each sensor.

[0047] Specifically, before acquiring the environmental images collected by each sensor, the robot may autonomously initiate a relocation task when it detects a loss of localization, and in response to the autonomously initiated relocation task, acquire the environmental images collected by each sensor.

[0048] S204, perform global matching of each environmental image with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0049] The prior environment image library can be stored in the robot's database. The prior map can be a pre-set map of the robot's current environment, which in practical applications is usually a prior map built by the robot using the SLAM (Simultaneous Localization and Mapping) method.

[0050] In one embodiment, the robot performs global matching between each environmental image and an image in a prior environmental image library to obtain the robot pose before the update corresponding to each sensor. This includes: the robot matching the poses corresponding to each environmental image collected by each sensor with the poses corresponding to the images in the prior environmental image library; determining from the prior environmental image library the poses with a pose matching degree greater than a first preset matching degree; and determining these poses as the robot poses before the update, so as to obtain the robot poses before the update corresponding to each sensor.

[0051] Specifically, the global matching of each environmental image with images in the prior environmental image library can be achieved by matching the image description values ​​of each environmental image with the image description values ​​in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor; or it can be achieved by extracting feature point description values ​​from each environmental image and matching them with images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0052] S206, perform local matching between each environmental image and the surrounding environment image corresponding to each robot pose before the update in the prior environment image library to obtain the score value of each updated robot pose.

[0053] Each robot pose before the update corresponds to a prior environment image in the prior environment image library. The surrounding environment image is the surrounding image of each prior environment image in the prior environment image library. Specifically, the surrounding environment image can be an image of each prior environment image within a first preset distance.

[0054] The score of the robot's updated pose can be determined by the matching degree between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update. The higher the matching degree, the higher the score.

[0055] Specifically, the updated robot pose score is obtained by locally matching each environmental image with the corresponding surrounding environment image in the prior environmental image library for each robot pose before the update. This is achieved by locally matching each environmental image acquired by each sensor with the corresponding surrounding environment image in the prior environmental image library for each robot pose before the update, and obtaining the sensor score for each sensor for the surrounding environment image corresponding to each robot pose before the update based on the matching degree. The updated robot pose score is then obtained based on the sensor score for each sensor for the surrounding environment image corresponding to each robot pose before the update. Alternatively, the sensor scores for each sensor for the surrounding environment image corresponding to each robot pose before the update can be summed to obtain the updated robot pose score. The sensor score is determined by the matching degree between the environmental image acquired by the corresponding sensor and the surrounding environment image corresponding to the robot pose before the update, and the matching degree is positively correlated with the sensor score.

[0056] For example, the robot includes a first sensor and a second sensor. The first sensor acquires a first environmental image, and the second sensor acquires a second environmental image. After globally matching the first environmental image and the second environmental image with images in a prior environmental image library, a first pre-update robot pose corresponding to the first sensor and a second pre-update robot pose corresponding to the second sensor are obtained, respectively. The first pre-update robot pose corresponds to the first surrounding environmental image, and the second pre-update robot pose corresponds to the second surrounding environmental image. The first environmental image is locally matched with the first surrounding environmental image and the second surrounding environmental image, and the second environmental image is locally matched with the first surrounding environmental image and the second surrounding environmental image, respectively. If, based on the matching degree between the first environmental image and the first surrounding environmental image, and the matching degree between the second environmental image and the first surrounding environmental image, the sensor score value of the first sensor for the first surrounding environmental image is 11, and the sensor score value of the second sensor for the first surrounding environmental image is 22, then the score value of the first updated robot pose corresponding to the first pre-update robot pose is 11 + 22 = 33. The process of determining the score of the robot pose after the second update, which corresponds to the robot pose before the second update, is the same, so it will not be repeated here.

[0057] Specifically, the method of locally matching each environmental image with the corresponding surrounding environmental image in the prior environment image library for each robot pose before the update can be achieved by extracting feature point description values ​​from each environmental image and matching them with the corresponding surrounding environmental image in the prior environment image library for each robot pose before the update, so as to obtain each updated robot pose. Furthermore, when local matching is based on feature point description values ​​and global matching is based on image description values, the matching accuracy of local matching is greater than that of global matching. Furthermore, when both local matching and global matching are achieved by extracting feature point description values, the matching accuracy of local matching is greater than that of global matching. That is to say, when both feature point description values ​​are extracted for matching, the number of feature points that need to be extracted for local matching is greater than the number of feature points that need to be extracted for global matching, or the matching degree of feature point description values ​​of local matching is higher than that of feature point description values ​​of global matching.

[0058] In one embodiment, local matching is performed between each environmental image and the surrounding environment image corresponding to each robot pose before the update in the prior environmental image library to obtain the score value of each updated robot pose. This includes: obtaining the pose of each environmental image using each sensor, the pose matching degree between each robot pose before the update and the pose corresponding to the surrounding environment image in the prior environmental image library, and determining the score value corresponding to each pose matching degree according to the mapping relationship between the pose matching degree and the score value, so as to obtain the score value of each updated robot pose.

[0059] Specifically, before performing global matching of each environmental image with images in the prior environmental image library, preprocessing operations such as image denoising can be performed on each environmental image to reduce the noise interference in each environmental image, thereby improving the global matching accuracy and subsequent local matching accuracy.

[0060] S208, Based on the scores of each updated robot pose, determine the relocalization result of the robot's current pose.

[0061] The relocalization result of the robot's current pose includes relocalization success and relocalization failure. Relocalization success means that the updated robot pose with the highest score successfully matches the pose of only one image in the prior environment image library. Relocalization failure has two scenarios: First, when the updated robot pose with the highest score successfully matches the poses of multiple images in the prior environment image library, it indicates that multiple identical poses exist in the current environment. Second, when the difference between the updated robot pose's score and the highest score is small, it indicates that other highly similar poses exist in the current environment, potentially leading to the robot confusing its current pose with other highly similar poses and causing relocalization errors. In both cases, because the relocalized pose corresponding to the robot's relocalization result is not unique in the prior environment image library, or even does not exist at all, the relocalization result of the robot's current pose is determined to be relocalization failure. In other words, relocalization success means that the relocalized pose corresponding to the relocalization result is unique in the prior environment image library.

[0062] Specifically, based on the score values ​​of each updated robot pose, the relocalization result of the robot's current pose is determined. This can be achieved by determining the updated robot pose with the highest score value as the localization pose corresponding to the relocalization result of the robot's current pose.

[0063] Furthermore, if the repositioning result of the robot's current pose is a repositioning failure, the robot can generate an alarm signal and send the alarm signal to the user terminal that has established a communication connection with the robot, so as to remind the operator corresponding to the user terminal that the current pose repositioning has failed and the operator needs to move the robot to re-execute the repositioning task; or, the robot can also move autonomously to a position within a second preset distance of the current pose and autonomously re-execute the repositioning task.

[0064] In the aforementioned multi-sensor-based relocalization method, environmental images acquired by each sensor are obtained; each environmental image is globally matched with images in a prior environmental image library to obtain the robot pose before the update corresponding to each sensor; each environmental image is locally matched with the surrounding environmental images corresponding to each robot pose before the update in the prior environmental image library to obtain the score value of each robot pose after the update; based on the score value of each robot pose after the update, the relocalization result of the robot's current pose is determined. By using the method provided in this application embodiment, and performing global and local matching between environmental images acquired by different types of sensors and images in the prior environmental image library, the inaccuracy of relocalization results due to the performance limitations of a single sensor is avoided, thereby ensuring the accuracy and reliability of the obtained relocalization result of the robot's current pose.

[0065] In one embodiment, the above-mentioned global matching of each environmental image with images in a prior environmental image library to obtain the robot pose before the update corresponding to each sensor includes:

[0066] Each environmental image is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to each environmental image.

[0067] The initial robot poses are matched for similarity, and the initial robot poses with similarity higher than the first preset threshold are merged to obtain the robot poses before the update.

[0068] Specifically, similarity matching of the initial robot poses corresponding to each environmental image involves performing similarity matching between each initial robot pose.

[0069] In this embodiment, when performing global matching between each environmental image and images in a prior environmental image library to obtain the robot pose before updating corresponding to each sensor, the process first involves globally matching each environmental image with images in the prior environmental image library to obtain the initial robot pose corresponding to each environmental image. Then, similarity matching is performed on each initial robot pose, and initial robot poses with similarity scores higher than a first preset threshold are merged to obtain the robot poses before updating. Thus, by performing similarity matching between the initial robot poses corresponding to the environmental images and merging similar robot poses to obtain the robot poses before updating corresponding to the sensors, the matching time of subsequent local matching is reduced by decreasing the number of robot poses before updating, thereby improving the relocalization efficiency of the robot's current pose.

[0070] In one embodiment, the above-mentioned global matching of each environmental image with images in a prior environmental image library to obtain the initial robot pose corresponding to each environmental image includes:

[0071] Each environmental image is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to K environmental images. K is the number of environmental images whose pose matching degree with the preset pose corresponding to the images in the prior environmental image library is greater than the second preset threshold in the current pose corresponding to each environmental image.

[0072] Similarity matching is performed on each initial robot pose, including:

[0073] Perform similarity matching on the initial robot poses corresponding to K environmental images.

[0074] In this embodiment, when performing global matching between each environmental image and images in a prior environmental image library to obtain the initial robot pose corresponding to each environmental image, firstly, each environmental image is globally matched with images in the prior environmental image library to obtain the initial robot pose corresponding to K environmental images. Then, similarity matching is performed on the initial robot poses corresponding to the K environmental images to merge the initial robot poses with similarity higher than a first preset threshold. Thus, during global matching, after obtaining K environmental images with pose matching degrees greater than a second preset threshold with preset poses corresponding to images in the prior environmental image library, similarity matching is performed on the initial robot poses corresponding to the K environmental images. On the one hand, by reducing the number of robot poses before the update, the matching time of subsequent local matching is reduced, improving the relocalization efficiency of the robot's current pose. On the other hand, by selecting K environmental images with high pose matching degrees with the prior environmental image library, the accuracy and reliability of the relocalization result of the robot's current pose are improved.

[0075] In one embodiment, the above method further includes:

[0076] Acquire historical environmental images collected by each sensor;

[0077] Perform global matching between each historical environment image and the images in the prior environment image library, and count the global matching time for each historical environment image.

[0078] The target sensor is determined as the sensor whose global matching time corresponding to the acquired historical environmental image meets the preset time condition.

[0079] The above-mentioned global matching of each environmental image with images in the prior environmental image library yields the robot pose before the update corresponding to each sensor, including:

[0080] The environmental image acquired by the target sensor is globally matched with the image in the prior environment image library to obtain the robot pose before the update corresponding to the target sensor.

[0081] The condition of satisfying the preset time can mean that, among the global matching times corresponding to the historical environmental images collected by each sensor, the global matching time corresponding to the historical environmental images collected by the target sensor is the shortest, or it can mean that the global matching time corresponding to the historical environmental images collected by the target sensor is less than the preset matching time.

[0082] In this embodiment, the sensor with the lowest matching time or a matching time less than a certain threshold is selected as the target sensor. During global matching, only these target sensors are used for matching, which can save computing resources and reduce the overall relocation time. For example, in practical engineering applications, the global matching time of RGB cameras is relatively short, while the global matching time of LiDAR is relatively long. Therefore, during global matching, RGB cameras are usually used as target sensors for matching.

[0083] In one embodiment, after globally matching the environmental image acquired by the target sensor with the image in the prior environmental image library to obtain the robot pose before the update corresponding to the target sensor, the above-mentioned performing local matching of each environmental image with the surrounding environmental image corresponding to each robot pose before the update in the prior environmental image library to obtain the score value of each robot pose after the update includes: performing local matching of each environmental image with the surrounding environmental image corresponding to the robot pose before the update corresponding to the target sensor in the prior environmental image library to obtain the score value of each robot pose after the update.

[0084] In one embodiment, before globally matching each environmental image with images in a prior environment image library, the method further includes: obtaining the relocation time requirement corresponding to the relocation task; and globally matching each environmental image with images in the prior environment image library to obtain the robot pose before update corresponding to each sensor, including: if the relocation time requirement is greater than a preset time requirement, then globally matching each environmental image acquired by each sensor with images in the prior environment image library to obtain the robot pose before update corresponding to each sensor; if the relocation time requirement is less than or equal to the preset time requirement, then globally matching the environmental image acquired by the target sensor with images in the prior environment image library to obtain the robot pose before update corresponding to the target sensor. The preset time requirement can be determined based on the user's sensitivity to the relocation time requirement. If the user is more sensitive to the robot's relocation time, a smaller preset time requirement can be set; conversely, a slightly larger preset time requirement can be set.

[0085] In this embodiment, a target sensor is selected based on the user's sensitivity to the robot's relocalization time. The environmental image acquired by the target sensor is globally matched with images in a prior environmental image library to obtain the robot pose before the update corresponding to the environmental image acquired by the target sensor. Then, the environmental images acquired by each sensor are locally matched with the surrounding environmental images corresponding to each robot pose before the update in the prior environmental image library to obtain the relocalization result. In this way, the needs of users with different time sensitivities can be met.

[0086] In one embodiment, the above-mentioned local matching of each environmental image with the surrounding environment image corresponding to each robot pose before the update in the prior environment image library to obtain the score value of each robot pose after the update includes:

[0087] Local matching is performed between each environmental image and the corresponding surrounding environmental image in the prior environmental image library for each robot pose before the update, to obtain the initial score value of each robot pose before the update.

[0088] Based on the sensor weight values ​​corresponding to each sensor and the initial score values ​​of the robot poses before each update, the score values ​​of the robot poses after each update are obtained.

[0089] The sensor weight values ​​for each sensor can be determined by the operator connected to the user terminal that establishes a communication link with the robot, based on actual needs. For example, the sensor weight values ​​can be determined based on the operator's performance preference for each sensor, or based on the robot's current environment.

[0090] It is understandable that different types of sensors experience varying degrees of performance interference in the same working environment. For example, taking a robot equipped with both RGB cameras and LiDAR sensors as an example, if the robot's current working environment contains many corners and pillars, the RGB camera will outperform the LiDAR in this environment. Conversely, if the lighting conditions in the robot's current environment are poor, the LiDAR will outperform the RGB camera. Therefore, to avoid the relocation results being affected by the performance interference experienced by each sensor, in one embodiment, the method further includes: acquiring the performance information of each sensor and the robot's current environment information; determining the environmental adaptation value of each sensor in the robot's current environment information based on the performance information of each sensor; determining the sensor weight value corresponding to each sensor based on the environmental adaptation value of each sensor, wherein the environmental adaptation value and the sensor weight value are positively correlated; the environmental adaptation value represents the degree of performance of each sensor in the robot's current environment, and the environmental adaptation value and the degree of performance are positively correlated. Therefore, in the task of repositioning the robot to its current pose, sensors that have a greater performance advantage in the current environment can better leverage their performance advantages, thereby making the repositioning results more accurate and reliable, and also making the robot more flexible in performing the task of repositioning to its current pose.

[0091] In this embodiment, when performing local matching between each environmental image and the corresponding surrounding environment image in the prior environment image library for each robot pose before the update, and obtaining the score value of each updated robot pose, the process first involves performing local matching on the corresponding surrounding environment image in the prior environment image library for each updated robot pose before the update to obtain the initial score value of each updated robot pose. Then, based on the sensor weight values ​​corresponding to each sensor and the initial score value of each updated robot pose before the update, the score value of each updated robot pose is obtained. Therefore, the score value of each updated robot pose is determined simultaneously by the pose matching degree and the sensor weight values, making the determination of the score value of each updated robot pose more reliable, thereby ensuring the accuracy and reliability of the obtained robot current pose relocalization result.

[0092] In one embodiment, determining the relocalization result of the robot's current pose based on the scores of each updated robot pose includes:

[0093] Based on the updated robot pose scores of each robot, determine the first score and the second score that meet the score conditions, with the first score being greater than the second score.

[0094] If the difference between the first score and the second score is greater than the third preset threshold, and the first score is greater than the fourth preset threshold, then the repositioning result for the current pose is determined to be a successful repositioning.

[0095] The condition for meeting the score criteria can be that the corresponding scores are the two highest scores among all updated robot poses. That is, the first score and the second score are the highest and second highest scores, respectively. Specifically, if the difference between the first score and the second score is less than or equal to a third preset threshold, it means that the relocalization result of the current robot pose is not unique. If the first score is less than or equal to a fourth preset threshold, it means that there is no corresponding pose with high similarity to the updated robot pose in the prior environment image library. Therefore, either of these two situations will result in a relocalization failure.

[0096] Specifically, determining the relocalization result of the robot's current pose can be done by determining the current pose as the pose corresponding to the relocalization result, or by determining the updated robot pose corresponding to the first score as the pose corresponding to the relocalization result.

[0097] In this embodiment, when determining the relocalization result of the robot's current pose based on the scores of each updated robot pose, a first score and a second score that meet the score conditions are first determined based on the scores of each updated robot pose. If the difference between the first score and the second score is greater than a third preset threshold, and the first score is greater than a fourth preset threshold, then the relocalization result for the current pose is determined to be successful. Thus, by determining a first score greater than the fourth preset threshold and a second score less than the first score that meet the score conditions, and simultaneously determining that the difference between the first score and the second score is greater than the third preset threshold, and then determining successful relocalization, the adverse situation of relocalization failure caused by the robot's updated robot poses not being similar to the poses in the prior environment image library, or the relocalization result not being unique, is avoided. This ensures that the obtained relocalization result of the robot's current pose has accuracy and reliability.

[0098] In one embodiment, after determining that the relocalization result of the robot's current pose is successful, the above method further includes:

[0099] Set the robot's current pose to the robot's corresponding preset power-on pose, which is the pose the robot will adopt when it restarts.

[0100] In this embodiment, after determining that the relocation result of the robot's current pose is successful, the robot's current pose is set as the corresponding pose when the robot restarts. Thus, when the robot's pose changes due to events such as system shutdown, power outage, or being moved by a person, the robot can re-determine its position on the map during relocation, thereby ensuring that the robot will not lose its position or be incorrectly positioned due to unforeseen circumstances.

[0101] Taking a robot with three types of sensors—RGB camera, depth camera, and LiDAR—as an example, such as Figure 3 As shown below, the application process of the above-mentioned multi-sensor-based relocation method will be illustrated with a detailed embodiment:

[0102] First, the robot acquires RGB images, depth images, and laser point clouds obtained from RGB cameras, depth cameras, and LiDAR, respectively.

[0103] Then, the robot performs global matching of the RGB image, depth image, and laser point cloud with the images in the prior environment image library to obtain the initial robot pose corresponding to K environment images. K is the number of environment images whose pose matching degree with the preset poses corresponding to the images in the prior environment image library is greater than a second preset threshold in the current pose corresponding to each environment image. The robot performs similarity matching on the initial robot poses corresponding to the K environment images and merges the initial robot poses with similarity higher than the first preset threshold to obtain the robot poses before the update corresponding to the RGB camera, depth camera, and LiDAR.

[0104] The robot performs local matching between the RGB image, depth image, and laser point cloud and the surrounding environment image corresponding to each robot pose before the update in the prior environment image library. Thus, the robot obtains the score value of each robot pose after the update based on the sensor weight value corresponding to each sensor and the initial score value of each robot pose before the update. These scores are score value 1, score value 2, and score value 3. Score value 1 is the score value of the updated robot pose 1 corresponding to the robot pose 1 before the update. Score values ​​2 and 3 are similar and will not be described in detail here.

[0105] The robot determines the highest score and the second highest score based on the scores of each updated robot pose. If the difference between the highest score and the second highest score is greater than the third preset threshold and the highest score is greater than the fourth preset threshold, then the relocalization result for the current pose is determined to be successful, and the current pose is determined as the pose corresponding to the relocalization result.

[0106] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0107] Based on the same inventive concept, this application also provides a multi-sensor-based relocation device for implementing the multi-sensor-based relocation method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the multi-sensor-based relocation device provided below can be found in the limitations of the multi-sensor-based relocation method described above, and will not be repeated here.

[0108] In one embodiment, such as Figure 4 As shown, a multi-sensor-based relocation device is provided, comprising: an acquisition module 1002, a global matching module 1004, a local matching module 1006, and a determination module 1008, wherein:

[0109] The acquisition module 1002 is used to acquire environmental images collected by each sensor.

[0110] The global matching module 1004 is used to perform global matching between each environmental image and the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor.

[0111] The local matching module 1006 is used to perform local matching between each environmental image and the surrounding environmental image corresponding to each robot pose before the update in the prior environmental image library, so as to obtain the score value of each robot pose after the update.

[0112] The determination module 1008 is used to determine the relocalization result of the robot's current pose based on the score values ​​of each updated robot pose.

[0113] In one embodiment, in performing global matching of each environmental image with images in a prior environmental image library to obtain each robot pose before the update corresponding to the sensor, the global matching module 1004 is further configured to:

[0114] Each environmental image is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to each environmental image.

[0115] The initial robot poses are matched for similarity, and the initial robot poses with similarity higher than the first preset threshold are merged to obtain the robot poses before the update.

[0116] In one embodiment, in performing global matching of each environmental image with images in a prior environmental image library to obtain each robot pose before the update corresponding to the sensor, the global matching module 1004 is further configured to:

[0117] Each environmental image is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to K environmental images; K is the number of environmental images whose pose matching degree with the preset poses corresponding to the images in the prior environmental image library is greater than the second preset threshold in the current pose corresponding to each environmental image.

[0118] Similarity matching with each initial robot pose is performed, including:

[0119] Perform similarity matching on the initial robot poses corresponding to K environmental images.

[0120] In one embodiment, such as Figure 5 As shown, the above method also includes a statistics module 1010, which is used for:

[0121] Acquire historical environmental images collected by each sensor;

[0122] Perform global matching between each historical environment image and the images in the prior environment image library, and count the global matching time for each historical environment image.

[0123] The target sensor is determined as the sensor whose global matching time corresponding to the acquired historical environmental image meets the preset time condition.

[0124] In performing global matching between each environmental image and images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor, the aforementioned global matching module 1004 is also used for:

[0125] The environmental image acquired by the target sensor is globally matched with the image in the prior environment image library to obtain the robot pose before the update corresponding to the target sensor.

[0126] In one embodiment, the local matching module 1006 is used to perform local matching between each environmental image and the surrounding environment image corresponding to each robot pose before the update in the prior environment image library to obtain the score value of each updated robot pose.

[0127] Local matching is performed between each environmental image and the corresponding surrounding environmental image in the prior environmental image library for each robot pose before the update, to obtain the initial score value of each robot pose before the update.

[0128] Based on the sensor weight values ​​corresponding to each sensor and the initial score values ​​of the robot poses before each update, the score values ​​of the robot poses after each update are obtained.

[0129] In one embodiment, in determining the relocalization result of the robot's current pose based on the scores of each updated robot pose, the determining module 1008 is further configured to:

[0130] Based on the scores of each updated robot pose, determine the first score and the second score that meet the score conditions, with the first score being greater than the second score.

[0131] If the difference between the first score and the second score is greater than the third preset threshold, and the first score is greater than the fourth preset threshold, then the repositioning result for the current pose is determined to be a successful repositioning.

[0132] In one embodiment, such as Figure 5 As shown, after the determining module 1008 determines that the repositioning result of the robot's current pose is a successful repositioning, the above device further includes a setting module 1012, which is used for:

[0133] Set the robot's current pose to the robot's corresponding preset power-on pose, which is the pose the robot will adopt when it restarts.

[0134] Each module in the aforementioned multi-sensor-based repositioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0135] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores prior environment image library data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a multi-sensor-based relocation method.

[0136] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a multi-sensor-based relocation method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0137] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A relocation method based on multiple sensors, characterized in that, Applied to a robot, wherein the robot includes at least two different types of sensors, the method includes: Acquire environmental images obtained by each of the aforementioned sensors; The environmental images are globally matched with the images in the prior environmental image library to obtain the robot pose before the update corresponding to each sensor. The environmental images are locally matched with the surrounding environment images corresponding to the robot poses before the update in the prior environment image library to obtain the initial score value of each robot pose before the update; the score value of each robot pose after the update is obtained according to the sensor weight value corresponding to each sensor and the initial score value of each robot pose before the update. Based on the scores of the updated robot poses, a first score and a second score that meet the score conditions are determined; the first score is greater than the second score; if the difference between the first score and the second score is greater than a third preset threshold, and the first score is greater than a fourth preset threshold, then the relocalization result for the current pose is determined to be successful. The method further includes: The system acquires the performance information of each sensor and the current environment information of the robot; based on the performance information of each sensor, it determines the environment adaptation value of each sensor in the current environment information of the robot; based on the environment adaptation value of each sensor, it determines the sensor weight value of each sensor, and there is a positive correlation between the environment adaptation value of each sensor and the sensor weight value of each sensor.

2. The method according to claim 1, characterized in that, The step of globally matching each of the environmental images with images in a prior environmental image library to obtain the robot pose before the update corresponding to each of the sensors includes: Each of the environmental images is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to each of the environmental images; The initial robot poses are matched for similarity, and the initial robot poses with similarity higher than a first preset threshold are merged to obtain the robot poses before the update.

3. The method according to claim 2, characterized in that, The step of globally matching each of the environmental images with images in a prior environmental image library to obtain the initial robot pose corresponding to each of the environmental images includes: Each of the environmental images is globally matched with the images in the prior environmental image library to obtain the initial robot pose corresponding to K environmental images. K is the number of environmental images whose pose matching degree with the preset pose corresponding to the images in the prior environmental image library is greater than the second preset threshold in the current pose corresponding to each of the environmental images. The similarity matching of the initial robot poses includes: Similarity matching is performed on the initial robot poses corresponding to the K environmental images.

4. The method according to claim 1, characterized in that, The method further includes: Acquire historical environmental images collected by each of the aforementioned sensors; Each historical environment image is globally matched with the images in the prior environment image library, and the global matching time corresponding to each historical environment image is counted. The target sensor is determined as a sensor whose global matching time corresponding to the acquired historical environmental image meets a preset time condition. The step of globally matching each of the environmental images with images in a prior environmental image library to obtain the robot pose before the update corresponding to each of the sensors includes: The environmental image acquired by the target sensor is globally matched with the image in the prior environmental image library to obtain the robot pose before the update corresponding to the target sensor.

5. The method according to claim 1, characterized in that, The acquisition of environmental images collected by each of the sensors includes: In response to a relocation task initiated by the user on the robot, environmental images acquired by each of the sensors are obtained.

6. The method according to claim 1, characterized in that, If the repositioning result of the robot's current pose is a repositioning failure, an alarm signal is generated; The alarm signal is sent to the user terminal that has established a communication connection with the robot.

7. The method according to any one of claims 1-5, characterized in that, After determining that the relocation result of the robot's current pose is successful, the method further includes: The robot's current pose is set to the robot's corresponding preset power-on pose, which is the pose of the robot when it restarts.

8. A repositioning device based on multiple sensors, characterized in that, Applied in robots, the robots include at least two different types of sensors, and the device includes: The acquisition module is used to acquire environmental images collected by each of the sensors. The global matching module is used to perform global matching between each of the environmental images and the images in the prior environmental image library to obtain the robot pose before the update corresponding to each of the sensors. The local matching module is used to perform local matching between each of the environmental images and the surrounding environmental images corresponding to each of the robot poses before the update in the prior environmental image library to obtain the initial score value of each robot pose before the update; and to obtain the score value of each robot pose after the update based on the sensor weight value corresponding to each of the sensors and the initial score value of each robot pose before the update. The determination module is used to determine a first score and a second score that meet the score conditions based on the score values ​​of each updated robot pose. The first score is greater than the second score. If the difference between the first score and the second score is greater than a third preset threshold and the first score is greater than a fourth preset threshold, then the relocalization result for the current pose is determined to be successful. The local matching module is further used for: The system acquires the performance information of each sensor and the current environment information of the robot; based on the performance information of each sensor, it determines the environment adaptation value of each sensor in the current environment information of the robot; based on the environment adaptation value of each sensor, it determines the sensor weight value of each sensor, and there is a positive correlation between the environment adaptation value of each sensor and the sensor weight value of each sensor.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images