Multi-sensor-based relocalization method and apparatus, device, and storage medium

Abstract

The present application relates to a multi-sensor-based relocalization method and apparatus, a device, and a storage medium. The method comprises: acquiring environment images acquired respectively by sensors; performing global matching for each environment image against images in a prior environment image library, so as to respectively obtain each pre-update robot pose corresponding to each sensor; performing local matching for each environment image against surrounding environment images corresponding to each pre-update robot pose in the prior environment image library, so as to obtain a score value of each post-update robot pose of a robot; and on the basis of the score value of each post-update robot pose, determining a relocalization result of a current pose of the robot.

Description

Multi-sensor based repositioning method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on December 28, 2023, with application number 202311862137.3 and application name “Multi-sensor based repositioning method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of robot positioning technology, and in particular to a multi-sensor based repositioning method, apparatus, device and storage medium. Background Art

[0004] With the rapid development of machine learning and perception technologies, mobile robots with autonomous navigation capabilities have entered our daily lives at an unprecedented rate. Mobile robots primarily use pre-configured sensors such as lidar and cameras to perceive environmental information and their own status, performing simultaneous localization and mapping, enabling autonomous movement toward a target in an environment with obstacles—a process known as autonomous navigation. The foundation for autonomous navigation is obtaining an initial position within a pre-defined map. However, if a mobile robot system is shut down, loses power, or is moved, its position and posture may change, making it impossible to locate its current map position and posture upon restart. In this case, the robot needs to re-locate its position within the map and use this initial position as the initial position to restart real-time positioning. This is the technology behind relocalization.

[0005] Existing robot repositioning methods typically use a preset start-up point or rely solely on a LiDAR sensor or visual sensor. However, presetting the start-up point can cause the robot to err in similar areas outside the preset start-up point. Repositioning based on a single sensor can also lead to start-up positioning failures due to the sensor's susceptibility to environmental conditions such as lighting, weather, and season.

[0006] Summary of the Invention

[0007] According to various embodiments of the present application, a multi-sensor based repositioning method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.

[0008] In a first aspect, the present application provides a multi-sensor based relocalization method for a robot, wherein the robot includes at least two different types of sensors. The method includes:

[0009] Acquire environmental images collected by each sensor;

[0010] Globally match each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor;

[0011] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of each updated robot posture of the robot;

[0012] According to the score value of each updated robot posture, the relocalization result of the robot's current posture is determined.

[0013] In a second aspect, the present application also provides a multi-sensor based repositioning device for use in a robot, wherein the robot includes at least two different types of sensors. The device includes:

[0014] An acquisition module is used to acquire the environment images collected by each sensor;

[0015] The global matching module is used to globally match each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor;

[0016] A local matching module is used to locally match each environment image with the surrounding environment image corresponding to each pre-update robot posture in the prior environment image library to obtain a score value of each updated robot posture;

[0017] The determination module is used to determine the relocation result of the robot's current posture according to the score value of each updated robot posture.

[0018] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0019] Acquire environmental images collected by each sensor;

[0020] Globally match each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor;

[0021] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of each updated robot posture of the robot;

[0022] According to the score value of each updated robot posture, the relocalization result of the robot's current posture is determined.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0024] Acquire environmental images collected by each sensor;

[0025] Globally match each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor;

[0026] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of each updated robot posture of the robot;

[0027] According to the score value of each updated robot posture, the relocalization result of the robot's current posture is determined.

[0028] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0029] Acquire environmental images collected by each sensor;

[0030] Globally match each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor;

[0031] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of each updated robot posture of the robot;

[0032] According to the score value of each updated robot posture, the relocation result of the robot's current posture is determined

[0033] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.

[0035] FIG1 is a diagram illustrating an application environment of a multi-sensor based relocation method according to an embodiment;

[0036] FIG2 is a schematic flow chart of a multi-sensor based relocation method according to an embodiment;

[0037] FIG3 is a schematic flow chart of a multi-sensor based relocation method according to another embodiment;

[0038] FIG4 is a block diagram of a multi-sensor based relocation device according to an embodiment;

[0039] FIG5 is a block diagram of a multi-sensor based relocation device according to another embodiment;

[0040] FIG6 is a diagram showing the internal structure of a computer device according to one embodiment;

[0041] FIG7 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0042] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0044] The multi-sensor relocalization method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. In which, the robot 102 includes a sensor array 104, and the sensor array 104 includes at least two different types of sensors (sensor 104a, sensor 104b, sensor 104c...). The robot 102 obtains the environment image collected by each sensor; the robot 102 globally matches the environment image collected by each sensor with the image in the prior environment image library, and obtains the robot posture before the update corresponding to each sensor; the robot 102 locally matches each environment image with the surrounding environment image corresponding to each pre-update robot posture in the prior environment image library, and obtains the score value of each updated robot posture of the robot; the robot 102 determines the relocalization result of the robot's current posture based on the score value of each updated robot posture. In which, the robot 102 includes but is not limited to various industrial robots that require autonomous movement (such as handling robots, stacking robots, spraying robots, etc.), service robots (such as cleaning robots, delivery robots, mowing robots, etc.) or special robots (firefighting robots, underwater robots, security robots, etc.). The robot 102 and the workstation can be connected via Bluetooth, USB (Universal Serial Bus) or network communication connection methods, and this application does not impose any restrictions on this.

[0045] In one embodiment, as shown in FIG2 , a multi-sensor based relocalization method is provided, which is applied to a robot. The robot includes at least two different types of sensors. The method is described by taking the robot 102 in FIG1 as an example, and includes the following steps S202 to S208:

[0046] S202: Acquire the environment image collected by each sensor.

[0047] Among them, the robot used in the embodiment of the present application can be any terminal device that can realize autonomous movement in an operating environment. The robot includes a memory and a processor, and the processor executes the above-mentioned multi-sensor based repositioning method.

[0048] The robot includes at least two different types of sensors for collecting environmental information of the surrounding environment during the movement of the robot. The sensor may include at least two of an RGB (Red Green Blue) sensor, a depth sensor, a lidar or other types of sensors, and the embodiments of the present application are not limited to this. Specifically, the sensor in the robot can be installed on the robot body in the form of a lens, a camera, a camera or a laser head. For example, if the sensor is an RGB sensor, the robot may include an RGB camera; for example, if the sensor is a depth sensor, the robot may include at least one of various commonly used active depth cameras (structured light cameras, ToF cameras, light field cameras, etc.) and passive depth cameras (binocular cameras, etc.). The environmental information collected by the RGB camera and the depth camera is usually expressed in the form of an image, and the environmental information collected by the lidar is usually expressed in the form of a laser point cloud. For convenience, they are collectively referred to as environmental images in this application.

[0049] Specifically, when the robot moves in the environment, it will sense the environment in which it is located through various sensors in different postures to obtain the environment images collected by each sensor. The posture is the position and posture of the robot when collecting the environment image. The position is used to represent the coordinate information of the robot in space, and the posture is used to represent the orientation of the robot in space. It can be understood that when the robot moves, it collects environmental images of the surrounding environment at a certain frequency through sensors. Each frame of the image corresponds to a posture, and different images correspond to different postures. The positioning or repositioning of the robot can be achieved by matching image frames with image frames, or directly by matching postures with postures. However, whether matching through image frames or matching through postures, the final positioning or repositioning of the robot cannot be separated from posture information.

[0050] Specifically, before acquiring the environment images respectively captured by the sensors, the user may initiate a relocation task on the robot, so that the robot responds to the relocation task and acquires the environment images respectively captured by the sensors.

[0051] Specifically, before acquiring the environment images respectively captured by each sensor, the robot may autonomously initiate a relocation task when detecting that positioning is lost, and acquire the environment images respectively captured by each sensor in response to the autonomously initiated relocation task.

[0052] S204 , globally matching each environment image with an image in a priori environment image library to obtain the robot pose before update corresponding to each sensor.

[0053] The prior environment image library can be stored in the robot's database. The prior map can be pre-set map data of the robot's current environment. In practical applications, it is usually a prior map created by the robot using SLAM (Simultaneous Localization and Mapping) methods.

[0054] In one embodiment, each environmental image is globally matched with an image in a prior environmental image library to obtain a robot pose before update corresponding to each sensor, including: the robot matches the pose corresponding to each environmental image respectively acquired by each sensor with the pose corresponding to the image in the prior environmental image library, determines from the prior environmental image library a pose having a pose matching degree with the pose of the environmental image greater than a first preset matching degree, and determines it as the robot pose before update, so as to obtain a robot pose before update corresponding to each sensor.

[0055] Specifically, the method of globally matching each environmental image with the images in the prior environmental image library can be achieved by matching the image description values ​​of each environmental image with the image description values ​​in the prior environmental image library to obtain the robot pose before update corresponding to each sensor; it can also be achieved by extracting feature point description values ​​from each environmental image and matching them with the images in the prior environmental image library to obtain the robot pose before update corresponding to each sensor.

[0056] S206 , locally matching each environment image with the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain a score value for each robot posture after update.

[0057] Among them, each robot posture before updating corresponds to a prior environment image in the prior environment image library, and the surrounding environment image is a surrounding image of each prior environment image in the prior environment image library. Specifically, the surrounding environment image can be an image of each prior environment image within a first preset distance.

[0058] The score value of the robot's updated robot posture can be determined by the matching degree between each environment image and the surrounding environment image corresponding to each robot posture before the update in the prior environment image library. The higher the matching degree, the higher the score value.

[0059] Specifically, each environment image is locally matched with the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of the robot's updated robot posture, which is to locally match each environment image collected by each sensor with the surrounding environment image corresponding to each robot posture before update in the prior environment image library, and obtain the sensor score value of each sensor for the surrounding environment image corresponding to each robot posture before update according to the matching degree, and then obtain the score value of the robot's updated robot posture according to the sensor score value of each sensor for the surrounding environment image corresponding to each robot posture before update. Furthermore, the sensor score value of each sensor for the surrounding environment image corresponding to each robot posture before update can be summed to obtain the score value of the robot's updated robot posture; the sensor score value is determined by the matching degree between the environment image collected by the corresponding sensor and the surrounding environment image corresponding to the robot posture before update, and the matching degree is positively correlated with the sensor score value.

[0060] Exemplarily, the robot includes a first sensor and a second sensor. The first sensor collects a first environment image, and the second sensor collects a second environment image. After globally matching the first environment image and the second environment image with images in a priori environment image library, a first robot pose before update corresponding to the first sensor and a second robot pose before update corresponding to the second sensor are obtained, respectively. The first robot pose before update corresponds to the first surrounding environment image, and the second robot pose before update corresponds to the second surrounding environment image. The first environment image is locally matched with the first surrounding environment image and the second surrounding environment image, respectively. At the same time, the second environment image is locally matched with the first surrounding environment image and the second surrounding environment image, respectively. If, based on the matching degree between the first environment image and the first surrounding environment image, and the matching degree between the second environment image and the first surrounding environment image, the sensor score value of the first sensor for the first surrounding environment image is 11, and the sensor score value of the second sensor for the first surrounding environment image is 22, then the score value of the first robot pose after update corresponding to the first robot pose before update is 11+22=33. The process of determining the score value of the second updated robot posture corresponding to the second pre-update robot posture is similar to that of determining the score value of the second updated robot posture, so it will not be repeated here.

[0061] Specifically, the method of locally matching each environment image with the surrounding environment image corresponding to each robot posture before update in the prior environment image library can be achieved by extracting feature point description values ​​from each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library for matching, so as to obtain each updated robot posture of the robot; further, when the local matching is performed by matching through feature point description values, and the global matching is performed by matching through image description values, the matching accuracy of the local matching is greater than the matching accuracy of the global matching; further, when both local matching and global matching are achieved by extracting feature point description values ​​for matching, the matching accuracy of the local matching is greater than the matching accuracy of the global matching, that is, when both feature point description values ​​are extracted for matching, the number of feature points that need to be extracted for local matching is greater than the number of feature points that need to be extracted for global matching, or the matching degree of the feature point description values ​​of the local matching is higher than the matching degree of the feature point description values ​​of the global matching.

[0062] In one embodiment, each environment image is locally matched with the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the score value of each updated robot posture of the robot, including: using each sensor to obtain the posture of each environment image respectively, and the posture matching degree between the posture corresponding to the surrounding environment image corresponding to each robot posture before update in the prior environment image library, and determining the score value corresponding to each posture matching degree according to the mapping relationship between the posture matching degree and the score value to obtain the score value of each updated robot posture of the robot.

[0063] Specifically, before globally matching each environmental image with images in the prior environmental image library, preprocessing operations such as image denoising can be performed on each environmental image to improve the global matching accuracy and subsequent local matching accuracy by reducing the noise interference in each environmental image.

[0064] S208: Determine the repositioning result of the robot's current posture according to the score value of each updated robot posture.

[0065] The relocalization results for the robot's current pose include successful relocalization and failed relocalization. A successful relocalization can mean that the updated robot pose with the highest score successfully matches the pose corresponding to only one image in the prior environment image library. Failed relocalization can occur in two cases: first, when the updated robot pose with the highest score successfully matches the poses corresponding to multiple images in the prior environment image library, this indicates that the robot's current pose exists in multiple identical poses in the current environment. Second, when the difference between the score of the updated robot pose and the highest score is small, this indicates that the robot's current pose exists in other highly similar poses in the current environment, potentially leading to the risk of the robot confusing the current pose with other highly similar poses and causing relocalization errors. In both cases, since the positioning pose corresponding to the robot's relocalization result is not unique in the prior environment image library, or even does not exist, the relocalization result for the robot's current pose is considered a failed relocalization. In other words, a successful relocalization means that the positioning pose corresponding to the relocalization result is unique in the prior environment image library.

[0066] Specifically, the repositioning result of the robot's current posture is determined according to the score values ​​of each updated robot posture, and the updated robot posture corresponding to the highest score value can be determined as the positioning posture corresponding to the repositioning result of the robot's current posture.

[0067] Furthermore, if the repositioning result of the robot's current posture is a repositioning failure, the robot can generate an alarm signal and send the alarm signal to the user terminal that has established a communication connection with the robot, so as to remind the operator corresponding to the user terminal that the current posture repositioning has failed and the operator is required to move the robot to re-execute the repositioning task; or, the robot can also move autonomously to a position within a second preset distance of the current posture and autonomously re-execute the repositioning task.

[0068] In the above-mentioned multi-sensor based relocalization method, the environmental images collected by each sensor are obtained; each environmental image is globally matched with the images in the prior environmental image library to obtain the robot posture before update corresponding to each sensor; each environmental image is locally matched with the surrounding environmental image corresponding to each robot posture before update in the prior environmental image library to obtain the score value of each updated robot posture of the robot; based on the score value of each updated robot posture, the relocalization result of the robot's current posture is determined. The method provided in the embodiment of the present application is used to perform global and local matching on the environmental images collected by different types of sensors with the images in the prior environmental image library, thereby avoiding the situation where the relocalization result of a single sensor is inaccurate due to performance limitations, thereby ensuring that the relocalization result of the robot's current posture is accurate and reliable.

[0069] In one embodiment, the above-mentioned global matching of each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor includes:

[0070] Perform global matching between each environment image and the images in the prior environment image library to obtain the initial robot pose corresponding to each environment image;

[0071] Similarity matching is performed on each initial robot posture, and corresponding initial robot postures with similarities higher than a first preset threshold are merged to obtain each pre-update robot posture.

[0072] Specifically, similarity matching is performed on the initial robot postures corresponding to the respective environment images, and similarity matching is performed on the initial robot postures respectively.

[0073] In this embodiment, when globally matching each environment image with images in the prior environment image library to obtain the pre-update robot pose corresponding to each sensor, each environment image is first globally matched with the images in the prior environment image library to obtain the initial robot pose corresponding to each environment image. Similarity matching is then performed on each initial robot pose, and corresponding initial robot poses with similarities exceeding a first preset threshold are merged to obtain the pre-update robot pose. Thus, by performing similarity matching on the initial robot poses corresponding to the environment images and merging similar robot poses to obtain the pre-update robot pose corresponding to the sensor, the time required for subsequent local matching is reduced by reducing the number of pre-update robot poses, thereby improving the efficiency of relocalizing the robot's current pose.

[0074] In one embodiment, the above-mentioned global matching of each environment image with the images in the prior environment image library to obtain the initial robot pose corresponding to each environment image includes:

[0075] Globally matching each environment image with images in the prior environment image library to obtain initial robot poses corresponding to K environment images, where K is the number of environment images whose pose matching degree between the current pose corresponding to each environment image and the preset pose corresponding to the image in the prior environment image library is greater than a second preset threshold;

[0076] Perform similarity matching on each initial robot pose, including:

[0077] Perform similarity matching on the initial robot poses corresponding to K environment images.

[0078] In this embodiment, when globally matching each environment image with images in the prior environment image library to obtain initial robot poses corresponding to each environment image, each environment image is first globally matched with images in the prior environment image library to obtain initial robot poses corresponding to K environment images. Then, similarity matching is performed on the initial robot poses corresponding to the K environment images, and initial robot poses corresponding to images with similarities exceeding a first preset threshold are merged. Thus, during global matching, after obtaining K environment images with pose matching degrees exceeding a second preset threshold between the preset poses corresponding to the images in the prior environment image library, similarity matching is then performed on the initial robot poses corresponding to the K environment images. This reduces the number of pre-update robot poses, thus reducing the time required for subsequent local matching and improving the efficiency of relocalizing the robot's current pose. Furthermore, by selecting K environment images with high pose matching degrees with the prior environment image library, the accuracy and reliability of the relocalization result for the robot's current pose are improved.

[0079] In one embodiment, the method further includes:

[0080] Acquire historical environmental images collected by each sensor;

[0081] Perform global matching on each historical environment image and the images in the prior environment image library, and count the global matching time corresponding to each historical environment image;

[0082] Determine a target sensor, where the target sensor is a sensor whose global matching time corresponding to the collected historical environment image meets a preset time condition;

[0083] The above globally matches each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor, including:

[0084] The environment image captured by the target sensor is globally matched with the image in the prior environment image library to obtain the robot pose before update corresponding to the target sensor.

[0085] Among them, satisfying the preset time condition may mean that among the global matching times corresponding to the historical environment images collected by each sensor, the global matching time corresponding to the historical environment image collected by the target sensor is the shortest, or it may mean that the global matching time corresponding to the historical environment image collected by the target sensor is less than the preset matching time.

[0086] In this embodiment, sensors with the lowest matching time, or those with matching time less than a certain matching time threshold, are selected as target sensors. During global matching, only these target sensors are used for matching, which saves computing resources and reduces the overall relocalization time. For example, in actual engineering applications, RGB cameras take a shorter time to match globally, while LiDAR takes a longer time to match globally. Therefore, RGB cameras are typically used as target sensors for global matching.

[0087] In one embodiment, after globally matching the environment image captured by the target sensor with the image in the prior environment image library to obtain the pre-update robot pose corresponding to the target sensor, the above-mentioned locally matching each environment image with the surrounding environment image corresponding to each pre-update robot pose in the prior environment image library to obtain the score value of each updated robot pose of the robot includes: locally matching each environment image with the surrounding environment image corresponding to the pre-update robot pose corresponding to the target sensor in the prior environment image library to obtain the score value of each updated robot pose of the robot.

[0088] In one embodiment, before globally matching each environmental image with an image in a prior environmental image library, the method further includes: obtaining a required relocalization time corresponding to the relocalization task; globally matching each environmental image with an image in the prior environmental image library to obtain a pre-update robot pose corresponding to each sensor, including: if the required relocalization time is greater than a preset required time, globally matching each environmental image acquired by each sensor with an image in the prior environmental image library to obtain a pre-update robot pose corresponding to each sensor; if the required relocalization time is less than or equal to the preset required time, globally matching the environmental image acquired by the target sensor with an image in the prior environmental image library to obtain a pre-update robot pose corresponding to the target sensor. The preset required time can be determined based on the user's sensitivity to the required relocalization time. If the user is more sensitive to the robot's relocalization time, a smaller preset required time can be set; otherwise, a slightly larger preset required time can be set.

[0089] In this embodiment, a target sensor is selected based on the user's sensitivity to the robot's relocalization time. The environment image captured by the target sensor is globally matched with images in the prior environment image library to obtain the robot's pre-update pose corresponding to the environment image captured by the target sensor. The environment image captured by each sensor is then locally matched with the surrounding environment image corresponding to each pre-update robot pose in the prior environment image library to obtain the relocalization result. This approach can meet the needs of users with different time sensitivities.

[0090] In one embodiment, locally matching each environment image with the surrounding environment image corresponding to each pre-update robot posture in the prior environment image library to obtain a score value for each updated robot posture of the robot includes:

[0091] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the initial score value of each robot posture before update;

[0092] The score values ​​of the robot's updated postures are obtained according to the sensor weight values ​​corresponding to the sensors and the initial score values ​​of the robot's postures before the updates.

[0093] The sensor weight value corresponding to each sensor can be determined by the operator of the user terminal that establishes a communication connection with the robot based on actual needs. For example, the sensor weight value corresponding to each sensor can be determined based on the operator's performance preference for each sensor, or based on the robot's current environment.

[0094] It is understandable that different types of sensors experience different levels of performance interference in the same operating environment. For example, taking a robot equipped with two types of sensors, an RGB camera and a LiDAR, as an example, if the robot's current operating environment contains many corners and pillars, the RGB camera can better demonstrate its performance advantages compared to the LiDAR in this environment. However, if the robot's current operating environment has poor lighting conditions, the LiDAR can better demonstrate its performance advantages compared to the RGB camera in this environment. Therefore, to prevent the relocalization results from being affected by the performance interference experienced by each sensor, in one embodiment, the method further includes: obtaining performance information of each sensor and information about the robot's current environment; determining an environmental adaptation value for each sensor in the robot's current environment based on the performance information of each sensor; determining a sensor weight value corresponding to each sensor based on the environmental adaptation value corresponding to each sensor, wherein the environmental adaptation value corresponding to each sensor is positively correlated with the sensor weight value corresponding to each sensor; the environmental adaptation value is used to represent the degree of performance of each sensor in the robot's current environment, and the environmental adaptation value is positively correlated with the degree of performance. Therefore, in the relocalization task of the robot's current posture, the sensor with greater performance advantages in the current environment can better exert its performance advantages, thereby making the relocalization result more accurate and reliable, and also making the robot more flexible in performing the relocalization task of the current posture.

[0095] In this embodiment, when locally matching each environment image with the surrounding environment image corresponding to each pre-update robot pose in the prior environment image library to obtain scores for each updated robot pose, the initial scores for each pre-update robot pose are first obtained by locally matching the surrounding environment image corresponding to each pre-update robot pose in the prior environment image library. Then, scores for each updated robot pose are obtained based on the sensor weights corresponding to each sensor and the initial scores for each pre-update robot pose. Consequently, the scores for each updated robot pose are determined simultaneously based on the pose matching degree and the sensor weights. This provides a more reliable scoring mechanism for determining the scores for each updated robot pose, thereby ensuring the accuracy and reliability of the relocalization results for the robot's current pose.

[0096] In one embodiment, the above-mentioned determination of the relocalization result of the robot's current posture according to the score value of each updated robot posture includes:

[0097] Determining a first score value and a second score value that meet a score condition according to the score value of the updated robot posture of each robot, wherein the first score value is greater than the second score value;

[0098] If the difference between the first score value and the second score value is greater than the third preset threshold, and the first score value is greater than the fourth preset threshold, it is determined that the repositioning result for the current posture is a successful repositioning.

[0099] Here, satisfying the score condition may mean that the corresponding scores are the two highest scores among the scores of each updated robot posture, that is, the first score and the second score are the highest and second highest scores among all scores, respectively. Specifically, if the difference between the first score and the second score is less than or equal to the third preset threshold, it means that the relocalization result of the current robot posture is not unique; if the first score is less than or equal to the fourth preset threshold, it means that the updated robot posture does not have a corresponding posture with high similarity in the prior environment image library. Therefore, either of these two situations will result in a relocalization failure.

[0100] Specifically, determining the relocation result of the current posture of the robot may be determining the current posture as the posture corresponding to the relocation result, or determining the updated robot posture of the robot corresponding to the first score value as the posture corresponding to the relocation result.

[0101] In this embodiment, when determining the relocalization result for the robot's current posture based on the scores of each updated robot posture, first, based on the scores of each updated robot posture, a first score and a second score that satisfy a score condition are determined. If the difference between the first score and the second score is greater than a third preset threshold, and the first score is greater than a fourth preset threshold, then the relocalization result for the current posture is determined to be a successful relocalization. Thus, by determining that the first score is greater than the fourth preset threshold and the second score is less than the first score that satisfy the score condition, and simultaneously determining that the difference between the first score and the second score is greater than the third preset threshold, then determining that the relocalization is successful, this avoids the undesirable situation of relocalization failure caused by the dissimilarity between each updated robot posture and the posture corresponding to the image in the prior environment image library, or the non-uniqueness of the relocalization result, thereby ensuring that the relocalization result for the robot's current posture is accurate and reliable.

[0102] In one embodiment, after determining that the relocation result of the robot's current posture is successful, the method further includes:

[0103] Set the robot's current posture to the robot's corresponding preset power-on posture. The preset power-on posture is the corresponding posture of the robot when it is restarted.

[0104] In this embodiment, after determining that the relocation result of the robot's current posture is successful, the robot's current posture is set as the corresponding posture when the robot is restarted. Therefore, when the robot's posture changes due to factors such as system shutdown, power outage, or manual movement, the robot can re-determine its position in the map when it is relocated, thereby ensuring that the robot will not lose its positioning or be mispositioned due to emergencies.

[0105] Taking a robot equipped with three sensors, namely an RGB camera, a depth camera, and a lidar, as an example, as shown in FIG3 , the following describes the application process of the multi-sensor based relocalization method in combination with a detailed embodiment:

[0106] First, the robot acquires RGB images, depth images, and laser point clouds collected by the RGB camera, depth camera, and lidar, respectively.

[0107] Then, the robot globally matches the RGB image, depth image, and laser point cloud with the images in the prior environment image library, respectively, to obtain the initial robot poses corresponding to K environment images, where K is the number of environment images whose pose matching degree between the current pose corresponding to each environment image and the preset pose corresponding to the image in the prior environment image library is greater than a second preset threshold; the robot performs similarity matching on the initial robot poses corresponding to the K environment images, and merges the initial robot poses corresponding to the similarities higher than the first preset threshold to obtain the robot poses before the update corresponding to the RGB camera, depth camera, and lidar;

[0108] The robot locally matches the RGB image, depth image, and laser point cloud with the surrounding environment image corresponding to each robot pose before the update in the prior environment image library. Thus, the robot obtains the score value of each updated robot pose of the robot according to the sensor weight value corresponding to each sensor and the initial score value of each robot pose before the update, which are score value 1, score value 2, and score value 3 respectively. Score value 1 is the score value of the updated robot pose 1 corresponding to the robot pose 1 before the update, and score value 2 and score value 3 are similar and are not repeated here.

[0109] The robot determines the highest score and the secondary score based on the score values ​​of each updated robot posture; if the difference between the highest score and the secondary score is greater than the third preset threshold, and the highest score is greater than the fourth preset threshold, then the relocation result for the current posture is determined to be a successful relocation, and the current posture is determined to be the posture corresponding to the relocation result.

[0110] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0111] Based on the same inventive concept, embodiments of the present application further provide a multi-sensor relocation device for implementing the multi-sensor relocation method described above. The implementation solution provided by this device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more multi-sensor relocation device embodiments provided below can be found in the above-mentioned limitations of the multi-sensor relocation method, and will not be repeated here.

[0112] In one embodiment, as shown in FIG4 , a multi-sensor based relocation apparatus is provided, comprising: an acquisition module 1002 , a global matching module 1004 , a local matching module 1006 , and a determination module 1008 , wherein:

[0113] The acquisition module 1002 is used to acquire the environment image collected by each sensor.

[0114] The global matching module 1004 is used to perform global matching on each environment image with the images in the prior environment image library to obtain the robot pose before update corresponding to each sensor.

[0115] The local matching module 1006 is used to perform local matching between each environment image and the surrounding environment image corresponding to each pre-update robot posture in the prior environment image library to obtain a score value of each updated robot posture of the robot.

[0116] The determination module 1008 is used to determine the relocation result of the robot's current posture according to the score value of each updated robot posture.

[0117] In one embodiment, in globally matching each environment image with images in the prior environment image library to obtain each pre-update robot pose corresponding to the sensor, the global matching module 1004 is further configured to:

[0118] Perform global matching between each environment image and the images in the prior environment image library to obtain the initial robot pose corresponding to each environment image;

[0119] Similarity matching is performed on each initial robot posture, and corresponding initial robot postures with similarities higher than a first preset threshold are merged to obtain each pre-update robot posture.

[0120] In one embodiment, in globally matching each environment image with images in the prior environment image library to obtain each pre-update robot pose corresponding to the sensor, the global matching module 1004 is further configured to:

[0121] Globally match each environment image with images in the prior environment image library to obtain initial robot poses corresponding to K environment images, respectively; K is the number of environment images in the current pose corresponding to each environment image for which the pose matching degree between the current pose corresponding to the image in the prior environment image library and the preset pose corresponding to the image in the prior environment image library is greater than a second preset threshold;

[0122] Perform similarity matching with each initial robot pose, including:

[0123] Perform similarity matching on the initial robot poses corresponding to K environment images.

[0124] In one embodiment, as shown in FIG5 , the method further includes a statistics module 1010, which is configured to:

[0125] Acquire historical environmental images collected by each sensor;

[0126] Perform global matching on each historical environment image and the images in the prior environment image library, and count the global matching time corresponding to each historical environment image;

[0127] Determine a target sensor, where the target sensor is a sensor whose global matching time corresponding to the collected historical environment image meets a preset time condition;

[0128] In terms of globally matching each environment image with the images in the prior environment image library to obtain the robot poses before updating corresponding to each sensor, the global matching module 1004 is further used to:

[0129] The environment image captured by the target sensor is globally matched with the image in the prior environment image library to obtain the robot pose before update corresponding to the target sensor.

[0130] In one embodiment, in locally matching each environment image with the surrounding environment image corresponding to each pre-update robot pose in the prior environment image library to obtain a score value for each updated robot pose, the local matching module 1006 is configured to:

[0131] Perform local matching between each environment image and the surrounding environment image corresponding to each robot posture before update in the prior environment image library to obtain the initial score value of each robot posture before update;

[0132] The score values ​​of the robot's updated postures are obtained according to the sensor weight values ​​corresponding to the sensors and the initial score values ​​of the robot's postures before the updates.

[0133] In one embodiment, in determining the relocalization result of the robot's current posture according to the score value of each updated robot posture, the determination module 1008 is further configured to:

[0134] Determining a first score value and a second score value that meet a score condition according to the score values ​​of each updated robot posture, wherein the first score value is greater than the second score value;

[0135] If the difference between the first score value and the second score value is greater than the third preset threshold, and the first score value is greater than the fourth preset threshold, it is determined that the repositioning result for the current posture is a successful repositioning.

[0136] In one embodiment, as shown in FIG5 , after the determination module 1008 determines that the relocation result of the robot's current posture is successful, the apparatus further includes a setting module 1012, which is configured to:

[0137] Set the robot's current posture to the robot's corresponding preset power-on posture. The preset power-on posture is the corresponding posture of the robot when it is restarted.

[0138] Each module in the multi-sensor-based relocation device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0139] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 6. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store prior environment image library data. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, a multi-sensor based relocation method is implemented.

[0140] In one embodiment, a computer device is provided, which may be a terminal. Its internal structure diagram may be as shown in FIG7 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and computer program in the non-volatile storage medium to run. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals via wired or wireless communication, where the wireless communication may be implemented via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a multi-sensor-based relocation method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0141] Those skilled in the art will understand that the structure shown in FIG7 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0146] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0147] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-sensor based relocalization method applied to a robot, where the robot includes at least two different types of sensors, and the method includes: Obtaining environmental images respectively collected by each of the sensors; Globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the pre-update robot poses corresponding to each of the sensors; Locally matching each of the environmental images with the surrounding environmental images corresponding to each of the pre-update robot poses in the prior environmental image library to obtain the score values of each of the post-update robot poses of the robot; Determining the relocalization result of the current pose of the robot according to the score values of each of the post-update robot poses.

2. The method according to claim 1, wherein The step of globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the pre-update robot poses corresponding to each of the sensors includes: Globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the initial robot poses corresponding to each of the environmental images; Performing similarity matching on each of the initial robot poses, and merging the corresponding initial robot poses with a similarity higher than a first preset threshold to obtain each of the pre-update robot poses.

3. The method according to claim 2, characterized in that, The step of globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the initial robot poses corresponding to each of the environmental images includes: Globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the initial robot poses corresponding to K environmental images, where K is the number of environmental images whose pose matching degree between the current pose corresponding to each of the environmental images and the preset pose corresponding to the images in the prior environmental image library is greater than a second preset threshold; The step of performing similarity matching on each of the initial robot poses includes: Performing similarity matching on the initial robot poses corresponding to the K environmental images.

4. The method according to claim 1, wherein The method further includes: Obtaining historical environmental images respectively collected by each of the sensors; Globally matching each of the historical environmental images with the images in the prior environmental image library and respectively counting the global matching times corresponding to each of the historical environmental images; Determining a target sensor, where the target sensor is a sensor whose global matching time corresponding to the collected historical environmental image meets a preset time condition; The step of globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the pre-update robot poses corresponding to each of the sensors includes: Globally matching the environmental image collected by the target sensor with the images in the prior environmental image library to obtain the pre-update robot pose corresponding to the target sensor.

5. The method according to claim 1, characterized in that The step of locally matching each of the environmental images with the surrounding environmental images corresponding to each of the pre-update robot poses in the prior environmental image library to obtain the score values of each of the post-update robot poses of the robot includes: Perform local matching between each of the environmental images and the surrounding environmental images corresponding to each of the pre-update robot poses in the prior environmental image library to obtain the initial score values of each of the pre-update robot poses of the robot; Based on the sensor weight values corresponding to each of the sensors and the initial score values of each of the pre-update robot poses, obtain the score values of each of the post-update robot poses of the robot.

6. The method according to any one of claims 1-5, characterized in that, The determining the relocalization result of the current pose of the robot according to the score values of each of the post-update robot poses includes: Determine a first score value and a second score value that meet the score conditions according to the score values of each of the post-update robot poses; the first score value is greater than the second score value; If the difference between the first score value and the second score value is greater than a third preset threshold and the first score value is greater than a fourth preset threshold, determine that the relocalization result for the current pose is successful relocalization.

7. The method according to any one of claims 1-5, characterized in that, After determining that the relocalization result of the current pose of the robot is successful relocalization, the method further includes: Set the current pose of the robot to the preset startup pose corresponding to the robot, and the preset startup pose is the corresponding pose of the robot when restarting.

8. The method according to claim 1, characterized in that The robot includes at least two of an RGB sensor, a depth sensor, and a lidar.

9. The method according to claim 1, characterized in that The globally matching each of the environmental images with the images in the prior environmental image library to respectively obtain the pre-update robot poses corresponding to each of the sensors includes: Match the poses corresponding to each of the environmental images respectively collected by each of the sensors with the poses corresponding to the images in the prior environmental image library, and determine from the prior environmental image library the poses whose pose matching degree with the pose of the environmental image is greater than a first preset matching degree; Determine the poses whose pose matching degree with the pose of the environmental image is greater than the first preset matching degree as the pre-update robot poses, so as to respectively obtain the pre-update robot poses corresponding to each sensor.

10. The method according to claim 1, wherein The locally matching each of the environmental images with the surrounding environmental images corresponding to each of the pre-update robot poses in the prior environmental image library to obtain the score values of each of the post-update robot poses of the robot includes: Use each of the sensors to respectively obtain the pose matching degree between the pose of each of the environmental images and the pose corresponding to the surrounding environmental image corresponding to each of the pre-update robot poses in the prior environmental image library; Determine the score values corresponding to each of the pose matching degrees according to the mapping relationship between the pose matching degree and the score value, so as to obtain the score values of each of the post-update robot poses of the robot.

11. The method according to claim 1, wherein The relocalization result of the current pose of the robot includes successful relocalization and failed relocalization; Wherein, successful relocalization means that the located pose corresponding to the relocalization result is unique in the prior environmental image library; failed relocalization means that the located pose corresponding to the relocalization result is not unique in the prior environmental image library.

12. The method according to claim 1, characterized in that, The determining the relocalization result of the current pose of the robot according to the score values of each of the post-update robot poses includes: Based on the score values of each updated robot pose, determine the localization pose corresponding to the relocalization result of the current robot pose as the updated robot pose with the highest score value.

13. The method according to claim 11, wherein If the relocalization result of the current robot pose is relocalization failure, the method further includes: Generating an alarm signal and sending the alarm signal to a user terminal that has established a communication connection relationship with the robot.

14. The method according to claim 4, characterized in that The satisfaction of the preset time condition means that among the global matching times corresponding to the historical environment images collected by each sensor, the global matching time corresponding to the historical environment image collected by the target sensor is the shortest or less than the preset matching time.

15. The method according to claim 1, characterized in that Before globally matching each of the environment images with the images in the prior environment image library, the method further includes: Obtaining the required time for relocalization corresponding to the relocalization task; The globally matching each of the environment images with the images in the prior environment image library to respectively obtain the pre-update robot poses corresponding to each sensor includes: If the required time for relocalization is greater than the preset required time, then globally match each of the environment images collected by each sensor with the images in the prior environment image library to respectively obtain the pre-update robot poses corresponding to each sensor; If the required time for relocalization is less than or equal to the preset required time, then globally match the environment image collected by the target sensor with the images in the prior environment image library to obtain the Pre-update robot pose.

16. The method according to claim 5, characterized in that, The sensor weight value corresponding to each sensor is determined according to the environment where the robot is currently located.

17. The method according to claim 6, wherein The first score value and the second score value are respectively the highest and the second highest score values among all the score values.

18. A multi-sensor based relocalization device applied to a robot, the robot including at least two different types of sensors, the device including: An acquisition module, configured to acquire environment images respectively collected by each of the sensors; A global matching module, configured to globally match each of the environment images with the images in the prior environment image library to respectively obtain the pre-update robot poses corresponding to each of the sensors; A local matching module, configured to locally match each of the environment images with the surrounding environment images corresponding to each of the pre-update robot poses in the prior environment image library to obtain the score values of each updated robot pose of the robot; A determination module, configured to determine the relocalization result of the current robot pose according to the score values of each updated robot pose.

19. A computer device, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the method according to any one of claims 1 to 17 when executing the computer program.

20. A computer-readable storage medium, having a computer program stored thereon, and the computer program implementing the steps of the method according to any one of claims 1 to 17 when being executed by a processor.

Images