A relocation method, apparatus and electronic device

Abstract

Embodiments of the present application provide a relocation method and device and electronic equipment, and relate to the technical field of image processing. Based on a first frame image collected by an image collection device carried by a target robot, an initial reference pose of the target robot is determined; a plurality of reference images collected by the image collection device are obtained, and based on the initial reference pose, a reference pose corresponding to each reference image is determined; a local map of a region is constructed using the plurality of reference images and the reference pose corresponding to each reference image, and a target sub-map matching a local map template of the region is determined; similarity matching is performed on a current image collected by the image collection device and each reference image associated with the target sub-map, a target reference image matched with the current image is determined, and a relocation pose of the target robot relative to an a priori map is determined according to a target pose associated with the target reference image. By applying the scheme provided in the embodiments of the present application, the mobile robot can be relocated.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a repositioning method, apparatus and electronic device. Background Technology

[0002] With the continuous development of artificial intelligence technology, mobile robots are being widely used in various scenarios. For example, in factories, inspection robots are used for inspections; in homes, robotic vacuum cleaners are used to clean floors.

[0003] Typically, when a mobile robot first enters a certain area, it can first explore that area and construct an exploration map of the area, which can also be called a prior map. In this way, while performing a designated task in that area, the mobile robot can determine its pose relative to the prior map in real time. Here, pose refers to the mobile robot's position and orientation, and the process of the mobile robot determining its pose relative to the prior map can be called the mobile robot's localization process.

[0004] However, in many cases, mobile robots experience location loss, meaning they cannot determine their pose relative to the prior map. For example, manually moving a mobile robot working in a designated area to another location within that area can cause location loss; another example is when a mobile robot is stopped and then restarted, which can also lead to location loss.

[0005] Therefore, how to relocalize a mobile robot that has lost its localization is an urgent problem to be solved. Relocalization refers to re-determining the pose of the mobile robot relative to the prior map when the robot loses its localization. Summary of the Invention

[0006] The purpose of this application is to provide a relocation method, apparatus, and electronic device for relocating a mobile robot that has lost its location. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a relocation method, the method comprising:

[0008] The initial reference pose of the target robot is determined based on the first frame image captured by the image acquisition device on the target robot.

[0009] The image acquisition device acquires multiple frames of reference images, and based on the initial reference pose, the reference pose of the target robot when acquiring each frame of reference image is determined, thus obtaining the reference pose corresponding to each frame of reference image.

[0010] Using the multi-frame reference images and the reference pose corresponding to each frame reference image, a local map of the area where the target robot is currently located is constructed, and a target sub-map matching the local map template is determined in a preset prior sub-map; wherein, the prior sub-map is obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, and the layout sub-map is obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the area segmentation line;

[0011] The current image acquired by the image acquisition device is matched with each reference image associated with the target sub-map to determine the target reference image that matches the current image. Based on the target pose associated with the target reference image, the repositioning pose of the target robot relative to the prior map is determined.

[0012] Optionally, in one specific implementation, the image acquisition device is a binocular camera, and determining the initial reference pose of the target robot based on the first frame image acquired by the image acquisition device mounted on the target robot includes:

[0013] Based on the first frame of synchronized left and right eye images acquired by the image acquisition device on the target robot, the initial reference pose of the target robot is determined.

[0014] Optionally, in one specific implementation, acquiring the multiple reference frames acquired by the image acquisition device includes:

[0015] Multiple keyframes acquired by the image acquisition device are used as multi-frame reference images;

[0016] The multiple keyframes include: multiple frames of images acquired at preset time intervals, multiple frames of images acquired at preset movement distances of the target robot, or multiple frames of images including preset image features.

[0017] Optionally, in one specific implementation, the method for constructing the prior sub-map includes:

[0018] Obtain the prior map and the layout sub-map;

[0019] In the prior map, template matching is performed on each layout sub-map to obtain the prior sub-map that matches the layout sub-map.

[0020] Optionally, in one specific implementation, the step of performing template matching on each layout sub-map in the prior map to obtain the prior sub-map matched by that layout sub-map includes:

[0021] In the prior map, template matching is performed on each layout sub-map to obtain the initial sub-map that matches the layout sub-map;

[0022] Obtain the connection relationship corresponding to each layout sub-map and the traversable relationship corresponding to each initial sub-map. The connection relationship corresponding to each layout sub-map is the connection relationship between the area represented by each layout sub-map and the traversable passage of the area represented by the layout sub-map. The traversable relationship corresponding to each initial sub-map is the traversable relationship between each initial sub-map and adjacent initial sub-maps, determined based on the movement trajectory of the target robot when constructing the prior map.

[0023] For each layout submap, the initial submap corresponding to the layout submap is corrected using the connection relationship corresponding to the layout submap and the traversability relationship corresponding to the initial submap corresponding to the layout submap, so as to obtain the prior submap matched by the layout submap.

[0024] Optionally, in one specific implementation, determining the target sub-map that matches the local map template of the region from the preset prior sub-map includes:

[0025] For each prior sub-map, template matching is performed between the prior sub-map and the local area map, and template matching is performed between the layout sub-map matched by the prior sub-map and the local area map to obtain the matching result;

[0026] Based on the matching results, the target sub-map is determined in the preset prior sub-map.

[0027] Optionally, in one specific implementation, the step of performing similarity matching between the current image acquired by the image acquisition device and each reference image associated with the target sub-map to determine the target reference image matching the current image includes:

[0028] The most recent keyframe acquired by the image acquisition device is matched with each reference image associated with the target sub-map to determine the target reference image that matches the current image.

[0029] Optionally, in one specific implementation, the pose associated with each reference image is: the pose of the image acquisition device relative to the prior map when the image acquisition device acquires the reference image;

[0030] Determining the repositioning pose of the target robot relative to the prior map based on the target pose associated with the target reference image includes:

[0031] Based on the relative pose of the image acquisition device and the target robot, pose transformation is performed on the target pose associated with the target reference image to obtain the repositioning pose of the target robot relative to the prior map.

[0032] Secondly, embodiments of this application provide a robot, the robot comprising:

[0033] An image acquisition device, used to acquire images of the area where the robot is located;

[0034] A processor is configured to execute, based on an image acquired by the image acquisition device, the steps of any of the relocation method embodiments described in the first aspect of this application.

[0035] Thirdly, embodiments of this application provide a repositioning device, the device comprising:

[0036] The initial pose determination module is used to determine the initial reference pose of the target robot based on the first frame image captured by the image acquisition device mounted on the target robot.

[0037] The reference pose determination module is used to acquire multiple frames of reference images acquired by the image acquisition device, and based on the initial reference pose, determine the reference pose of the target robot when acquiring each frame of reference image, so as to obtain the reference pose corresponding to each frame of reference image.

[0038] The local map construction module is used to construct a local map of the area where the target robot is currently located using the multi-frame reference images and the reference pose corresponding to each frame reference image, and to determine a target sub-map that matches the local map template in a preset prior sub-map; wherein, the prior sub-map is obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, and the layout sub-map is obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the area division lines;

[0039] The relocalization module is used to perform similarity matching between the current image acquired by the image acquisition device and each reference image associated with the target sub-map, determine the target reference image that matches the current image, and determine the relocalization pose of the target robot relative to the prior map based on the target pose associated with the target reference image.

[0040] Optionally, in one specific implementation, the image acquisition device is a binocular camera, and the initial pose determination module is specifically used for:

[0041] Based on the first frame of synchronized left and right eye images acquired by the image acquisition device on the target robot, the initial reference pose of the target robot is determined.

[0042] Optionally, in one specific implementation, the reference pose determination module is specifically used for:

[0043] Multiple keyframes acquired by the image acquisition device are used as multi-frame reference images;

[0044] The multiple keyframes include: multiple frames of images acquired at preset time intervals, multiple frames of images acquired at preset movement distances of the target robot, or multiple frames of images including preset image features.

[0045] Optionally, in one specific implementation, the apparatus further includes a priori sub-map construction module, the priori sub-map construction module comprising:

[0046] The acquisition submodule is used to acquire the prior map and the layout submap;

[0047] The prior submap construction submodule is used to perform template matching on each layout submap in the prior map to obtain the prior submap matched by the layout submap.

[0048] Optionally, in one specific implementation, the prior sub-map construction sub-module is specifically used for:

[0049] In the prior map, template matching is performed on each layout sub-map to obtain the initial sub-map that matches the layout sub-map;

[0050] Obtain the connection relationship corresponding to each layout sub-map and the traversable relationship corresponding to each initial sub-map. The connection relationship corresponding to each layout sub-map is the connection relationship between the area represented by each layout sub-map and the traversable passage of the area represented by the layout sub-map. The traversable relationship corresponding to each initial sub-map is the traversable relationship between each initial sub-map and adjacent initial sub-maps, determined based on the movement trajectory of the target robot when constructing the prior map.

[0051] For each layout submap, the initial submap corresponding to the layout submap is corrected using the connection relationship corresponding to the layout submap and the traversability relationship corresponding to the initial submap corresponding to the layout submap, so as to obtain the prior submap matched by the layout submap.

[0052] Optionally, in one specific implementation, the local map construction module is specifically used for:

[0053] For each prior sub-map, template matching is performed between the prior sub-map and the local area map, and template matching is performed between the layout sub-map matched by the prior sub-map and the local area map to obtain the matching result;

[0054] Based on the matching results, the target sub-map is determined in the preset prior sub-map.

[0055] Optionally, in one specific implementation, the relocation module is specifically used for:

[0056] The most recent keyframe acquired by the image acquisition device is matched with each reference image associated with the target sub-map to determine the target reference image that matches the current image.

[0057] Optionally, in one specific implementation, the pose associated with each reference image is: the pose of the image acquisition device relative to the prior map when the image acquisition device acquires the reference image;

[0058] The relocation module is specifically used for:

[0059] Based on the relative pose of the image acquisition device and the target robot, pose transformation is performed on the target pose associated with the target reference image to obtain the repositioning pose of the target robot relative to the prior map.

[0060] Fourthly, embodiments of this application provide an electronic device, including:

[0061] Memory, used to store computer programs;

[0062] When a processor executes a program stored in memory, it implements the steps of any of the above method embodiments.

[0063] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above method embodiments.

[0064] Sixthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of any of the above method embodiments.

[0065] Beneficial effects of the embodiments in this application:

[0066] As can be seen from the above, by applying the solution provided in the embodiments of this application, in order to relocate the target robot, the spatial layout map of the space to which the target robot belongs can be divided in advance according to the area division line to obtain a layout sub-map. Then, the prior map of the space to which the target robot belongs can be divided using the above layout sub-map to obtain a prior sub-map. In this way, when relocalizing the target robot, the initial reference pose of the target robot can be determined first based on the first frame image captured by the image acquisition device on the target robot. Then, multiple reference images captured by the image acquisition device can be acquired, and the reference pose of the target robot when capturing each reference image can be determined based on the initial reference pose, thus obtaining the reference pose corresponding to each reference image. Next, a local map of the area where the target robot is currently located can be constructed using the multiple reference images and the reference pose corresponding to each reference image, and a target sub-map that matches the local map template can be determined in the preset prior sub-map. Furthermore, similarity matching can be performed between the current image captured by the image acquisition device and each reference image associated with the target sub-map to determine the target reference image that matches the current image. Thus, the relocalization pose of the target robot relative to the prior map can be determined based on the target pose associated with the target reference image.

[0067] Based on this, the solution provided in the embodiments of this application can achieve relocalization of the target robot. Furthermore, template matching between the local map and the prior sub-map can improve the accuracy of regional localization, thereby reducing the possibility of relocalization failure due to regional mislocalization and improving the accuracy of relocalization. Similarity matching between the current image and each reference image of the target sub-map can further improve the accuracy of relocalization. In addition, since only the current image needs to be matched with each reference image of the target sub-map, without traversing all reference images associated with the complete prior map for similarity matching, the computational load of relocalization can be reduced, thereby improving relocalization efficiency.

[0068] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0070] Figure 1 A schematic diagram of a mobile robot provided in an embodiment of this application;

[0071] Figure 2 A flowchart illustrating a method for constructing a priori submap provided in an embodiment of this application;

[0072] Figure 3 A flowchart illustrating a relocation method provided in an embodiment of this application;

[0073] Figure 4 A flowchart illustrating a specific example of the relocation method provided in this application embodiment;

[0074] Figure 5 A schematic diagram of the structure of a robot provided in an embodiment of this application;

[0075] Figure 6 This is a schematic diagram of the structure of a repositioning device provided in an embodiment of this application;

[0076] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0078] In many cases, mobile robots experience localization loss, meaning they cannot determine their pose relative to a prior map. Therefore, how to relocalize a mobile robot that has experienced localization loss is a pressing issue. Relocalization refers to the process of re-determining the mobile robot's pose relative to a prior map when localization loss occurs.

[0079] To address the aforementioned technical problems, this application provides a relocation method.

[0080] This method can be applied to various application scenarios that require repositioning of mobile robots, such as repositioning mobile robots that have been moved to other locations during operation, and repositioning cleaning robots that have been restarted after being stopped.

[0081] Furthermore, this method can be applied to mobile robots equipped with image acquisition devices, such as those equipped with data processing modules. After acquiring images using their own image acquisition devices, these robots execute the method using their own data processing modules. It can also be applied to image acquisition devices mounted on target robots that have data processing capabilities, such as those equipped with data processing modules. After acquiring images, these devices execute the method using their own data processing modules. Furthermore, it can be applied to various electronic devices capable of communicating with mobile robots equipped with image acquisition devices and providing relocation services to the mobile robots, such as management platforms. When the executing entity of this method is an electronic device, the electronic device can be an independent electronic device or a cluster of multiple electronic devices, hereinafter referred to as electronic devices.

[0082] Therefore, the embodiments of this application do not specifically limit the application scenarios and execution entities of the method.

[0083] A relocation method provided in this application embodiment may include the following steps:

[0084] The initial reference pose of the target robot is determined based on the first frame image captured by the image acquisition device on the target robot.

[0085] The image acquisition device acquires multiple frames of reference images, and based on the initial reference pose, the reference pose of the target robot when acquiring each frame of reference image is determined, thus obtaining the reference pose corresponding to each frame of reference image.

[0086] Using the multi-frame reference images and the reference pose corresponding to each frame reference image, a local map of the area where the target robot is currently located is constructed, and a target sub-map matching the local map template is determined in a preset prior sub-map; wherein, the prior sub-map is obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, and the layout sub-map is obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the area segmentation line;

[0087] The current image acquired by the image acquisition device is matched with each reference image associated with the target sub-map to determine the target reference image that matches the current image. Based on the target pose associated with the target reference image, the repositioning pose of the target robot relative to the prior map is determined.

[0088] As can be seen from the above, by applying the solution provided in the embodiments of this application, in order to relocate the target robot, the spatial layout map of the space to which the target robot belongs can be divided in advance according to the area division line to obtain a layout sub-map. Then, the prior map of the space to which the target robot belongs can be divided using the above layout sub-map to obtain a prior sub-map. In this way, when relocalizing the target robot, the initial reference pose of the target robot can be determined first based on the first frame image captured by the image acquisition device on the target robot. Then, multiple reference images captured by the image acquisition device can be acquired, and the reference pose of the target robot when capturing each reference image can be determined based on the initial reference pose, thus obtaining the reference pose corresponding to each reference image. Next, a local map of the area where the target robot is currently located can be constructed using the multiple reference images and the reference pose corresponding to each reference image, and a target sub-map that matches the local map template can be determined in the preset prior sub-map. Furthermore, similarity matching can be performed between the current image captured by the image acquisition device and each reference image associated with the target sub-map to determine the target reference image that matches the current image. Thus, the relocalization pose of the target robot relative to the prior map can be determined based on the target pose associated with the target reference image.

[0089] Based on this, the solution provided in the embodiments of this application can achieve relocalization of the target robot. Furthermore, template matching between the local map and the prior sub-map can improve the accuracy of regional localization, thereby reducing the possibility of relocalization failure due to regional mislocalization and improving the accuracy of relocalization. Similarity matching between the current image and each reference image of the target sub-map can further improve the accuracy of relocalization. In addition, since only the current image needs to be matched with each reference image of the target sub-map, without traversing all reference images associated with the complete prior map for similarity matching, the computational load of relocalization can be reduced, thereby improving relocalization efficiency.

[0090] To facilitate understanding of the relocation method provided in the embodiments of this application, the prior map, spatial layout map, and prior sub-map involved in the embodiments of this application will be explained first.

[0091] Typically, when a mobile robot first enters a certain area, it can explore that area and build a priori map of that area.

[0092] The process of constructing the prior map described above can be called SLAM (Simultaneous Localization and Mapping). In other words, when a mobile robot first enters a certain area, it can use its current location as its initial position within that area. Subsequently, the mobile robot can start from this initial position and move within the area to explore it. During area exploration, the mobile robot can use its onboard image acquisition device to capture multiple frames of images of the area. Furthermore, for each frame, the robot's pose at the time the image acquisition device captured that frame can be determined as the corresponding pose for that frame.

[0093] Then, after completing the exploration of the area, multiple frames of images acquired by the image acquisition device and the pose corresponding to each frame can be used to construct an exploration map of the area, which can serve as a priori map of the area.

[0094] The image acquisition device carried by the mobile robot can be either a monocular camera or a binocular camera, which is reasonable and is not specifically limited in this embodiment.

[0095] Furthermore, since the aforementioned prior map is constructed based on multiple frames of images and the pose corresponding to each frame, each map region of the prior map is associated with multiple frames of images, and each frame of images is associated with the pose of the mobile robot.

[0096] The pose of the mobile robot associated with each frame of image refers to the pose of the mobile robot when the image acquisition device on the mobile robot acquires that frame of image.

[0097] Furthermore, the pose of a mobile robot refers to its position and orientation. Specifically, the position refers to the robot's exact location within the area, while its orientation refers to its facing direction.

[0098] Since each frame of the image is captured by an image acquisition device, the pose of the image acquisition device can be determined based on each frame of the image. Since the image acquisition device is mounted on a mobile robot, the position of the image acquisition device and the position of the mobile robot are the same. Furthermore, the pose of the image acquisition device and the pose of the mobile robot can be determined by the installation position of the image acquisition device. Thus, the pose of the image acquisition device and the pose of the mobile robot are correlated, that is, the image acquisition device and the mobile robot have relative poses.

[0099] For example, such as Figure 1As shown, when the direction of movement of the mobile robot is considered as its forward direction, the binocular camera mounted on the robot can acquire images of the area behind the robot. The angle between the camera's viewing angle and the horizontal plane is α, and the vertical field of view is β. That is, when the mobile robot moves on the ground in a designated area, the binocular camera mounted on the robot can acquire images of walls, doors, windows, etc., located within the vertical field of view of the camera behind the robot, with a vertical field of view angle β.

[0100] Each area can include various obstacles, such as walls, furniture, people, and animals. Therefore, the prior map obtained by the mobile robot during its exploration of that area can include all these obstacles.

[0101] After constructing the aforementioned prior map, the mobile robot can store it. Thus, when the mobile robot moves within the area, it can determine its pose relative to the prior map in real time; that is, it can use the prior map to determine its pose within the area and information about obstacles around its location.

[0102] Then, when the mobile robot loses its localization—that is, when it cannot determine its pose relative to the prior map—it needs to be relocalized to regain its pose in the prior map. After relocalizing its pose relative to the prior map, the mobile robot can continue to use the prior map to determine its pose in the area and information about obstacles around its location. This allows the prior map to be used to enhance the mobile robot's obstacle avoidance capabilities and localization robustness.

[0103] Since the prior map is constructed based on multiple frames of images captured by the image acquisition device on the mobile robot and the pose corresponding to each frame, the constructed prior map may be incomplete due to factors such as the acquisition frequency of the image acquisition device, the acquisition range of the image acquisition device, and the shape and size of the mobile robot itself.

[0104] For example, due to its shape limitations, the mobile robot cannot reach some locations in the area. As a result, the image acquisition device carried by the mobile robot cannot capture images of all locations in the area. Thus, the prior map of the area constructed using multiple frames of images of the area captured by the image acquisition device is incomplete.

[0105] Therefore, to improve the localization accuracy of mobile robot relocalization, the spatial layout map of the aforementioned area can be used to enrich and supplement the prior map, thereby obtaining a more complete prior map. Then, this more complete prior map can be used to relocalize the mobile robot.

[0106] The aforementioned spatial layout map refers to an image representing the spatial layout of the area. This spatial layout map may include spatial structures such as walls, doors, and windows within the area. Furthermore, these walls and windows can serve as area dividing lines, dividing the area into multiple sub-areas. Correspondingly, the aforementioned spatial layout map can be divided according to these area dividing lines to obtain multiple layout sub-maps.

[0107] For example, when using a cleaning robot to clean a house, the spatial layout of the house can be a floor plan of the house, which marks the location of each wall and each door and window in the house. Furthermore, by using the aforementioned walls and each door and window as area dividing lines, the house can be divided into different sub-areas such as a living room and a bedroom.

[0108] Furthermore, the aforementioned spatial layout diagram may also include accessible passageways between areas, such as door frames within a house; based on this, the aforementioned spatial layout diagram may also include the connection relationship between the area represented by each layout sub-map and the accessible passageways of the area represented by that layout sub-map, as the connection relationship of the layout sub-map. For example, the aforementioned spatial layout diagram may include the connection relationship between the kitchen represented by the layout sub-map S and the door frames in that kitchen.

[0109] Furthermore, since the aforementioned accessible passage can connect two different areas, for example, room A and room B can be connected through door C, the connection relationship for each layout sub-map can optionally be used to represent the connection relationship established between the layout sub-map and another layout sub-map through the accessible passage of the area represented by the layout sub-map, that is, the connection relationship between the layout sub-maps corresponding to the two areas of the accessible passage, for example: the connection relationship established between layout sub-map D and layout sub-map E through accessible passage F, etc.

[0110] Optionally, the connection between two different local submaps can also represent the traversable path connecting these two local submaps. For example, the j-th layout submap and the (j+1)-th layout submap are connected by a gate M. Thus, the connection between the j-th layout submap and the (j+1)-th layout submap can be represented as L(j, j+1). In this way, L(j, j+1) can represent both the connection between the j-th layout submap and the (j+1)-th layout submap, and the gate M connecting the j-th layout submap and the (j+1)-th layout submap.

[0111] Alternatively, after obtaining the aforementioned prior area, the prior map can be processed into a planar raster map. Furthermore, since the aforementioned spatial layout map is typically a 3D map, after obtaining the spatial layout map, it can be identified to obtain a planar representation of the spatial layout map. This planar representation of the spatial layout map can then be used to supplement the aforementioned planar raster map.

[0112] Typically, relocalization of a mobile robot within a certain area is achieved by traversing multiple frames of image data included in that area. Furthermore, the total number of image data in the entire area is greater than the number of image data in a sub-area of ​​the entire area. Thus, the computational resources required for relocalization by traversing the individual image data within the entire area are significantly higher than those required by traversing the individual image data in a sub-area of ​​the entire area.

[0113] Furthermore, to conserve computational resources, when relocalizing a mobile robot, the sub-map to which the robot's current position belongs can be determined first within the complete prior map. Then, the image data encompassed by the determined sub-region can be used to determine the robot's pose relative to that sub-map. This relocalization method, which determines the robot's pose within a smaller sub-region, requires less computation and is more efficient than directly determining the robot's pose within the complete prior map.

[0114] Based on this, before relocalizing the mobile robot, the aforementioned prior map can be divided into multiple prior sub-maps. In this way, when relocalizing the mobile robot, the prior sub-map in which the mobile robot is located can be determined first, and then the relocalization pose of the mobile robot can be determined using the determined prior sub-map.

[0115] The following section explains how the prior submap is constructed.

[0116] Alternatively, in one specific implementation, such as Figure 2 As shown, the construction method of the prior submap may include the following steps S201-S202:

[0117] S201: Obtain the prior map and layout sub-map;

[0118] S202: In the prior map, perform template matching on each layout submap to obtain the prior submap that matches the layout submap.

[0119] In this specific implementation, when constructing a priori submap for any space, the priori map and layout submap for that space can be obtained first.

[0120] As mentioned above, the prior maps are maps obtained based on the region exploration of the mobile robot. Each prior map includes map information of the space and pose information of the mobile robot. The layout sub-maps are sub-maps obtained by dividing the spatial layout map of the space according to the region dividing lines in the spatial layout map of the space.

[0121] Since the prior map and the layout region represent the same space, for each layout sub-map, a prior sub-map representing the same sub-region can be determined within the prior map. Based on this, after obtaining the prior map and layout sub-maps for the space, template matching can be performed on each layout sub-map within the prior map, thereby determining the prior sub-map whose template matches that of each layout sub-map. In this way, by performing template matching on each local sub-map, multiple prior sub-maps for the space can be obtained.

[0122] Template matching refers to determining the smallest map region in the prior map that matches the template of the aforementioned layout sub-map.

[0123] For example, when constructing a priori submap of a house, a priori map and a layout submap of the house can be obtained first. Then, for the layout submap used to represent the kitchen area of ​​the house, template matching can be performed between the layout submap and the aforementioned prior map. Thus, in the prior map, the map area that matches the template of the layout submap is determined as the priori submap that matches the layout submap, which is then obtained as the priori submap used to represent the kitchen area.

[0124] In some cases, the templates of various layout submaps may be quite similar; however, as mentioned earlier, the connection relationships corresponding to each local submap are different. Therefore, for each local submap, the connection relationships corresponding to that submap can be combined to determine a priori submap that matches that local submap in the prior map.

[0125] Based on this, optionally, a priori map, layout sub-maps, and the connection relationships corresponding to each layout sub-map can be obtained. Then, for each layout sub-map, template matching is performed on the priori map, and the corresponding connection relationships are used to determine the priori sub-map that matches the layout sub-map. The connection relationships corresponding to each layout sub-map are: the connection relationships between the area represented by each layout sub-map and the passable passages within that area.

[0126] Furthermore, in the prior map, the traversability between various sub-maps in the prior map can be determined based on the movement trajectory of the target robot. When a layout sub-map matches a certain prior sub-map, the connection relationship of the layout sub-map is consistent with the traversability relationship of the prior sub-map that the layout sub-map matches.

[0127] Based on this, in one optional implementation, step S202 above, which involves performing template matching on each layout sub-map in the prior map to obtain the prior sub-map matched by that layout sub-map, may include the following steps 11-13:

[0128] Step 11: In the prior map, perform template matching on each layout submap to obtain the initial submap that matches the layout submap;

[0129] Step 12: Obtain the connection relationship corresponding to each layout sub-map and the traversability relationship corresponding to each initial sub-map. The connection relationship corresponding to each layout sub-map is the connection relationship between the area represented by each layout sub-map and the traversable channels of the area represented by that layout sub-map. The traversability relationship corresponding to each initial sub-map is the traversability relationship between each initial sub-map and adjacent initial sub-maps, determined based on the movement trajectory of the target robot when constructing the prior map.

[0130] Step 13: For each layout submap, use the connection relationship corresponding to the layout submap and the traversability relationship corresponding to the initial submap corresponding to the layout submap to correct the initial submap corresponding to the layout submap, and obtain the prior submap matched by the layout submap.

[0131] In this specific implementation, after obtaining the prior map and the layout sub-map, template matching can be performed on each layout sub-map in the prior map to obtain the initial sub-map matched by that layout sub-map.

[0132] Then, the connection relationships corresponding to each layout sub-map and the traversability relationships corresponding to each initial sub-map can be determined. The connection relationship corresponding to each layout sub-map is the connection relationship between the area represented by each layout sub-map and the traversable channels of the area represented by that layout sub-map. The traversability relationship corresponding to each initial sub-map is the traversability relationship between each initial sub-map and adjacent initial sub-maps, determined based on the movement trajectory of the target robot when constructing the prior map.

[0133] Furthermore, the connectivity relationships corresponding to each layout submap and the traversability relationships corresponding to each initial submap can be obtained. Next, for each layout submap, the connectivity relationships corresponding to that layout submap and the traversability relationships corresponding to its initial submap can be used to correct the initial submap corresponding to that layout submap, and the corrected initial submap can be used as the prior submap matched by that layout submap.

[0134] Since the prior submap includes multiple camera pose images, and the layout submap can include walkable passages in the spatial layout, when determining the prior submap that matches the layout submap, a correspondence between walkable passages in the layout submap and camera poses in the prior submap can be established.

[0135] In this way, after constructing a priori submap of the area, if the mobile robot loses its localization in the area, it can be relocalized using the pre-constructed priori submap of the area.

[0136] The following is a detailed description of a relocation method provided by an embodiment of this application, with reference to the accompanying drawings.

[0137] Figure 3 This is a flowchart illustrating a relocation method provided in an embodiment of this application, as shown below. Figure 3 As shown, the method may include the following steps S301-S304.

[0138] S301: Determine the initial reference pose of the target robot based on the first frame image captured by the image acquisition device mounted on the target robot;

[0139] When a target robot loses its localization, and it is necessary to determine the repositioning pose of the target robot relative to the prior map, the first frame image captured by the image acquisition device on the target robot can be obtained first. Then, based on the first frame image, the initial reference pose of the target robot can be determined.

[0140] The initial reference pose of the target robot is the pose of the target robot relative to the position where the first frame image was acquired, and includes the map of the position.

[0141] After acquiring the first frame image, pose analysis can be performed on the first frame image to obtain the pose of the target robot when the image acquisition device acquires the first frame image.

[0142] Optionally, a preset pose analysis algorithm can be used to determine the pose of the target robot when the image acquisition device acquires the first frame image. The pose analysis algorithm can be a BA (Bundle Adjustment) optimization algorithm or other algorithms; all of these are reasonable and are not specifically limited in this embodiment.

[0143] When the target image acquisition device is a binocular camera, the left and right cameras of the binocular camera may be out of sync, resulting in the left-eye image acquired by the left camera being out of sync with the right-eye image acquired by the right camera. For example, the image acquisition times of the left and right cameras may be out of sync, thus resulting in the left-eye image being out of sync with the right-eye image.

[0144] When the left and right visual images are out of sync, they are not images from the same moment. Therefore, the disparity and depth values ​​calculated using these images are inaccurate. Consequently, determining the robot's pose using these images results in a significant error. To improve positioning accuracy, when using a binocular camera as the image acquisition device, before determining the initial reference pose of the target robot using the first frame acquired by the binocular camera, it is advisable to first verify whether the left and right visual images acquired by the binocular camera are synchronized.

[0145] Based on this, in one optional implementation, the image acquisition device is a binocular camera, and step S301 above may include the following step 21:

[0146] Step 21: Based on the first frame of synchronized left and right eye images acquired by the image acquisition device on the target robot, determine the initial reference pose of the target robot.

[0147] In this specific implementation, when the image acquisition device is a binocular camera, the left eye image acquired by the left eye camera of the binocular camera mounted on the target robot and the right eye image acquired by the right eye camera of the binocular camera can be acquired first. Then, it is determined whether the left eye image and the right eye image are synchronized.

[0148] Furthermore, when the left and right eye images are first determined to be synchronized, the synchronized left and right eye images are the first frame of synchronized left and right eye images captured by the binocular camera. Thus, the initial reference pose of the target robot can be determined using the first frame of synchronized left and right eye images.

[0149] In the aforementioned judgment process, whenever the left and right eye images are found to be out of sync, the binocular camera can be adjusted. Then, the adjusted left and right eye images acquired by the binocular camera are obtained, and the synchronization of the acquired left and right eye images is judged again. This continues until the first synchronized left and right eye images are determined. These synchronized images are then used as the first synchronized left and right eye images acquired by the binocular camera, and the initial reference pose of the target robot is determined using these first synchronized left and right eye images.

[0150] Optionally, the synchronization of the left and right eye images captured by the binocular camera can be determined by judging whether the left and right eye cameras are synchronized. If the left and right eye cameras are synchronized, then the left and right eye images captured by the binocular camera are synchronized; if the left and right eye cameras are not synchronized, then the left and right eye images captured by the binocular camera are also not synchronized.

[0151] In this way, since the error between the synchronized left and right images is small, the initial reference pose of the target robot determined using the first frame of synchronized left and right images is relatively accurate.

[0152] S302: Acquire multiple reference images acquired by the image acquisition device, and based on the initial reference pose, determine the reference pose of the target robot when acquiring each reference image, and obtain the reference pose corresponding to each reference image.

[0153] After determining the initial reference pose of the target robot, its movement can be controlled using this initial reference pose as a reference. Furthermore, during the robot's movement, the image acquisition device mounted on the target robot can capture multiple frames of images. In this way, the electronic equipment can acquire the multiple reference frames captured by the image acquisition device.

[0154] Optionally, each frame of the image captured by the image acquisition device can be acquired, and each frame can be used as a reference image to obtain multiple reference images.

[0155] Since image acquisition devices capture a large number of images, using each frame as a reference image and determining the reference pose of each frame requires significant computational resources. Therefore, to conserve computational resources, keyframes can be identified within the images captured by the image acquisition device, and these keyframes can be used as reference images.

[0156] Optionally, in one specific implementation, the above S302 may include the following step 31:

[0157] Step 31: Acquire multiple keyframes captured by the image acquisition device as multi-frame reference images;

[0158] Among them, multiple keyframes include: multiple frames of images acquired at preset time intervals, multiple frames of images acquired at preset moving distances of the target robot, or multiple frames of images including preset image features.

[0159] In this specific implementation, the electronic device can acquire multiple key frames acquired by the image acquisition device and use the acquired key frames as multi-frame reference images.

[0160] The aforementioned multiple keyframes may include multiple frames of images acquired by the image acquisition device at preset time intervals, multiple frames of images acquired by the image acquisition device at preset moving distances of the target robot, or multiple frames of images including preset image features.

[0161] Furthermore, the aforementioned preset time interval, preset movement distance, and preset image features can all be set according to actual needs. For example, the aforementioned preset time interval can be 3 seconds, 5 seconds, etc.; the aforementioned preset movement distance can be 1 meter, 5 meters, etc.; and the aforementioned preset image features can be obstacles, etc. These are all reasonable and are not specifically limited in this application embodiment.

[0162] In addition, optionally, the image acquisition device can acquire images at preset time intervals and send the acquired multiple frames of images to the electronic device. In this way, the electronic device can obtain multiple key frames acquired by the image acquisition device at preset time intervals and use the multiple key frames as multiple reference images.

[0163] Optionally, the electronic device can receive each frame of image acquired by the image acquisition device, and use the image that includes preset image features among the acquired multi-frame images as keyframes, thereby using the determined multiple keyframes as multi-frame reference images.

[0164] After acquiring multiple reference images, the electronic device can determine the reference pose of the target robot when the image acquisition device acquires each reference image, based on the aforementioned initial reference pose. Then, for each reference image, the reference pose of the target robot when the image acquisition device acquires that reference image is determined as the reference pose corresponding to that reference image, thus obtaining the reference pose corresponding to each reference image.

[0165] S303: Using multiple reference images and the reference pose corresponding to each reference image, construct a local map of the area where the target robot is currently located, and determine the target sub-map that matches the local map template in the preset prior sub-map.

[0166] Among them, the prior sub-map is obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, and the layout sub-map is obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the region dividing lines.

[0167] After obtaining the above-mentioned multiple reference images and determining the reference pose of each reference image, a local map of the area where the target robot is currently located can be constructed using the above-mentioned multiple reference images and the reference pose of each reference image.

[0168] After obtaining a local map of the area where the target robot is currently located, the electronic device can acquire a preset prior sub-map of the space to which the target robot belongs.

[0169] The aforementioned prior sub-map can be obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, while the aforementioned layout sub-map, as mentioned above, can be obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the region dividing lines.

[0170] Furthermore, the construction method of the aforementioned prior submap is as described above and will not be repeated here.

[0171] Optionally, when the aforementioned prior sub-map is a raster map, in order to facilitate template matching between the aforementioned regional raster map and the aforementioned prior sub-map, the aforementioned regional local map can be processed into a raster map.

[0172] In this way, after obtaining the preset prior sub-map, the electronic device can determine the target sub-map that matches the regional local map template from the aforementioned prior sub-map.

[0173] Optionally, the template similarity between the local map and each prior sub-map can be calculated separately, and the prior sub-map corresponding to the maximum template similarity can be determined as the target sub-map that matches the template of the local map.

[0174] Furthermore, since each of the aforementioned prior submaps has a layout submap that matches it, when determining the matching relationship between the local area map and each prior submap, template matching can be performed between the local area map and each prior submap, and template matching can be performed between the local area map and the layout submap that matches each prior submap, thereby improving the reliability of template matching.

[0175] Based on this, in one optional implementation, step S303 above, determining the target sub-map that matches the regional local map template in the preset prior sub-map, may include the following steps 41-42:

[0176] Step 41: For each prior sub-map, perform template matching between the prior sub-map and the local area map, and perform template matching between the layout sub-map and the local area map matched by the prior sub-map to obtain the matching results;

[0177] Step 42: Based on the matching results, determine the target sub-map in the preset prior sub-map.

[0178] In this specific implementation, after constructing a local map of the area where the target robot is currently located, for each prior sub-map, the aforementioned local map can be matched with the prior sub-map using a template, and the layout sub-map matched by the prior sub-map can be matched with the local map using a template to obtain the matching result.

[0179] Subsequently, when the local map of the region represented by the above matching result matches the prior sub-map template, and the local map of the region matches the layout sub-map template corresponding to the prior sub-map, the prior sub-map can be identified as the target sub-map corresponding to the local map of the region.

[0180] After determining the target sub-map that matches the local map template of the area, the sub-region where the target robot is located in the prior map can be determined. However, since the process of determining the target sub-map of the local map of the area where the target robot is currently located can determine the sub-region corresponding to the target robot's current location in the prior map, but cannot determine the target robot's pose relative to the prior map, the process of determining the target sub-map of the local map of the area where the target robot is currently located can be called the coarse localization process of the target robot.

[0181] Furthermore, since the prior submap is derived from the layout submap, for any prior submap, the local map of the region that matches that prior submap can also be matched with the layout submap corresponding to that prior submap. Therefore, to improve the accuracy of the determined layout submap, after identifying the target submap that matches the local map in the preset prior submaps, the matching relationship between the local map and the target submap can be verified by determining whether the local map matches the layout submap that matches the target submap.

[0182] Since both the prior map and the local area map are obtained by the target robot, and the obtained prior map and local area map may have phenomena such as blurred edges and distortion, the global deviation between the prior map and the layout map can be determined. Then, the local area map can be corrected using this global deviation.

[0183] Based on this, optionally, a priori map and a spatial layout map can be obtained. Then, the priori map and the spatial layout map are matched in shape to determine the global deviation between them. This global deviation is then used to correct the local area map. Next, among each layout sub-map, the layout sub-map that matches the target sub-map is determined, and the corrected local area map is matched with the determined layout sub-map. Furthermore, if the local area map matches the determined layout sub-map, then the local area map matches the target sub-map; if the local area map does not match the determined layout sub-map, a notification message indicating that the local area map matching has failed can be output. In this way, by matching the local area map with the layout sub-map that matches the target sub-map, the matching relationship between the local area map and the target sub-map can be verified, thereby improving the accuracy of relocation.

[0184] S304: Perform similarity matching between the current image acquired by the image acquisition device and each reference image associated with the target sub-map, determine the target reference image that matches the current image, and determine the repositioning pose of the target robot relative to the prior map based on the target pose associated with the target reference image.

[0185] Since the prior map is constructed based on multiple frames of images captured by the image acquisition device on the target robot, each prior sub-map is associated with multiple frames of images captured by the image acquisition device. Based on this, the images associated with each prior sub-map can be used as reference images.

[0186] Based on this, after determining the target sub-map that matches the aforementioned local map template, in order to determine the repositioning pose of the target robot relative to the prior map, we can determine the reference images associated with the target sub-image and obtain the current image acquired by the image acquisition device mounted on the target robot.

[0187] Then, the current image and each reference image can be compared for similarity to determine the target reference image that matches the current image. In other words, the electronic device can calculate the similarity between the current image and each reference image, and use the reference image with the highest similarity as the target reference image that matches the current image.

[0188] Optionally, in one specific implementation, step S304 may include the following step 51:

[0189] Step 51: Perform similarity matching between the most recent keyframe acquired by the image acquisition device and each reference image associated with the target sub-map to determine the target reference image that matches the current image.

[0190] In this specific implementation, the pose associated with the most recently acquired image by the image acquisition device can be used as the pose of the target robot, and the key frame can be a representative image acquired by the image acquisition device. Based on this, in order to make the repositioned pose close to the current pose of the target robot and to improve the accuracy of the positioning, the most recently acquired key frame by the image acquisition device can be obtained, and the repositioned pose of the target robot can be determined using the most recently acquired key frame.

[0191] Electronic devices can acquire the most recent keyframes captured by image acquisition devices. For example, among multiple frames of images acquired by the image acquisition device, the electronic device can determine the keyframe whose acquisition time is closest to the current moment and use it as the most recent keyframe acquired by the image acquisition device. Or, for another example, the electronic device can use the most recently received image acquired by the image acquisition device as the most recent keyframe acquired by the image acquisition device.

[0192] Subsequently, the electronic device can perform similarity matching between the most recent keyframe acquired by the aforementioned image acquisition device and each reference image associated with the target sub-map, thereby determining the target reference image that matches the current image based on the similarity matching results.

[0193] Optionally, the electronic device can calculate the similarity between the most recent keyframe acquired by the image acquisition device and each reference image, and then determine the target reference image that matches the current image from the reference image corresponding to the maximum similarity.

[0194] Typically, when an image acquisition device acquires each reference image, the target robot has a pose corresponding to that reference image; thus, each reference image is associated with a pose.

[0195] Based on this, after determining the target reference image that matches the current image, the electronic device can determine the repositioning pose of the target robot relative to the prior map according to the target pose associated with the target reference image.

[0196] Optionally, the pose associated with each of the above reference images is: when the image acquisition device acquires the reference image, the target pose associated with the above target reference image is the repositioning pose of the target robot relative to the prior map.

[0197] Optionally, in one specific implementation, the pose associated with each reference image is: the pose of the image acquisition device relative to the prior map when the image acquisition device acquires the reference image;

[0198] Step S304 above, which determines the repositioning pose of the target robot relative to the prior map based on the target pose associated with the target reference image, may include the following step 61:

[0199] Step 61: Based on the relative pose of the image acquisition device and the target robot, perform pose transformation on the target pose associated with the target reference image to obtain the repositioning pose of the target robot relative to the prior map.

[0200] In this specific implementation, the pose associated with each of the above reference images is: the pose of the image acquisition device relative to the prior map when the image acquisition device acquires the reference image.

[0201] Therefore, the target pose associated with the aforementioned target reference image is: the target pose of the image acquisition device relative to the prior map when the image acquisition device acquires the target reference image.

[0202] As mentioned earlier, the relative pose of the image acquisition device and the target robot can be determined based on their positional relationship. In other words, when determining the pose of the target image acquisition device, the pose of the target robot can be determined using the aforementioned target image acquisition device and its relative pose to the target robot.

[0203] Based on this, when determining the target pose of the image acquisition device relative to the prior map based on the aforementioned target reference image, the target pose can be transformed according to the relative pose of the image acquisition device and the target robot. That is, the target pose of the image acquisition device relative to the prior map can be transformed into the pose of the target robot relative to the prior map, thereby obtaining the repositioning pose of the target robot relative to the prior map.

[0204] In this process, the repositioning pose of the target robot relative to the prior map is determined based on the target pose associated with the target reference image. This process can be further improved by determining the pose of the target robot relative to the prior map based on the sub-region corresponding to the current location of the target robot in the prior map. Therefore, the process of determining the repositioning pose of the target robot relative to the prior map based on the target pose associated with the target reference image can be called the fine localization process of the target robot.

[0205] As can be seen from the above, the solution provided in this application can achieve relocalization of the target robot. Furthermore, template matching between the local map and the prior sub-map can improve the accuracy of regional localization, thereby reducing the possibility of relocalization failure due to regional mislocalization and increasing the success rate of relocalization. Further, similarity matching between the current image and each reference image of the target sub-map can improve the accuracy of relocalization. Moreover, since only the current image needs to be matched with each reference image of the target sub-map, without traversing all reference images associated with the complete prior map, the computational load of relocalization can be reduced, thus improving relocalization efficiency.

[0206] To facilitate understanding of the relocation method provided in the embodiments of this application, the following is combined with... Figure 4 A specific example of a robot relocation method is shown below for illustration.

[0207] like Figure 4 As shown, the floor plan in the figure is the spatial layout diagram in this application embodiment, and the global prior map is the prior map in this application embodiment. When the target robot performs a cleaning task in the designated house, its positioning is lost, therefore, it is necessary to reposition the target robot. The repositioning of the target robot may include the following steps S401-S413:

[0208] S401: Import floor plan;

[0209] S402: Import global prior map;

[0210] S403: Floor plan recognition generates a 2D map, which is then used to construct the first layer of a pyramid.

[0211] S404: Divide the map area to form sub-map sets D and the connection relationship DT between the passable passages and the sub-map sets;

[0212] S405: Estimate the second layer of the pyramid construction for a 2D raster map;

[0213] S406: Perform shape matching on two layers of maps, and combine DT segmentation of the second layer of map to obtain a sub-map set P;

[0214] S407: Determine whether the left and right eye images are synchronized; if so, proceed to step S408.

[0215] S408: Acquire left and right eye images and determine the initial pose;

[0216] S409: Create keyframe images;

[0217] S410: Local optimization, estimating robot pose;

[0218] S411: Construct a 2D raster map and form the third layer of the local map into a pyramid;

[0219] S412: The third-layer map is template-matched with the second-layer map and verified using DT to determine the target sub-map, thereby achieving coarse positioning;

[0220] S413: Perform keyframe similarity matching within the target sub-map to achieve precise localization.

[0221] Before relocalizing the target robot, a floor plan of the specified house and a global prior map obtained by the target robot during its exploration of the specified house area can be pre-imported. Then, the imported floor plan is identified to obtain a 2D (2-dimensional) map of the floor plan, which is used as the first layer of the map pyramid. Using the locations of each wall in the floor plan, the 2D map is segmented into sub-map sets D, and the connections DT between each sub-map and the passable passageways in the floor plan are determined.

[0222] The imported global prior map is estimated to obtain a 2D raster map of the global prior map, which is then used as the second layer of the map pyramid. Next, the first and second layer maps are shape-matched, and the 2D raster map is segmented using the connectivity relation DT to obtain a subset P of the global prior map.

[0223] After the target robot restarts, the left and right eye images captured by the binocular cameras on the target robot are first acquired. Then, it is determined whether the left and right eye images are synchronized. When the left and right eye images are synchronized, the first frame of the left and right eye images is acquired, and the initial pose of the target robot is determined using the first frame of the left and right eye images.

[0224] Next, multiple frames of images captured by the binocular camera are acquired, and keyframe images are determined using these images. Then, local optimization is performed using these keyframe images to determine the pose and map points of the target robot within the keyframe images. Finally, using the keyframe images and the target robot's pose, a 2D grid map is constructed to obtain a local map of the area where the target robot is currently located, and this local map is used as the third layer of the map pyramid.

[0225] The third-layer map performs template matching with the second-layer map, that is, it performs template matching between the local map and each sub-map in the sub-map set P of the global prior map, and combines the above connection relationship DT to determine the target sub-map that matches the local map, thereby achieving coarse localization of the target robot.

[0226] Next, the most recent keyframe collected by the target robot is matched with each keyframe associated with the target sub-map. The keyframe with the highest similarity to the most recent keyframe is determined as the target keyframe, and the target pose associated with the target keyframe is determined as the relocalization pose of the target robot, thereby achieving precise localization of the target robot.

[0227] Based on the same inventive concept, and corresponding to the embodiments provided in this application above... Figure 3 The present application also provides a robot, in addition to the relocation method shown in the embodiment.

[0228] Figure 5 This application provides a schematic diagram of the structure of a robot, as shown in the embodiment of the present application. Figure 5 As shown, the robot may include an image acquisition device 100 and a processor 200.

[0229] The image acquisition device 100 is used to acquire images of the area where the robot is located;

[0230] The processor 200 is used to execute a relocation method provided in this application embodiment based on the image acquired by the image acquisition device 100.

[0231] In this specific implementation, for the robot to relocalize, the image acquisition device 100 can acquire images of the area where the robot is located, and then send the acquired images to the processor 200. Thus, the processor 200 can, based on the images acquired by the image acquisition device 100, execute a relocalization method provided in this application embodiment to determine the robot's relocalization pose relative to the prior map.

[0232] The specific implementation of the relocation method provided in the embodiments of this application has been described in detail above. The processor 200 executes each step of the relocation method provided in the embodiments of this application according to the specific implementation described above, and will not be repeated here.

[0233] Based on the same inventive concept, and corresponding to the embodiments provided in this application above... Figure 3 The present application also provides a repositioning device in addition to the repositioning method shown in the embodiment.

[0234] Figure 6 This is a schematic diagram of the structure of a repositioning device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device may include the following modules:

[0235] The initial pose determination module 610 is used to determine the initial reference pose of the target robot based on the first frame image acquired by the image acquisition device mounted on the target robot.

[0236] The reference pose determination module 620 is used to acquire multiple frames of reference images acquired by the image acquisition device, and based on the initial reference pose, determine the reference pose of the target robot when acquiring each frame of reference image, so as to obtain the reference pose corresponding to each frame of reference image.

[0237] The local map construction module 630 is used to construct a local map of the area where the target robot is currently located using the multi-frame reference images and the reference pose corresponding to each frame reference image, and to determine a target sub-map that matches the local map template in a preset prior sub-map; wherein, the prior sub-map is obtained by segmenting the prior map of the space to which the target robot belongs using a preset layout sub-map, and the layout sub-map is obtained by segmenting the spatial layout map of the space to which the target robot belongs according to the area division line;

[0238] The repositioning module 640 is used to perform similarity matching between the current image acquired by the image acquisition device and each reference image associated with the target sub-map, determine the target reference image that matches the current image, and determine the repositioning pose of the target robot relative to the prior map based on the target pose associated with the target reference image.

[0239] As can be seen from the above, the solution provided in this application can achieve relocalization of the target robot. Furthermore, template matching between the local map and the prior sub-map can improve the accuracy of regional localization, thereby reducing the possibility of relocalization failure due to regional mislocalization and increasing the success rate of relocalization. Further, similarity matching between the current image and each reference image of the target sub-map can improve the accuracy of relocalization. Moreover, since only the current image needs to be matched with each reference image of the target sub-map, without traversing all reference images associated with the complete prior map, the computational load of relocalization can be reduced, thus improving relocalization efficiency.

[0240] Optionally, in one specific implementation, the image acquisition device is a binocular camera, and the initial pose determination module 610 is specifically used for:

[0241] Based on the first frame of synchronized left and right eye images acquired by the image acquisition device on the target robot, the initial reference pose of the target robot is determined.

[0242] Optionally, in one specific implementation, the reference pose determination module 620 is specifically used for:

[0243] Multiple keyframes acquired by the image acquisition device are used as multi-frame reference images;

[0244] The multiple keyframes include: multiple frames of images acquired at preset time intervals, multiple frames of images acquired at preset movement distances of the target robot, or multiple frames of images including preset image features.

[0245] Optionally, in one specific implementation, the apparatus further includes a priori sub-map construction module, the priori sub-map construction module comprising:

[0246] The acquisition submodule is used to acquire the prior map and the layout submap;

[0247] The prior submap construction submodule is used to perform template matching on each layout submap in the prior map to obtain the prior submap matched by the layout submap.

[0248] Optionally, in one specific implementation, the prior sub-map construction sub-module is specifically used for:

[0249] In the prior map, template matching is performed on each layout sub-map to obtain the initial sub-map that matches the layout sub-map;

[0250] Obtain the connection relationship corresponding to each layout sub-map and the traversability relationship corresponding to each initial sub-map. The connection relationship corresponding to each layout sub-map is the connection relationship between the area represented by each layout sub-map and the traversable channels of the area represented by the layout sub-map. The traversability relationship corresponding to each initial sub-map is the traversability relationship between each initial sub-map and adjacent initial sub-maps, determined based on the movement trajectory of the target robot when constructing the prior map.

[0251] For each layout submap, the initial submap corresponding to the layout submap is corrected using the connection relationship corresponding to the layout submap and the traversability relationship corresponding to the initial submap corresponding to the layout submap, so as to obtain the prior submap matched by the layout submap.

[0252] Optionally, in one specific implementation, the local map construction module 630 is specifically used for:

[0253] For each prior sub-map, template matching is performed between the prior sub-map and the local area map, and template matching is also performed between the layout sub-map matched by the prior sub-map and the local area map to obtain the matching result;

[0254] Based on the matching results, the target sub-map is determined in the preset prior sub-map.

[0255] Optionally, in one specific implementation, the relocation module 640 is specifically used for:

[0256] The most recent keyframe acquired by the image acquisition device is matched with each reference image associated with the target sub-map to determine the target reference image that matches the current image.

[0257] Optionally, in one specific implementation, the pose associated with each reference image is: the pose of the image acquisition device relative to the prior map when the image acquisition device acquires the reference image;

[0258] The relocation module 640 is specifically used for:

[0259] Based on the relative pose of the image acquisition device and the target robot, pose transformation is performed on the target pose associated with the target reference image to obtain the repositioning pose of the target robot relative to the prior map.

[0260] This application also provides an electronic device, such as... Figure 7 As shown, it includes:

[0261] Memory 701 is used to store computer programs;

[0262] When the processor 702 executes the program stored in the memory 701, it implements the steps of any of the relocation methods provided in the embodiments of this application.

[0263] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, the communication interface, and the memory 701 communicating with each other via the communication bus.

[0264] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0265] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0266] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0267] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0268] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above relocation methods.

[0269] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the relocation methods described above.

[0270] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0271] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0272] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0273] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method of relocating, characterized by, The method comprises: determining an initial reference pose of the target robot based on a first frame image collected by an image collection device carried by the target robot; obtaining a plurality of reference images collected by the image collection device, and determining a reference pose of the target robot when each reference image is collected based on the initial reference pose, to obtain a reference pose corresponding to each reference image; constructing a local map of a region where the target robot is currently located by using the plurality of reference images and the reference pose corresponding to each reference image, and determining a target submap matched with the local map template in a preset prior submap; wherein the prior submap is obtained by segmenting a prior map of a space to which the target robot belongs by using a preset layout submap, and the layout submap is obtained by segmenting a space layout map of the space to which the target robot belongs according to a region segmentation line; performing similarity matching on a current image collected by the image collection device and each reference image associated with the target submap to determine a target reference image matched with the current image, and determining a repositioning pose of the target robot relative to the prior map according to a target pose associated with the target reference image; the construction method of the prior submap comprises: obtaining the prior map and the layout submap; performing template matching on each layout submap in the prior map to obtain a prior submap matched with the layout submap.

2. The method of claim 1, wherein, The image collection device is a binocular camera, and the initial reference pose of the target robot is determined based on a first frame of left-eye image and right-eye image collected by the image collection device carried by the target robot. The plurality of reference images collected by the image collection device comprises:

3. The method of claim 1, wherein, a plurality of key frames collected by the image collection device are obtained as the plurality of reference images; wherein the plurality of key frames comprise a plurality of images collected at a preset time interval, a plurality of images collected at a preset moving distance of the target robot, or a plurality of images comprising a preset image feature. The template matching on each layout submap in the prior map to obtain a prior submap matched with the layout submap comprises:

4. The method of claim 1, wherein, performing template matching on each layout submap in the prior map to obtain an initial submap matched with the layout submap; obtaining a connection relationship corresponding to each layout submap and a passable relationship corresponding to each initial submap, wherein the connection relationship corresponding to each layout submap is a connection relationship between a region represented by each layout submap and a passable channel of the region represented by the layout submap, and the passable relationship corresponding to each initial submap is a passable relationship between each initial submap and an adjacent initial submap determined based on a moving track of the target robot when the prior map is constructed; ​ For each layout submap, the initial submap corresponding to the layout submap is corrected using the connection relationship corresponding to the layout submap and the passable relationship corresponding to the initial submap corresponding to the layout submap, to obtain a priori submap matched by the layout submap.

5. The method of claim 1, wherein, The method further includes: For each priori submap, template matching is performed on the priori submap and the area local map, and template matching is performed on the layout submap matched by the priori submap and the area local map, to obtain a matching result; Based on the matching result, a target submap is determined from the preset priori submaps.

6. The method of claim 3, wherein, The method further includes: The method further includes:

7. The method according to any of claims 1 to 6, characterized in that The pose associated with each reference image is a pose of the image acquisition device relative to the priori map when the image acquisition device acquires the reference image. The method further includes: The pose associated with each reference image is a pose of the image acquisition device relative to the priori map when the image acquisition device acquires the reference image.

8. A robot, characterized in that The method further includes: The method further includes: The robot includes:

9. A repositioning device, characterized in that An image acquisition device configured to acquire images of an area in which the robot is located; A processor configured to execute the method according to any one of claims 1-7 based on the images acquired by the image acquisition device. The apparatus includes: An initial pose determination module configured to determine an initial reference pose of a target robot based on a first frame of image acquired by an image acquisition device carried by the target robot; A reference pose determination module configured to acquire a plurality of frames of reference image acquired by the image acquisition device, and determine a reference pose of the target robot when each frame of reference image is acquired based on the initial reference pose, to obtain a reference pose corresponding to each frame of reference image; A local map construction module configured to construct an area local map of a region in which the target robot is currently located using the plurality of frames of reference image and the reference pose corresponding to each frame of reference image, and determine a target submap matched with the area local map template from preset priori submaps; wherein the priori submaps are obtained by segmenting a priori map of a space to which the target robot belongs using preset layout submaps, and the layout submaps are obtained by segmenting a space layout map of the space to which the target robot belongs according to area segmentation lines. The repositioning module is configured to perform similarity matching on the current image collected by the image collection device and each reference image associated with the target submap, determine a target reference image matched with the current image, and determine a repositioning pose of the target robot relative to the prior map according to a target pose associated with the target reference image. The device further comprises a prior submap construction module, which comprises: An acquisition sub-module is configured to acquire the prior map and the layout submap. A prior submap construction sub-module is configured to perform template matching on each layout submap in the prior map to obtain a prior submap matched with the layout submap.

10. The apparatus of claim 9, wherein, The image collection device is a binocular camera, and the initial pose determination module is specifically configured to: determine an initial reference pose of the target robot based on first frames of left-eye images and right-eye images collected by the image collection device carried by the target robot; and / or, The reference pose determination module is specifically configured to: acquire a plurality of key frames collected by the image collection device as a plurality of reference images; wherein the plurality of key frames include a plurality of images collected at preset time intervals, a plurality of images collected at preset movement distances of the target robot, or a plurality of images including preset image features; and / or, The prior submap construction sub-module is specifically configured to: perform template matching on each layout submap in the prior map to obtain an initial submap matched with the layout submap; acquire a connection relationship corresponding to each layout submap and a passable relationship corresponding to each initial submap, wherein the connection relationship corresponding to each layout submap is a connection relationship between a region represented by each layout submap and a passable channel of the region represented by the layout submap, and the passable relationship corresponding to each initial submap is a passable relationship between each initial submap and an adjacent initial submap determined based on a movement trajectory of the target robot when the prior map is constructed; for each layout submap, correct the initial submap corresponding to the layout submap using the connection relationship corresponding to the layout submap and the passable relationship corresponding to the initial submap corresponding to the layout submap to obtain a prior submap matched with the layout submap; and / or, The local map construction module is specifically configured to: for each prior submap, perform template matching on the prior submap and the regional local map, and perform template matching on a layout submap matched with the prior submap and the regional local map to obtain a matching result; determine a target submap in a preset prior submap based on the matching result; and / or, The repositioning module is specifically configured to: perform similarity matching on a most recent key frame collected by the image collection device and each reference image associated with the target submap to determine a target reference image matched with the current image; and / or, the pose associated with each reference image is a pose of the image collection device relative to the prior map when the image collection device collects the reference image. The repositioning module is specifically configured to: According to the relative pose of the image acquisition device and the target robot, the target pose associated with the target reference image is pose-converted to obtain a repositioning pose of the target robot relative to the prior map.

11. An electronic device, comprising: Comprise: Memory for storing computer programs; The processor is used to execute the program stored on the memory, and the method of any one of claims 1-7 is realized.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method of any one of claims 1-7.

Images