Relocation method, electronic equipment and storage medium
By matching local panoramic images and point cloud information, the target ray is determined and mapped to a grid map, which solves the problem of low relocalization efficiency for robots in large scenes, reduces the amount of computation and storage, and improves relocalization efficiency and accuracy.
Patent Information
- Application Number
- CN202610365515.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-24
- Publication Date
- 2026-05-26
AI Technical Summary
Current robot relocalization methods require traversing the entire map in large-scale scenarios, resulting in excessive consumption of computing and storage resources and affecting relocalization efficiency.
By acquiring local panoramic images and local point clouds, and matching the local panoramic images with historical image sets, the target ray and orientation information are determined, mapped to a raster map, and relocalized using the local point cloud.
It effectively reduces computational and storage requirements, and improves relocation efficiency and accuracy.
Smart Images
Figure CN122090099A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of mobile robots, and in particular to a relocation method, electronic device, and storage medium for a mobile robot. Background Technology
[0002] Current robot intelligence technologies, especially those applied to mobile robots, often cause robots to err on their own or even stop executing any commands when positioning anomalies occur, thus requiring repositioning.
[0003] However, in current technical solutions, when the target scene is too large, it is necessary to traverse the entire map, perform a lot of calculations, and require a lot of storage resources to store the corresponding data, which consumes a lot of computing and storage resources and affects the relocation efficiency. Summary of the Invention
[0004] The technical solution to the main technical problem addressed in this application is to provide a relocation method, electronic device, and storage medium, which can effectively reduce the amount of computation and storage required during the relocation process, thereby improving relocation efficiency.
[0005] To address the aforementioned technical problems, this application provides a relocation method comprising: responding to a target object triggering relocation, acquiring a local panoramic image and a local point cloud corresponding to the target object at its current position, wherein the local panoramic image is composed of multiple local images; matching the local panoramic image with a set of historical images at the current position to obtain a matching target local image and a target historical keyframe image; acquiring the relative pose relationship between the target historical keyframe image and the target local image, and determining the target ray corresponding to the current position and the direction information and starting position of the target ray based on the relative pose relationship, wherein the target ray is a ray whose starting position is the global positioning information corresponding to the target historical keyframe image; mapping the target ray to a preset grid map using the direction information and starting position of the target ray, acquiring the target ray range from the grid map, wherein the grid map is determined by the scene where the current position is located, the target ray is not obstructed within the target ray range, and determining the relocation information of the target object using the target ray range and the local point cloud.
[0006] In some embodiments, the step of obtaining a local panoramic image and a local point cloud corresponding to the target object at the current position in response to the target object triggering relocation includes: rotating around the current position as the center, obtaining a first image set and a first point cloud information during the rotation process, wherein the first image set includes a local image and a point cloud at each angle during the rotation process, and the first image set is located in the relocation local coordinate system; stitching together all the local images in the first image set to obtain the local panoramic image corresponding to the current position, and stitching together all the point clouds in the first point cloud information to obtain the local point cloud corresponding to the current position.
[0007] In some embodiments, the step of matching the local panoramic image and the historical image set at the current location to obtain a matching target local image and a target historical keyframe image includes: acquiring multiple keyframe positions in the historical image set; in response to the target object reaching any keyframe position, acquiring multi-view image information of the target object at the current location, combining the image information with the current location to determine the historical image set, wherein the historical image set is located in a historical global coordinate system; matching the local panoramic image and the historical image set at the current location to obtain a matching score; acquiring the target historical keyframe image corresponding to the matching score that is greater than a preset minimum matching score threshold; acquiring the corresponding target local image using the best matching region of the target historical keyframe image in the local panoramic image, and constructing a matching association.
[0008] In some embodiments, obtaining multiple keyframe positions in the historical image set includes: during the normal operation of the target object, obtaining the current position information of the target object and the grid map it occupies; dividing the grid map into equally spaced blocks to obtain multiple keyframe positions in the grid map.
[0009] In some embodiments, the method further includes: determining occlusion of the image information of the target object acquired at the current location to obtain an occlusion score; in response to the occlusion score being greater than or equal to a score threshold, retaining the corresponding image information in the historical image set and marking the current location as acquired; or in response to the occlusion score being less than the score threshold, acquiring images again from the surrounding area of the current location.
[0010] In some embodiments, obtaining the relative pose relationship between the target historical keyframe image and the target local image, and determining the target ray corresponding to the current position and the direction information and starting position of the target ray based on the relative pose relationship, includes: obtaining relative rotation data and translation amount between the target historical keyframe image and the target local image to determine the relative pose relationship; using the translation amount in the relative pose relationship, combined with the global positioning information corresponding to the target historical keyframe image, determining the target ray with the global positioning information as the starting position in the historical global coordinate system, wherein the direction information is determined by the direction corresponding to the relative rotation data.
[0011] In some embodiments, the step of mapping the target ray to a preset grid map using the direction information and starting position of the target ray, and obtaining the range of the target ray from the grid map, includes: obtaining a blocking area in the grid map; mapping the target ray to the preset grid map, using the blocking area in the grid map to block the target ray, and combining the direction information and starting position of the target ray to obtain the range of the target ray from the grid map.
[0012] In some embodiments, determining the relocation information of the target object using the target ray range and the local point cloud includes: using the target ray range as an initial pose; matching the initial pose and the local point cloud with the grid map to obtain a precise pose; and using the precise pose to determine the relocation information of the target object.
[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, the electronic device including a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the method as described above.
[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium having at least one program, which, when loaded and executed by a processor, is used to implement the method described above.
[0015] Unlike current technologies, the relocation method provided in this application includes: in response to a target object triggering relocation, acquiring a local panoramic image and a local point cloud corresponding to the target object at its current position, wherein the local panoramic image is composed of multiple local images; matching the local panoramic image with a set of historical images at the current position to obtain a matching target local image and a target historical keyframe image; acquiring the relative pose relationship between the target historical keyframe image and the target local image, and determining the target ray and its direction information and starting position corresponding to the current position based on the relative pose relationship, wherein the target ray is a ray whose starting position is the global positioning information corresponding to the target historical keyframe image; mapping the target ray to a preset grid map using the direction information and starting position of the target ray, obtaining the target ray range from the grid map, wherein the grid map is determined by the scene where the current position is located, the target ray is not obstructed within the target ray range, and determining the relocation information of the target object using the target ray range and the local point cloud. In this application, the most relevant local target image and historical image set are obtained by matching the current local panoramic image and the target historical keyframe image. Then, the target ray with the current position as the starting position is determined based on the relative position relationship between the two. The target ray is narrowed down by using a grid map. Then, the relocation information of the target object is determined by using the narrowed target ray range and the local point cloud of the current position. In other words, this application can effectively narrow down the local range of the target object, reduce the amount of computation and storage, and combine image information and point cloud information to calculate, which can effectively improve the relocation efficiency and relocation accuracy. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the relocation method in this application; Figure 2 It is the grid in this application Figure 1 A schematic diagram of the embodiment; Figure 3 This is a schematic diagram of an embodiment of the mapping relationship between local panoramic images and historical keyframe images in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device in this application; Figure 5 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application. Detailed Implementation
[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0019] When a mobile robot experiences a positioning anomaly, it often leads to incorrect execution of commands or even complete cessation of command execution. Self-cleaning devices typically operate in complex environments and are frequently subjected to positioning anomalies caused by external forces (such as being picked up or having sensors obstructed), or encountering degraded environments that cause positioning failures. In such cases, repositioning technology is necessary to enable the robot to accurately obtain its current position information and continue executing commands.
[0020] Current relocalization methods suffer from several drawbacks. First, when the target scene is large, they require traversing the entire map, consuming significant computational resources and resulting in low efficiency. Second, semantic information extraction is also computationally expensive, and the results are easily affected by dynamic obstacles. Furthermore, indoor environments often contain two or more scenes with identical semantics, leading to mismatches during semantic matching and failing to provide effective prior information, further reducing relocalization efficiency. Third, due to the limited field of view of image sensors, keyframe selection is highly sensitive to angle information. Therefore, the global image map needs to collect keyframe information from multiple angles at the same location, requiring substantial storage resources and computational power for feature matching. Obtaining the global relative pose also necessitates extensive computation for scale recovery. These high storage and computational costs cannot guarantee efficient relocalization.
[0021] Therefore, a relocation method, electronic device, and storage medium are provided that can effectively narrow down the local area of the target object, reduce the amount of computation and storage, and combine image information and point cloud information for calculation, which can effectively improve the relocation efficiency and relocation accuracy.
[0022] Please see Figure 1 , Figure 1This is a flowchart illustrating an embodiment of the relocation method in this application; it should be noted that, if there are substantial results, the method of this application does not rely on... Figure 1 The sequence of processes shown is limited.
[0023] like Figure 1 As shown, the relocation method of this application may include the following operations.
[0024] S10. In response to the target object triggering relocation, obtain the local panoramic image and local point cloud corresponding to the target object at the current position, wherein the local panoramic image is composed of multiple local images.
[0025] The target object refers to the object that needs to be relocalized, such as a moving device or person; the local panoramic image refers to all local images from a 360° perspective at the current position; and the local point cloud refers to the set of 3D feature points in the spatial region where the current position is located.
[0026] Specifically, if the target object experiences a positioning anomaly due to some external understanding or fails to locate due to a degraded scene, relocation is triggered. When the target object triggers relocation, the image acquisition device is used to acquire all local images of the target object from a 360° perspective at the current position to determine the local panoramic image; and the set of 3D feature points in the spatial area where the current position is located is acquired as the local point cloud of the current position.
[0027] S20. Match the local panoramic image with the historical image set of the current location to obtain the matching target local image and target historical keyframe image respectively.
[0028] Among them, the historical image set refers to the set of images acquired when the current position is normally located; the target historical keyframe image refers to the historical keyframe image most relevant to the current position, and the historical keyframe image refers to the image acquired at the keyframe position. The scene corresponding to the current position includes multiple keyframe positions; the target local image refers to the local image in the local panoramic image that is most relevant to the target historical keyframe image.
[0029] Specifically, the local panoramic image acquired at the current location is matched with a set of normal historical images acquired at the current location in the past; then the two images with the highest matching degree are obtained, where the matching image from the historical image set is the target historical keyframe image, and the matching image from the target local image set is the target local image.
[0030] S30. Obtain the relative pose relationship between the target historical keyframe image and the target local image, and determine the target ray corresponding to the current position and the direction information and starting position of the target ray based on the relative pose relationship. The target ray is a ray whose starting position is the global positioning information corresponding to the target historical keyframe image.
[0031] The relative pose relationship refers to the relative rotation and translation transformations between the target's historical keyframe images and the target's local images, including relative rotation data and translation amounts, characterizing the spatial motion relationship between the two images. A target ray is a ray that starts at the global positioning information corresponding to the target's historical keyframe image and uses the direction information in the relative pose relationship as its direction. Global positioning information refers to the position information corresponding to each historical keyframe image; each historical keyframe image has a corresponding position information, and each historical keyframe image could potentially be a target historical keyframe image.
[0032] Specifically, the coordinate information of the target's historical keyframe image and the target's local image are obtained. The relative pose relationship between the two images is determined using their coordinate information; for example, the corresponding coordinate information is obtained in the same global coordinate system. The direction information of the target ray is determined using the relative rotation data and translation in the relative pose relationship, and the starting position of the target ray is taken from the coordinate information corresponding to the historical keyframe image. It can be understood that the coordinate information is also the position information.
[0033] S40. Using the direction information and starting position of the target ray, map the target ray onto a preset grid map, and obtain the range of the target ray from the grid map. The grid map is determined by the scene where the current position is located, and the target ray is not blocked within the range of the target ray.
[0034] Here, the grid map refers to the grid cells of the current location's environment; the target ray range refers to the remaining range of the target ray after it has been blocked by the grid area in the grid map.
[0035] Specifically, the grid map corresponding to the scene where the current location is located is obtained. After determining the direction information and starting position of the target ray, the target ray is mapped onto the preset grid map. Since there are blocking areas in the grid map, the range of the target ray will be reduced under the blocking area, thus obtaining the range of the target ray that is not blocked.
[0036] S50. Using the target ray range and local point cloud, determine the relocation information of the target object.
[0037] Among them, relocation information refers to the accurate location information determined after the target object is relocated.
[0038] Specifically, by utilizing the acquired target ray range and local point cloud, fine matching can be performed on the grid map to obtain the precise pose of the current target object, and then the precise relocation information of the target object can be determined based on the precise pose.
[0039] In this embodiment, the most relevant target local image and target historical keyframe image are obtained by matching the current local panoramic image and the historical image set. Then, based on the relative positional relationship between the two, the target ray is determined with the global positioning information corresponding to the target historical keyframe image as the starting position. The target ray is narrowed down using a grid map. Then, the relocation information of the target object is determined by using the narrowed target ray range and the local point cloud of the current position. That is, in this application, the local range of the target object can be effectively narrowed down, reducing the amount of computation and storage. By combining image information and point cloud information for calculation, the relocation efficiency and relocation accuracy can be effectively improved.
[0040] In some embodiments, in response to a target object triggering relocation, acquiring a local panoramic image and a local point cloud corresponding to the target object at its current location may include the following:
[0041] Rotate around the current position to obtain the first image set and the first point cloud information during the rotation process. The first image set includes local images and point clouds at each angle during the rotation process, and the first image set is located in the repositioning local coordinate system.
[0042] All local images in the first image set are stitched together to obtain a local panoramic image corresponding to the current position, and all point clouds in the first point cloud information are stitched together to obtain a local point cloud corresponding to the current position.
[0043] In order to obtain all local images of the current position, it is necessary to rotate around the current position and obtain local images from each viewpoint in turn during the rotation process, thereby forming the first image set; and obtain the corresponding lidar point cloud information, thereby forming the first point cloud information.
[0044] Specifically, the current position of the target object is obtained, and then the target object is rotated with the current position as the center. During the rotation, local images from each viewpoint are acquired sequentially. The acquired local images are then combined into a first image set, and all local images in the first image set are stitched together in the acquisition order to obtain a local panoramic image corresponding to the current position. In addition, the lidar point cloud information during the rotation is acquired, and the point cloud information from each viewpoint is stitched together to obtain a local point cloud corresponding to the current position.
[0045] It is understandable that repositioning the local coordinate system refers to a coordinate system constructed with the current position as the center. The direction corresponding to the initial pose of the current position is the X-axis, the vertical direction is the Z-axis, and the Y-axis is determined in the rotation angle.
[0046] In this embodiment, by stitching together local images from each viewpoint, the size of the subsequent historical image set can be effectively reduced, thereby reducing storage and subsequent computational load and effectively improving relocation efficiency.
[0047] In some embodiments, matching is performed using a local panoramic image and a set of historical images of the current location to obtain a matching target local image and a target historical keyframe image, which may include the following:
[0048] Obtain the positions of multiple keyframes from a collection of historical images.
[0049] In response to the target object reaching any keyframe position, multi-view image information of the target object at the current position is obtained, and the image information is combined with the current position to determine the historical image set, wherein the historical image set is located in the historical global coordinate system.
[0050] A matching score is obtained by matching the local panoramic image with a set of historical images of the current location.
[0051] Obtain the target historical keyframe image corresponding to the matching score that is greater than the preset minimum matching score threshold.
[0052] The corresponding local image of the target is obtained by using the best matching region in the local panoramic image of the target's historical keyframe image, and a matching association is constructed.
[0053] Here, the historical image set refers to the set of standard images that can be correctly located at the current position during the normal operation of the target object. The keyframe position refers to the keyframe position of the raster map determined by the historical image set. The minimum matching score threshold is used to filter suitable matching scores and can be set according to actual conditions to ensure a suitable number of matching scores that meet the criteria.
[0054] The target historical keyframe image refers to the historical keyframe image with the highest matching score. Historical keyframe images are the images acquired at each keyframe position, and each keyframe position can acquire multiple keyframe images from multiple viewpoints. The best matching region refers to the region with the highest matching score in the local panoramic image; the target local image refers to the local image that best matches the target historical keyframe image. Matching association refers to associating the target local image with the target historical keyframe image.
[0055] Specifically, during the normal operation of the target object, a historical image set is acquired, and multiple keyframe positions are determined based on the raster images corresponding to the historical image set. Then, during the current relocalization process, when the target object reaches any keyframe position, the reached keyframe position is taken as the current position, and multi-view image information of the target object at the current position is acquired; then, all image information and the corresponding current position are combined to obtain the historical image set for each current position; it can be understood that each position reached by the target object can be used as the current position.
[0056] Then, the local panoramic image and the historical image set corresponding to the current position are matched to obtain the matching score between the local panoramic image and the historical image set for each viewpoint; then, the candidate historical image keyframes are obtained again based on the minimum matching score threshold, which are used as the target historical image keyframes.
[0057] Furthermore, a maximum number of matching keys can be set, and the matching scores of the maximum number of matching associations can be filtered out. Then, the matching scores can be filtered again based on the minimum matching score threshold to obtain the target historical image keyframes.
[0058] After acquiring keyframes of historical images, the location is determined based on the historical keyframe images. Then, the region corresponding to the location is used as the best matching region in the local panoramic image. Then, the local image corresponding to the target historical keyframe image is found from the best matching region in the local panoramic image and used as the target local image. Finally, a matching association is constructed between the target local image and the target historical keyframe image.
[0059] In this embodiment, by matching images at key frame locations, matching local images and historical key frame images are determined, and the most matching target local image and target historical key frame image are determined by the minimum matching score threshold, which effectively reduces the number of images and thus improves relocation efficiency.
[0060] In some embodiments, obtaining multiple keyframe locations from a collection of historical images may include the following.
[0061] During the normal operation of the target object, obtain the target object's current location information and the grid map it occupies.
[0062] Divide the raster map into equally spaced blocks to obtain the locations of multiple keyframes in the raster map.
[0063] "Normal operation" refers to the ability of the target object to correctly acquire location information during normal movement. A grid map refers to a gridded map of the space in which the current location is situated.
[0064] Specifically, the current position information of the target object during normal operation is obtained, along with a raster map of the space occupied by that current position. The raster map is then divided into equally spaced blocks, resulting in multiple blocks, with the block boundaries used as keyframe locations.
[0065] Understandably, this data can be continuously generated as the target object works normally, and the latest current location information, raster map, and keyframe positions should be retained.
[0066] In some embodiments, the following may also be included.
[0067] Occlusion is assessed based on the image information of the target object at its current location, resulting in an occlusion score.
[0068] If the occlusion score is greater than or equal to the score threshold, the corresponding image information is retained in the historical image set, and the current location is marked as captured.
[0069] Alternatively, if the occlusion score is less than the score threshold, then the image around the current location will be captured again.
[0070] Among them, occlusion judgment refers to whether the acquired image is occluded by obstacles in the raster map.
[0071] Specifically, after image information is acquired at the current location of the target object, occlusion and dynamic obstacle detection are performed using obstacles in the raster map to score the acquired image information. If the occlusion score is greater than or equal to a set scoring threshold, the acquired image information is considered valid and retained in the historical image set. Simultaneously, the keyframe location is marked as an image acquired to avoid duplicate image acquisition and reduce the number of images. If the occlusion score is less than the scoring threshold, the acquired image information is considered invalid and needs to be re-acquired. Re-acquisition can be performed at another location within a preset range of the keyframe location.
[0072] It is understandable that the current position can be a keyframe position or a position within a preset range of keyframe positions.
[0073] In this embodiment, by acquiring images during normal operation, the grid map and keyframe positions under normal conditions can be determined. By using a marking method, the acquisition of images can be reduced, effectively reducing the amount of computation and thus improving the relocation efficiency.
[0074] To better illustrate the raster map, the following explanation is provided in conjunction with the attached diagram.
[0075] Please see Figure 2 and Figure 3 , Figure 2 It is the grid in this application Figure 1 A schematic diagram of the embodiment; Figure 3 This is a schematic diagram of an embodiment of the mapping relationship between local panoramic images and historical keyframe images in this application.
[0076] For example, when the robot is working normally, based on the localization and grid map information output by the current laser SLAM, visual keyframe positions are selected at equal intervals on the grid map. The keyframe positions are as follows: Figure 2 The dots shown. When the robot reaches a keyframe position, it records visual image information from any viewpoint at that current position. Combined with current location information Composing a historical visual collection ,in For image information, The global localization information for the corresponding image is provided by laser SLAM. This represents the historical global coordinate system. Specifically, the distance between keyframes is adaptively adjusted based on the current environment size to ensure the size of the historical vision set remains within an appropriate range. When the robot reaches the vicinity of a keyframe position, image acquisition can begin. The acquired images are scored using occlusion and dynamic obstacle detection. If the score exceeds a set threshold, the acquired image is considered valid, saved to the historical vision set, and the keyframe position is marked as acquired. Otherwise, image acquisition continues in the vicinity of that keyframe position.
[0077] Then, when the robot triggers relocalization, it rotates around its current position at a certain speed for one revolution, and collects a set of images during the rotation process. and lidar point cloud information, This represents the repositioning of the local coordinate system. The image and point cloud information acquired during the rotation process are stitched together to generate a local panoramic image and local point cloud of the current location. Please refer to [link / reference]. Figure 3 .
[0078] Next, keyframe images from the historical image set are matched one by one with the local panoramic images. Based on the matching score, the target historical keyframe image that best matches the current location is obtained. And the target local image corresponding to the best matching region on the local panoramic image. Construct matching associations for image keyframes that meet the scoring requirements. Specifically, set a minimum matching score threshold. and the maximum number of matching associations Before selection The match with the highest score that is greater than the minimum match score threshold. The matching results are used as candidate historical keyframe images and as target historical keyframe images; and the best matching region is combined with the corresponding target local image in the local panoramic image set to construct a matching association. Please refer to [link to relevant documentation]. Figure 3 .
[0079] In some embodiments, obtaining the relative pose relationship between the target historical keyframe image and the target local image, and determining the target ray corresponding to the current position and the direction information and starting position of the target ray based on the relative pose relationship, may include the following:
[0080] Acquire the relative rotation data and translation amount between the target's historical keyframe images and the target's local images to determine the relative pose relationship.
[0081] By utilizing the translation amount in the relative pose relationship and combining it with the global positioning information corresponding to the target's historical keyframe images, the target ray with the global positioning information as the starting point is determined in the historical global coordinate system. The direction information is determined by the direction corresponding to the relative rotation data.
[0082] The relative rotation data and translation amounts can be obtained in a coordinate system. Therefore, it is necessary to transform the target historical keyframe image and the target local image to the same coordinate system. That is, to reposition the target local image in the local coordinate system and the target historical keyframe image in the historical global coordinate system to the same coordinate system, thereby determining the relative rotation data and translation amounts between the target historical keyframe image and the target local image, which in turn determines the relative pose relationship. The target ray representation is matched and associated with the target historical keyframe image to determine the possible location region of the target object and its orientation information from the image viewpoint.
[0083] Specifically, the target historical keyframe image and the target local image are transformed to the same coordinate system. Then, based on the positional information of the target historical keyframe image and the target local image, the relative rotation data and translation between them can be determined. Based on the relative rotation data and translation, the relative pose relationship between the target historical keyframe image and the target local image can be determined. Next, the direction information is determined using the relative rotation data and translation in the relative pose relationship, and the target ray is determined using the global positioning information corresponding to the target historical keyframe image as the starting position.
[0084] Understandably, converting the target's historical keyframe images and local images to the same coordinate system can be achieved using existing technologies.
[0085] In some embodiments, relative rotation data in the relative pose relationship can also be used. The system determines the global orientation information of the current position from the image viewpoint, and then determines the target ray based on the global orientation information and translation. The global orientation information includes the direction information of the target ray.
[0086] For example, the relative poses of two images in a matching association are calculated sequentially based on their matching scores. Specifically, the matching between two images has already been completed using feature points. The relative pose relationship between the two images can be obtained by combining epipolar geometric constraints with the eigenvalue matrix or homography matrix. , The data represents relative rotation, with t representing the translation. Since constraints are constructed solely using 2D pixels, the calculated translation t in the relative pose lacks scale information. This is combined with global localization information from historical keyframe images. Solve the current position in the image Global orientation information at any given moment It is understandable that global positioning information refers to the location information corresponding to historical keyframe images; based on scale-free translation... Obtain a globally positioned information in the global coordinate system. The target ray originating from, such as Figure 2 The dashed line indicates the region where the robot might currently be located, based on the matching associations created for candidate historical image keyframes of the current position, and within the image... Global position information of the robot from the perspective .
[0087] In this embodiment, by obtaining the relative pose relationship between two images, the target ray starting from the current position is determined, thereby narrowing down the range of the target object, further reducing the amount of data, and effectively improving computational efficiency.
[0088] In some embodiments, the target ray is mapped onto a preset grid map using the direction information and starting position of the target ray to obtain the range of the target ray, which may include the following:
[0089] Get the block area in the grid map.
[0090] The target ray is mapped onto a preset grid map. The grid area in the grid map is used to block the target ray. The range of the target ray is obtained by combining the direction information and starting position of the target ray.
[0091] In this context, the blocking area refers to any obstruction in the raster map, such as walls, seats, or moving objects, that may affect the movement of the target object. The target ray range refers to the remaining ray range after the target object has passed through the blocking area.
[0092] Specifically, the grid map clearly has blocked areas and non-blocked areas. Therefore, when mapping the target ray onto the grid map, the extension direction corresponding to the direction information of the target ray can be interrupted according to the blocked areas in the grid map, while retaining the target ray from the starting position to the blocked area, thereby determining the reduced range of the target ray.
[0093] In this embodiment, the target ray is narrowed down by using a grid map, which can further reduce the ray range, thereby reducing the corresponding computational load and further improving the relocation efficiency.
[0094] In some embodiments, the relocation information of the target object is determined using the target ray range and local point cloud, which may include the following.
[0095] The target ray range is used as the initial pose.
[0096] The accurate pose is obtained by matching the initial pose and local point cloud with the grid map.
[0097] Use precise pose to determine the relocation information of the target object.
[0098] The pose includes position information and orientation information. Therefore, after obtaining the accurate pose, the accurate relocation information of the target object can be obtained.
[0099] Specifically, after obtaining the reduced target ray range, the target ray range is used as the initial pose of the target object. Then, the initial pose and the previously obtained local point cloud are used to perform fine matching with the grid map to determine the precise pose of the target object. Finally, the position information in the precise pose is used to determine the relocation information of the target object.
[0100] For example: Based on the starting position and direction information of the target ray, the target ray is mapped onto a two-dimensional grid map. Since the occupied points in the two-dimensional grid map often represent obstacles, combined with the principles of image imaging, the occupied points in the two-dimensional grid map that may be walls or large pieces of furniture are used to further narrow down the ray range, providing a relatively accurate location area, thus obtaining the range of the target ray; for example: Figure 2 As shown by the solid line segment. The area represented by the solid line segment and... Using the initial position as the orientation, a fine match is performed between the local point cloud and the grid map to obtain the current global precise pose of the robot, thereby determining the current robot's relocalization information.
[0101] In this embodiment, the most relevant target local image and target historical keyframe image are obtained by matching the current local panoramic image and the historical image set. Then, the target ray with the current position as the starting position is determined based on the relative position relationship between the two. The target ray is narrowed down by using a grid map. Then, the relocation information of the target object is determined by using the narrowed target ray range and the local point cloud of the current position. That is, in this application, the local range of the target object can be effectively narrowed down, reducing the amount of computation and storage. By combining image information and point cloud information for calculation, the relocation efficiency and relocation accuracy can be effectively improved.
[0102] This application also provides an electronic device.
[0103] See Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the electronic device in this application. The electronic device can perform the steps in the above method.
[0104] The electronic device 200 includes a memory 220, a processor 210 coupled to the memory, and at least one computer program stored in the memory 220 and executable on the processor 210. When the processor 210 loads and executes the at least one computer program, it implements the steps of the relocation method described above. For related details, please refer to the detailed description in the above method; further elaboration will not be repeated here.
[0105] This application also includes a computer-readable storage medium.
[0106] Please see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application.
[0107] The computer-readable storage medium 300 stores at least one program 310, which, when loaded and executed by a processor, is used to implement the steps of the relocation method described above. For related details, please refer to the detailed description in the above method; it will not be repeated here.
[0108] The above scheme matches the current local panoramic image with the historical image set to obtain the most relevant target local image and target historical keyframe image. Then, based on the relative positional relationship between the two, it determines the target ray starting from the current position. The target ray is then narrowed down using a grid map. Finally, using the narrowed target ray range and the local point cloud of the current position, the relocation information of the target object is determined. That is, in this application, the local range of the target object can be effectively narrowed down, reducing the amount of computation and storage. By combining image information and point cloud information for calculation, the relocation efficiency and relocation accuracy can be effectively improved.
[0109] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0111] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0113] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A relocation method, characterized in that, include: In response to a target object triggering relocation, a local panoramic image and a local point cloud corresponding to the target object at its current position are obtained, wherein the local panoramic image is composed of multiple local images; The local panoramic image and the set of historical images at the current location are matched to obtain the matching target local image and the target historical keyframe image, respectively. The relative pose relationship between the target historical keyframe image and the target local image is obtained, and the target ray corresponding to the current position and the direction information and starting position of the target ray are determined based on the relative pose relationship. The target ray is a ray whose starting position is the global positioning information corresponding to the target historical keyframe image. Using the direction information and starting position of the target ray, the target ray is mapped onto a preset grid map to obtain the range of the target ray. The grid map is determined by the scene where the current position is located, and the target ray is not blocked within the range of the target ray. The relocation information of the target object is determined using the target ray range and the local point cloud.
2. The method according to claim 1, characterized in that, The step of responding to a target object triggering relocalization and acquiring a local panoramic image and local point cloud corresponding to the target object at its current position includes: Rotate around the current position to obtain a first image set and a first point cloud information during the rotation process. The first image set includes a local image and a point cloud at each angle during the rotation process, and the first image set is located in the repositioning local coordinate system. All local images in the first image set are stitched together to obtain the local panoramic image corresponding to the current position, and all point clouds in the first point cloud information are stitched together to obtain the local point cloud corresponding to the current position.
3. The method according to claim 1, characterized in that, The step of matching the local panoramic image with the historical image set at the current location to obtain the matched target local image and target historical keyframe image includes: Obtain the positions of multiple keyframes in the historical image set; In response to the target object reaching any keyframe position, multi-view image information of the target object at the current position is obtained, and the historical image set is determined, wherein the historical image set is located in the historical global coordinate system; A matching score is obtained by matching the local panoramic image with the set of historical images at the current location; Obtain the target historical keyframe image corresponding to the matching score that is greater than the preset minimum matching score threshold; The corresponding local image of the target is obtained by using the best matching region in the local panoramic image of the target historical keyframe image, and a matching association is constructed.
4. The method according to claim 3, characterized in that, Obtaining the locations of multiple keyframes in the historical image set includes: During the normal operation of the target object, the current location information of the target object and the grid map it occupies are obtained; The raster map is divided into equally spaced blocks to obtain multiple keyframe locations in the raster map.
5. The method according to claim 3, characterized in that, Also includes: Occlusion assessment is performed on the image information of the target object acquired at the current location to obtain an occlusion score; If the occlusion score is greater than or equal to the score threshold, the corresponding image information is retained in the historical image set, and the current location is marked as having been captured. Alternatively, in response to the occlusion score being less than the score threshold, the surrounding area of the current location is re-captured.
6. The method according to claim 1, characterized in that, The step of acquiring the relative pose relationship between the target historical keyframe image and the target local image, and determining the target ray corresponding to the current position and the direction information and starting position of the target ray based on the relative pose relationship, includes: The relative rotation data and translation amount between the target historical keyframe image and the target local image are obtained to determine the relative pose relationship; Using the translation amount in the relative pose relationship and the global positioning information corresponding to the target historical keyframe image, the target ray with the global positioning information as the starting position is determined in the historical global coordinate system, wherein the direction information is determined by the direction corresponding to the relative rotation data.
7. The method according to claim 1, characterized in that, The step of mapping the target ray onto a preset grid map using the direction information and starting position of the target ray to obtain the range of the target ray includes: Obtain the grid area in the grid map; The target ray is mapped onto a preset grid map, and the target ray is interrupted by the grid area in the grid map. The range of the target ray is obtained by combining the direction information and starting position of the target ray.
8. The method according to claim 7, characterized in that, include: The step of determining the relocation information of the target object using the target ray range and the local point cloud includes: The target ray range is used as the initial pose; The precise pose is obtained by matching the initial pose and the local point cloud with the grid map. The precise pose is used to determine the relocation information of the target object.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the method as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium has at least one program that, when loaded and executed by a processor, is used to implement the method as described in any one of claims 1-8.