Home service robot repositioning method, home service robot, and system

Abstract

This application relates to a relocalization method for a home service robot, a home service robot, and a system. The method includes: during the relocalization process, acquiring current point cloud data collected by the robot's LiDAR from the currently located indoor area, converting the current point cloud data into a point cloud image, and acquiring camera images collected by a camera installed in the indoor area; performing edge recognition on objects with straight edges in the camera images to obtain at least two first straight edge edges, and determining at least two second straight edge edges contained in the point cloud image; matching the at least two first straight edge edges and the at least two second straight edge edges to determine the robot's orientation in a global map; determining the robot's position in the global map; and based on the robot's orientation and position in the global map, determining the robot's pose information in the global map. This method can improve relocalization efficiency.

Description

Technical Field

[0001] This application relates to the fields of Internet and computer technology, and in particular to a method for repositioning a home service robot, a home service robot, and a system. Background Technology

[0002] Before performing a task, mobile robots, such as home service robots, scan their environment using LiDAR to build an environmental map. They then navigate based on this map. During navigation, if the robot loses power, experiences environmental changes, or is moved, it often loses its location information. Therefore, it needs to re-determine its position on the environmental map, a process known as relocalization.

[0003] In traditional techniques, during relocalization, mobile robots use their built-in LiDAR to scan the surrounding environment. By comparing environmental features with those in an environmental map, they determine their own position. The robot then updates the map based on the new location information and continues to complete the remaining tasks.

[0004] However, the method of relocalization by comparing the surrounding environmental features with those in the environmental map has the problem of slow relocalization speed, which affects the relocalization efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a relocation method for home service robots, a home service robot, and a home service robot system that can improve relocation efficiency in response to the above-mentioned technical problems.

[0006] On one hand, this application provides a relocalization method for a home service robot, comprising: during the relocalization process, acquiring current point cloud data collected by the robot's lidar from the current indoor area, and converting the current point cloud data into a point cloud image; and acquiring camera images collected by a camera installed in the indoor area; performing edge recognition on objects with straight edges in the camera images to obtain at least two first straight edges, and determining at least two second straight edges contained in the point cloud image; matching the at least two first straight edges and the at least two second straight edges to determine the robot's orientation in a global map; determining the robot's position in the global map; and determining the robot's pose information in the global map based on the robot's orientation and position in the global map.

[0007] On the other hand, this application also provides a home service robot, including: an image acquisition module, used to acquire current point cloud data collected by the robot's lidar from the current indoor area during the relocation process, and convert the current point cloud data into a point cloud image, and acquire camera images collected by a camera installed in the indoor area; an edge determination module, used to perform edge recognition on objects with straight edges in the camera image to obtain at least two first straight edges, and determine at least two second straight edges contained in the point cloud image; an orientation determination module, used to match the at least two first straight edges and the at least two second straight edges to determine the robot's orientation in a global map; a position determination module, used to determine the robot's position in the global map; and a pose determination module, used to determine the robot's pose information in the global map based on the robot's orientation and position in the global map.

[0008] On the other hand, this application also provides a home service robot system, which includes a home service robot.

[0009] The aforementioned home service robot relocalization method, home service robot, and home service robot system, during the relocalization process, acquire current point cloud data collected by the robot's LiDAR from the current indoor area, convert the current point cloud data into a point cloud image, and acquire camera images collected by cameras installed in the indoor area. Edge recognition is performed on objects with straight edges in the camera images to obtain at least two first straight edges, and at least two second straight edges are determined within the point cloud image. The at least two first straight edges and at least two second straight edges are matched to determine the robot's orientation in the global map, determine the robot's position in the global map, and, based on the robot's orientation and position in the global map, determine the robot's pose information in the global map. Since the camera images captured by the camera can reflect the robot's position from a global perspective, the robot's orientation in the global map can be quickly determined by combining the first straight line edge in the camera image and the second straight line edge in the point cloud image, thus improving the efficiency of orientation determination. Furthermore, since the robot's orientation in the global map has been determined before determining its position, it avoids performing position matching calculations on all possible orientations, thereby improving the efficiency of position confirmation and thus improving the efficiency of determining pose information, which in turn improves relocalization efficiency. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a diagram illustrating the application environment of the home service robot relocation method in some embodiments;

[0012] Figure 2 This is a flowchart illustrating the relocation method for a home service robot in some embodiments;

[0013] Figure 3 This is a schematic diagram of a home environment in some embodiments;

[0014] Figure 4 This is a schematic diagram of camera image segmentation results in some embodiments;

[0015] Figure 5 This is a flowchart illustrating the relocation method for a home service robot in some other embodiments;

[0016] Figure 6 Here are some structural block diagrams of the home service robot in some embodiments;

[0017] Figure 7 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] The home service robot relocation method provided in this application embodiment can be applied to, for example... Figure 1The illustration depicts a home indoor environment. This environment includes a home service robot 102 and an external camera 104. The external camera 104 refers to a camera located outside the home service robot 102, as opposed to a camera built into the home service robot 102. The camera can be a fixedly installed camera within the home indoor environment. The home service robot 102 possesses sensing, movement, and interaction capabilities and can interact with its environment in real time. It can capture information about its surrounding environment through sensory organs such as cameras, LiDAR, or tactile LiDAR installed on the home service robot 102. The home service robot 102 can be, but is not limited to, a cleaning robot, a companion robot, or a humanoid robot. The cleaning robot can be, but is not limited to, a sweeping robot, a mopping robot, and a combined sweeping and mopping robot. The home service robot 102 and the external camera 104 can communicate wirelessly. When the home service robot 102 needs repositioning, it can acquire images captured by the external camera 104 via wireless communication and use these images to assist in repositioning itself.

[0020] In some embodiments, such as Figure 2 As shown, a relocation method for a home service robot is provided, executed by the home service robot (hereinafter referred to as the robot), including the following steps:

[0021] Step 202: During the relocation process, acquire the current point cloud data collected by the LiDAR in the robot from the current indoor area, convert the current point cloud data into a point cloud image, and acquire the camera image collected by the camera installed in the indoor area.

[0022] Optionally, at the same time, the robot can acquire current point cloud data collected by the LiDAR in the robot from its current location in the indoor area, and acquire camera images collected by the cameras installed in the indoor area.

[0023] Among them, such as Figure 3 As shown, rooms A and B are illustrated. The robot is currently in room A. Circles represent the robot, and ellipses within these circles represent the robot's lidar. The laser beams emitted by the lidar are represented by dashed lines surrounding it. Semicircle 302 represents a camera (for illustration only), 304 represents the door to room A, and 306 represents the door to room B. Solid lines represent wall structures. The current point cloud data and camera images can be captured in room A.

[0024] When the lidar is a single-line lidar, the point cloud image can be a grayscale image; when the lidar is a 3D (three-dimensional) lidar or a multi-dimensional lidar, the point cloud image can be a depth image.

[0025] In some embodiments, the current point cloud data is a collection of multiple data points, each of which records its coordinates in the LiDAR coordinate system. To convert the current point cloud data into a point cloud image, the robot can transform the coordinates of the data points in the LiDAR coordinate system to coordinates in the same Cartesian coordinate system, divide the plane containing the data points into grids, with each grid corresponding to a pixel in the image, thereby determining the grid corresponding to each data point. Based on the data points and the correspondence between the data points and the grids, a point cloud image is generated.

[0026] In some embodiments, the robot and the camera can communicate via a wireless network or short-range communication technology, which may include, but is not limited to, Bluetooth. During relocation, the robot can send an image acquisition request to the camera, and the camera, in response to the image acquisition request, returns the currently acquired image, i.e., the camera image, to the robot.

[0027] Step 204: Perform edge recognition on objects with straight edges in the camera image to obtain at least two first straight edges, and determine at least two second straight edges contained in the point cloud image.

[0028] In some embodiments, the robot can input camera images into a visual model, using the visual model to identify a target region from the camera images, where the target region has straight edges. The target region can be a region with straight edges within an indoor area, such as a wall, table, or bed. The visual model can be, but is not limited to, a ViT (VisionTransformer) model or a CNN (Convolutional Neural Network) model. The ViT model is a computer vision model based on the Transformer architecture. Then, the straight edges in the target region can be further obtained through the visual model to obtain the first straight edge. Figure 4 As shown, a schematic diagram of a camera image is provided, in which four wall surfaces are segmented from the camera image, namely wall surface 1 to wall surface 4.

[0029] In some embodiments, the visual model can also identify a robot region from the camera image, which is the image area occupied by the robot. The robot can determine the pixel at the center of the robot region, which can be called the pixel center of the robot, and determine the two intersection points of the straight line passing through the pixel center with the boundary of the robot region. The distance between these two intersection points is the diameter of the robot region, and the unit of the diameter is pixels, so the diameter can be called the pixel diameter.

[0030] In some embodiments, the robot can identify straight lines from a point cloud image by point cloud fitting and the RANSAC (Random Sample Consensus) algorithm to obtain the edge of a second straight line.

[0031] Among them, the first straight edge and the second straight edge are straight edges with straight line characteristics, such as the edges of walls, tables or beds with straight line characteristics.

[0032] Step 206: Match at least two first straight line edges and at least two second straight line edges to determine the robot's orientation in the global map.

[0033] The robot's orientation refers to the direction in which it moves. The global map is a map constructed by the robot after collecting environmental information about the entire home environment using LiDAR, and the laser point cloud data collected by the LiDAR at each location is stored in the global map. The home environment refers to the home environment to which an indoor area belongs. The home environment can contain multiple different indoor areas, such as room A, room B, and room C.

[0034] In some embodiments, the specific process of matching at least two first straight line edges and at least two second straight line edges may include: the robot determining the actual length of each first straight line edge in the indoor area and the actual length of each second straight line edge in the indoor area; traversing the at least two first straight line edges; for the traversed current first straight line edge, calculating the length difference between the actual length of each second straight line edge and the actual length of the current first straight line edge based on the actual length of the current first straight line edge; comparing the length difference with a preset length difference threshold; if the length difference is less than or equal to the length difference threshold, the second straight line edge is taken as a candidate second straight line edge corresponding to the current first straight line edge; if the candidate second straight line edge meets a preset condition, determining that the candidate second straight line edge and the current first straight line edge have a matching relationship, thereby obtaining the target first straight line edge (current first straight line edge) and the target second straight line edge (candidate second straight line edge) with a matching relationship. Here, due to the influence of noise (such as crumpled paper on the ground or a cluttered environment) or the impact of recognition accuracy, the length difference threshold can take into account the influence of noise, which helps to improve the accuracy of matching. The length difference threshold can be set according to actual needs, for example, it can be any value between 0m and 0.5m.

[0035] In some embodiments, the preset conditions include a first preset condition, which is that there is only one candidate second straight edge. Therefore, if the candidate second straight edge meets the preset condition, it can be determined that the candidate second straight edge and the current first straight edge have a matching relationship. This can include: if there is only one candidate second straight edge, then it can be determined that the candidate second straight edge and the current first straight edge have a matching relationship.

[0036] In some embodiments, if multiple candidate second straight line edges exist, a candidate second straight line edge that matches the current first straight line edge is determined from among the multiple candidate second straight line edges. Specifically, the preset condition further includes a second preset condition, which is that the relative positional relationship is consistent. Determining that the candidate second straight line edge matches the current first straight line edge when the candidate second straight line edge meets the preset condition may include: obtaining a first relative positional relationship between the current first straight line edge and the remaining first straight line edges, and obtaining a second relative positional relationship between the candidate second straight line edge and the remaining second straight line edges, wherein the actual lengths of the remaining first straight line edges and the actual lengths of the remaining second straight line edges have a one-to-one matching relationship. When the first relative positional relationship and the second relative positional relationship are consistent, it is determined that the candidate second straight line edge matches the current first straight line edge.

[0037] It should be noted that the first straight line edge may or may not match the second straight line edge. For ease of description, if the first straight line edge and the second straight line edge successfully match, the first straight line edge is referred to as the target first straight line edge, and the second straight line edge is referred to as the target second straight line edge.

[0038] In some embodiments, the robot can determine its orientation in the global map based on the angle between the first straight edge of the target and the second straight edge of the target, which have a matching relationship.

[0039] Step 208: Determine the robot's position on the global map.

[0040] In some embodiments, the robot can determine target historical point cloud data that matches the current point cloud data from historical point cloud data corresponding to multiple locations in a local map of the indoor area. The historical point cloud data corresponding to multiple locations in the local map is point cloud data collected by the robot using LiDAR when it was located at the corresponding location at a historical time. The historical time refers to the time when the robot was located at the corresponding location during the construction of the global map. The robot can determine its position in the global map based on the target position corresponding to the target historical point cloud data. Optionally, the robot can directly use the target position as its position in the global map to complete the robot's relocalization. The local map is a portion of the global map.

[0041] In some embodiments, the robot can determine the target historical point cloud data that matches the current point cloud data from the historical point cloud data corresponding to multiple locations in the global map. Furthermore, the target location corresponding to the target historical point cloud data is used as the robot's location in the global map to complete the robot's relocalization.

[0042] In some embodiments, for each of the multiple locations, the robot can determine the matching degree between the historical point cloud data corresponding to that location and the current point cloud data, and determine the target historical point cloud data that matches the current point cloud data from each historical point cloud data based on the matching degree.

[0043] Normally, point cloud data matching can be used to find the robot's current pose information (including position and orientation) on the global map. In this step, since the robot's orientation on the global map has already been determined in step 206 before determining its position, point cloud data matching in step 208 can be performed on the already determined orientation, avoiding the need to perform matching calculations on all possible orientations. This significantly improves the matching speed.

[0044] Step 210: Determine the robot's pose information in the global map based on the robot's orientation and position in the global map.

[0045] The pose information includes the robot's orientation and position on the global map.

[0046] In the aforementioned home service robot relocalization method, during the relocalization process, the current point cloud data collected by the robot's LiDAR from the current indoor area is acquired and converted into a point cloud image. Additionally, camera images collected by cameras installed in the indoor area are acquired. Edge recognition is performed on objects with straight edges in the camera images to obtain at least two first straight edges. At least two second straight edges are also identified in the point cloud image. The at least two first straight edges and the at least two second straight edges are matched to determine the robot's orientation in the global map, and the robot's position in the global map. Based on the robot's orientation and position in the global map, the robot's pose information in the global map is determined. Since the camera images captured by the camera can reflect the robot's position from a global perspective, the robot's orientation in the global map can be quickly determined by combining the first straight line edge in the camera image and the second straight line edge in the point cloud image, thus improving the efficiency of orientation determination. Furthermore, since the robot's orientation in the global map has been determined before determining its position, it avoids performing position matching calculations on all possible orientations, thereby improving the efficiency of position confirmation and thus improving the efficiency of determining pose information, which in turn improves relocalization efficiency.

[0047] In some embodiments, in step S206, matching at least two first straight line edges and at least two second straight line edges to determine the robot's orientation in the global map includes: determining a reference angle based on the angle between the target first straight line edge and the target second straight line edge that have a matching relationship among the at least two first straight line edges and at least two second straight line edges, wherein the matching relationship refers to the same edge in an indoor area; and determining the robot's orientation in the global map based on the robot's orientation in the point cloud image and the reference angle.

[0048] If the first straight line edge of the target and the second straight line edge of the target have a matching relationship, then it can be determined that the first straight line edge of the target and the second straight line edge of the target belong to the same edge in the indoor space.

[0049] The robot's orientation in the point cloud image can be determined based on its orientation in the LiDAR coordinate system. This orientation is predetermined and unchanging. Specifically, it can be determined using the robot's orientation in the LiDAR coordinate system and the coordinate transformation relationship when converting the current point cloud data into a point cloud image. The robot's orientation in the LiDAR coordinate system is the angle of its direction of travel within the LiDAR coordinate system. For example, the robot's direction of travel in the LiDAR coordinate system is aligned with the X-axis, with an angle of 0°.

[0050] In some embodiments, the first and second straight line edges of a target that have a matching relationship can be referred to as a pair of straight line edges. In the case where only a pair of straight line edges exist, the included angle between this pair of straight line edges is used as a reference angle.

[0051] In some embodiments, when there are multiple pairs of straight line edges, the average of the included angles between each pair of straight line edges can be used as the reference included angle, or the included angle between any pair of straight line edges can be used as the reference included angle.

[0052] In some embodiments, a reference angle is determined based on the angle between a target first straight edge and a target second straight edge that have a matching relationship among at least two first straight edge edges and at least two second straight edge edges. Specifically, this includes: determining a first angle of the target first straight edge in the image coordinate system of the camera image, and determining a second angle of the target second straight edge in the Cartesian coordinate system of the point cloud image, and using the difference between the first angle and the second angle as the reference angle.

[0053] In this embodiment, the reference angle can reflect the angular offset between the first angle and the second angle, and also reflects the angular offset between the robot's angle in the image coordinate system of the camera image and the angle in the Cartesian coordinate system of the point cloud image. Therefore, by converting the robot's orientation in the point cloud image according to the reference angle, the robot's orientation in the camera image can be obtained, and then the robot's orientation in the global map can be obtained. The method of obtaining the orientation is simple and improves the efficiency of determining the orientation.

[0054] In some embodiments, determining a reference angle based on the angle between a target first straight edge and a target second straight edge that have a matching relationship among at least two first straight edges and at least two second straight edges includes: determining the actual length of each of the at least two first straight edges in the indoor area, and determining the actual length of each of the at least two second straight edges in the indoor area; determining the relative positional relationship between each of the first straight edges and the relative positional relationship between each of the second straight edges; determining the target first straight edge and the target second straight edge that have a matching relationship from the at least two first straight edges and at least two second straight edges based on the actual length and the relative positional relationship; and determining the reference angle based on the angle between the target first straight edge and the target second straight edge.

[0055] It should be noted that the specific process of determining the target first straight line edge and the target second straight line edge with a matching relationship from at least two first straight line edges and at least two second straight line edges based on the actual length and relative position relationship can be the same as the specific process of matching at least two first straight line edges and at least two second straight line edges described above.

[0056] The second straight line edge can be understood as a group of data points in the current point cloud data. This group of data points is arranged in a straight line. Therefore, the distance between the data points at the two endpoints of this group of data points can be used as the actual length of the second straight line edge.

[0057] In some embodiments, the robot can obtain the pixel length of the first straight line edge and determine the actual length of the first straight line edge in the indoor area based on the pixel length. Here, the pixel length of the first straight line edge refers to the length of the first straight line edge in the camera image, measured in pixels.

[0058] In this embodiment, based on the actual length and relative positional relationship, the first straight line edge and the second straight line edge of the target with a matching relationship can be accurately and quickly determined, thereby quickly determining the reference angle.

[0059] In some embodiments, determining the actual length of each of at least two first straight line edges in the indoor region includes: determining the pixel diameter of the robot from a camera image and determining the diameter ratio between the pixel diameter and the actual diameter of the robot; for each first straight line edge, determining the actual length of the first straight line edge in the indoor region based on the pixel length and diameter ratio of the first straight line edge in the camera image.

[0060] The pixel diameter has already been described and will not be repeated here. Optionally, the robot is circular, and its actual diameter is predetermined. The actual diameter refers to the robot's true diameter size. The diameter ratio is either pixel diameter / actual diameter or actual diameter / pixel diameter.

[0061] In some embodiments, if the diameter ratio is pixel diameter / actual diameter, then the pixel length of the first straight edge / actual length of the first straight edge = diameter ratio, and thus, the actual length of the first straight edge = pixel length of the first straight edge / diameter ratio.

[0062] In some embodiments, if the diameter ratio is actual diameter / pixel diameter, then the actual length of the first straight line edge / pixel length of the first straight line edge = diameter ratio, and the actual length of the first straight line edge = pixel length of the first straight line edge × diameter ratio.

[0063] In this embodiment, the actual length is determined based on the diameter ratio determined by the robot. This method of determining the actual length is quick and can improve the efficiency of robot repositioning.

[0064] In some embodiments, determining the robot's orientation in the global map based on the robot's orientation and reference angle in the point cloud image includes: determining a first orientation of the robot in the image coordinate system of the camera image based on the robot's orientation and reference angle in the point cloud image; determining a second orientation of the robot in the camera coordinate system based on the first orientation; and determining the robot's orientation in the global map based on the camera's extrinsic parameter matrix and the second orientation.

[0065] The camera's extrinsic parameter matrix is ​​used to characterize the rotation and translation relationship between the camera coordinate system and the map coordinate system. The map coordinate system refers to the coordinate system used by the global map.

[0066] In some embodiments, the robot can determine its orientation in the camera coordinate system based on its orientation in the point cloud image and a reference angle, i.e., the robot's first orientation in the image coordinate system of the camera image. Then, based on the robot's orientation in the image coordinate system, the robot's second orientation in the camera coordinate system is determined. Finally, based on the camera's extrinsic parameter matrix and the robot's orientation in the camera coordinate system, the robot's orientation in the global map is determined.

[0067] It should be noted that when the camera captures the image from a top-down perspective, the robot's orientation in the image coordinate system of the camera image is consistent with the robot's orientation in the camera coordinate system. If the actual view is not from a top-down perspective, there will be some loss of accuracy, but this loss is acceptable within the error range.

[0068] For example, if the robot's orientation in the point cloud image is the angle of the robot's direction of travel in the Cartesian coordinate system as 0° and the reference angle is 30°, then the robot's first orientation in the camera image is the angle of the robot's direction of travel in the image coordinate system as 30°, and thus the robot's second orientation is the angle of the robot's direction of travel in the camera coordinate system as 30°. Then, based on the camera's extrinsic parameter matrix, the robot's orientation in the global map can be obtained.

[0069] In this embodiment, based on the transformation relationship between coordinate systems, the first orientation can be conveniently and quickly converted to the orientation in the global map.

[0070] In some embodiments, determining the position of the robot in the global map includes: determining the actual vertical distance between the robot's body center and each first straight line edge, and determining the shortest actual vertical distance from the actual vertical distances; determining the robot's relocation area from a local map corresponding to the indoor area based on the shortest actual vertical distance; and determining the robot's position in the global map from multiple locations in the relocation area.

[0071] The local map is a part of the global map. The coordinate system of the local map shares the same orientation as the global map, but their origins differ. The actual vertical distance can be calculated based on the diameter scale. The shortest actual vertical distance is the shortest of all actual vertical distances. The actual vertical distance is the length of the perpendicular line from the center of the fuselage to the edge of the first straight line, measured in meters (m).

[0072] In this context, the area of ​​the relocation region is smaller than the area of ​​the local map, and the distance between the boundary of the relocation region and the boundary of the local map is less than or equal to the shortest actual vertical distance. For example, consider three first straight edges, line edge 1 to line edge 3. The actual vertical distance between the fuselage center and line edge 1 is 3 meters, the actual vertical distance between the fuselage center and line edge 2 is 2 meters, and the actual vertical distance between the fuselage center and line edge 3 is 1 meter. Since the shortest actual vertical distance is 1 meter, the distance between the boundary of the relocation region and the boundary of the local map is less than or equal to 1 meter. For example, if the local map is rectangular, the relocation region is also rectangular, and the distance between the boundary of the relocation region and the boundary of the local map is 1 meter. That is, the distance between the left boundary of the relocation region and the left boundary of the local map is 1 meter, the distance between the right boundary of the relocation region and the right boundary of the local map is 1 meter, the distance between the upper boundary of the relocation region and the upper boundary of the local map is 1 meter, and the distance between the lower boundary of the relocation region and the lower boundary of the local map is 1 meter.

[0073] In some embodiments, the robot's position in the global map can be determined from each position in the relocation area, or a set number of positions can be uniformly selected in the relocation area according to a set distance interval and / or a set angle range, and the robot's position in the global map can be determined from the selected set number of positions.

[0074] In some embodiments, the robot determines the target historical point cloud data that matches the current point cloud data from the historical point cloud data corresponding to multiple locations in the relocation area, and determines the location corresponding to the target historical point cloud data as the robot's location in the global map.

[0075] In this embodiment, by defining a relocation area, the robot's position can be quickly confirmed, improving relocation efficiency. Furthermore, since the distance between the boundary of the relocation area and the boundary of the local map is less than or equal to the shortest actual vertical distance, the area of ​​the relocation area in the local map can be maximized. While improving relocation efficiency, this avoids the problem that the robot's position in the global map cannot be determined from multiple locations in the relocation area due to the relocation area being too small, leading to relocation failure. This improves the success rate of relocation.

[0076] In some embodiments, determining the actual vertical distance between the robot's body center and each first straight line edge includes: determining the robot's pixel diameter from a camera image and determining the diameter ratio between the pixel diameter and the robot's actual diameter; determining the robot's center pixel from the camera image and determining the pixel vertical distance between the center pixel and each first straight line edge; and for each first straight line edge, determining the actual vertical distance between the robot's body center and the first straight line edge based on the diameter ratio and the pixel vertical distance.

[0077] Here, the center pixel refers to the center of the aforementioned pixel. The vertical distance of the pixel is the length of the perpendicular line from the center pixel to the edge of the first straight line, and the unit is pixels.

[0078] In some embodiments, if the diameter ratio is pixel diameter / actual diameter, then pixel vertical distance / actual vertical distance = diameter ratio, and thus, actual vertical distance = pixel vertical distance / diameter ratio.

[0079] In some embodiments, if the diameter ratio is actual diameter / pixel diameter, then actual vertical distance / pixel vertical distance = diameter ratio, and actual vertical distance = pixel vertical distance × diameter ratio.

[0080] In this embodiment, the actual vertical distance is determined based on the diameter ratio determined by the robot. This method of determining the actual vertical distance is quick and can improve the efficiency of robot repositioning.

[0081] In some embodiments, determining the robot's position in the global map from multiple locations in the relocation region includes: determining the robot's position in a local map from multiple locations in the relocation region; and determining the robot's position in the global map based on the robot's position in the local map and the position of the local map in the global map.

[0082] In some embodiments, as described above, the robot can determine the matching degree between historical point cloud data corresponding to a location and current point cloud data, and determine the target historical point cloud data that matches the current point cloud data from each historical point cloud data based on the matching degree. Furthermore, the robot can determine the historical point cloud data with the highest matching degree, identify the historical point cloud data with the highest matching degree as the target historical point cloud data, and use the location corresponding to the target historical point cloud data as the robot's location on the local map.

[0083] When matching historical point cloud data with current point cloud data, the matching can be based on the distribution pattern of data points in the historical point cloud data and the distribution pattern of data points in the current point cloud data.

[0084] In some embodiments, for each of the multiple locations in the relocation area, the historical point cloud data corresponding to the location in the relocation area is matched with the current point cloud data to obtain the matching degree of each of the multiple locations in the relocation area. The robot can determine the maximum matching degree among the matching degrees of each location. If the maximum matching degree is greater than a preset matching degree threshold, the location corresponding to the maximum matching degree is taken as the robot's location in the local map.

[0085] The location corresponding to the target's historical point cloud data is used as the robot's position on the local map. The matching threshold can be set according to actual needs, and the matching score can be used as the matching degree.

[0086] In some embodiments, the robot can convert its position in the local map into its position in the global map based on the position of the local map in the global map.

[0087] In this embodiment, the robot's position in the local map can be quickly converted into its position in the global map based on the position of the local map in the global map.

[0088] In some embodiments, before determining the robot's relocation area from the local map corresponding to the indoor area based on the shortest actual vertical distance, the method further includes: determining the local map corresponding to the indoor area based on the association between the camera and the local map.

[0089] The robot can pre-store the association between cameras and local maps. For example, in a home environment, there are two rooms, room A and room B. Camera 1 is installed in room A and camera 2 is installed in room B. If the camera image is captured by camera 1, the local map associated with camera 1 can be determined from the global map to obtain the local map corresponding to the indoor area, i.e., room A.

[0090] In this embodiment, the local map corresponding to the indoor area can be quickly determined by the pre-stored association between the camera and the local map.

[0091] Optionally, the location of the camera installation in the home environment can be determined in advance, and the indoor area to which the installation location belongs can be confirmed. Furthermore, the camera can be bound to a local map corresponding to the indoor area. For example, the location of the camera can be marked on the corresponding local map through an APP (Application) to establish the association between the camera and the local map.

[0092] In some embodiments, such as Figure 5 As shown, a method for relocation of a home service robot is provided, including the following steps:

[0093] Step 502: During the relocation process, acquire the current point cloud data collected by the LiDAR in the robot from the current indoor area, convert the current point cloud data into a point cloud image, and acquire the camera image collected by the camera installed in the indoor area.

[0094] Step 504: Perform edge recognition on objects with straight edges in the camera image to obtain at least two first straight edges, and determine at least two second straight edges contained in the point cloud image.

[0095] Step 506: Determine the actual length of each of the at least two first straight line edges in the indoor area, and determine the actual length of each of the at least two second straight line edges in the indoor area.

[0096] Step 508: Determine the relative positional relationship between each first straight line edge and the relative positional relationship between each second straight line edge. Based on the actual length and relative positional relationship, determine the target first straight line edge and the target second straight line edge that have a matching relationship from at least two first straight line edges and at least two second straight line edges.

[0097] Step 510: Determine the reference angle based on the angle between the first straight edge of the target and the second straight edge of the target.

[0098] Step 512: Based on the robot's orientation and reference angle in the point cloud image, determine the robot's first orientation in the image coordinate system of the camera image; based on the first orientation, determine the robot's second orientation in the camera coordinate system of the camera; based on the camera's extrinsic matrix and the second orientation, determine the robot's orientation in the global map.

[0099] Step 514: Determine the actual vertical distance between the robot's body center and the edge of each first straight line, and determine the shortest actual vertical distance from each actual vertical distance. Based on the shortest actual vertical distance, determine the robot's relocation area from the local map corresponding to the indoor area.

[0100] Step 516: Determine the robot's position in the local map from multiple locations in the relocation area.

[0101] Step 518: Determine the robot's position in the global map based on the robot's position in the local map and the position of the local map in the global map.

[0102] In this embodiment, since the camera image captured by the camera can reflect the robot's position from a global perspective, the robot's orientation in the global map can be quickly determined by combining the first straight line edge in the camera image and the second straight line edge in the point cloud image, which improves the efficiency of determining the orientation and thus improves the efficiency of determining the pose information, thereby improving the relocation efficiency; and by limiting the relocation area, the robot's position can be quickly confirmed, further improving the relocation efficiency.

[0103] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0104] Based on the same inventive concept, this application also provides a home service robot for implementing the aforementioned home service robot relocation method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more home service robot embodiments provided below can be found in the limitations of the home service robot relocation method described above, and will not be repeated here.

[0105] In some embodiments, such as Figure 6 As shown, a home service robot is provided, including: an image acquisition module 602, an edge determination module 604, an orientation determination module 606, a position determination module 608, and a pose determination module 610, wherein:

[0106] The image acquisition module 602 is used to acquire the current point cloud data collected by the LiDAR in the robot from the current indoor area during the repositioning process, convert the current point cloud data into a point cloud image, and acquire camera images collected by the camera installed in the indoor area.

[0107] The edge determination module 604 is used to perform edge recognition on objects with straight edges in the camera image, obtain at least two first straight edges, and determine at least two second straight edges contained in the point cloud image.

[0108] Orientation determination module 606 is used to match at least two first straight line edges and at least two second straight line edges to determine the robot's orientation in the global map.

[0109] The location determination module 608 is used to determine the robot's position on the global map.

[0110] The pose determination module 610 is used to determine the pose information of the robot in the global map based on the robot's orientation and position in the global map.

[0111] In some embodiments, the orientation determination module 606 is further configured to determine a reference angle based on the angle between a target first straight edge and a target second straight edge that have a matching relationship among at least two first straight edge edges and at least two second straight edge edges, wherein the matching relationship refers to the same edge in an indoor area; and to determine the orientation of the robot in the global map based on the robot's orientation in the point cloud image and the reference angle.

[0112] In some embodiments, the orientation determination module 606 is further configured to determine the actual length of each of the at least two first straight line edges in the indoor area, and to determine the actual length of each of the at least two second straight line edges in the indoor area; determine the relative positional relationship between each of the first straight line edges and the relative positional relationship between each of the second straight line edges; based on the actual length and relative positional relationship, determine a target first straight line edge and a target second straight line edge with a matching relationship from the at least two first straight line edges and the at least two second straight line edges; and determine a reference angle based on the angle between the target first straight line edge and the target second straight line edge.

[0113] In some embodiments, the orientation determination module 606 is further configured to determine the pixel diameter of the robot from the camera image and determine the diameter ratio between the pixel diameter and the actual diameter of the robot; for each first straight line edge, the actual length of the first straight line edge in the indoor area is determined based on the pixel length and diameter ratio of the first straight line edge in the camera image.

[0114] In some embodiments, the orientation determination module 606 is further configured to determine a first orientation of the robot in the image coordinate system of the camera image based on the robot's orientation in the point cloud image and a reference angle; determine a second orientation of the robot in the camera coordinate system based on the first orientation; and determine the robot's orientation in the global map based on the camera's extrinsic matrix and the second orientation.

[0115] In some embodiments, the position determination module 608 is further configured to determine the actual vertical distance between the robot's body center and each first straight line edge, and determine the shortest actual vertical distance from each actual vertical distance; determine the robot's relocation area from the local map corresponding to the indoor area based on the shortest actual vertical distance; wherein the area of ​​the relocation area is smaller than the area of ​​the local map, and the distance between the boundary of the relocation area and the boundary of the local map is less than or equal to the shortest actual vertical distance; and determine the robot's position in the global map from multiple positions in the relocation area.

[0116] In some embodiments, the position determination module 608 is further configured to determine the pixel diameter of the robot from the camera image and determine the diameter ratio between the pixel diameter and the actual diameter of the robot; determine the center pixel of the robot from the camera image and determine the pixel vertical distance between the center pixel and each first straight line edge; and for each first straight line edge, determine the actual vertical distance between the robot's body center and the first straight line edge based on the diameter ratio and the pixel vertical distance.

[0117] In some embodiments, the position determination module 608 is further configured to determine the position of the robot in the local map from multiple positions in the relocation area; and determine the position of the robot in the global map based on the position of the robot in the local map and the position of the local map in the global map.

[0118] In some embodiments, the location determination module 608 is further configured to determine the local map corresponding to the indoor area based on the association between the camera and the local map.

[0119] The various modules in the aforementioned home service robot can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0120] In some embodiments, a home service robot system is provided, the home service robot system including a home service robot.

[0121] In some embodiments, a computer device is provided, which may be a control device in a home service robot, and the home service robot relocation method may be performed by the control device. Its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data involved in the home service robot relocation method. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a home service robot relocation method.

[0122] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described home service robot relocation method.

[0124] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described home service robot relocation method.

[0125] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described home service robot relocation method.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for repositioning a home service robot, characterized in that, The method includes: During the relocation process, the robot acquires the current point cloud data collected by the LiDAR in the current indoor area, converts the current point cloud data into a point cloud image, and acquires the camera image collected by the camera installed in the indoor area. Edge recognition is performed on objects with straight edges in the camera image to obtain at least two first straight edges, and at least two second straight edges contained in the point cloud image are determined. Based on the angle between the target first straight edge and the target second straight edge that have a matching relationship among the at least two first straight edge and the at least two second straight edge, a reference angle is determined, wherein the matching relationship refers to the same edge in the indoor area; Based on the robot's orientation in the point cloud image and the reference angle, the robot's orientation in the global map is determined; Determine the robot's position on the global map; Based on the robot's orientation and position in the global map, the robot's pose information in the global map is determined.

2. The method according to claim 1, characterized in that, Determining the robot's position in the global map includes: From the historical point cloud data corresponding to multiple locations in the global map, determine the target historical point cloud data that matches the current point cloud data; The target location corresponding to the historical point cloud data is used as the robot's position in the global map.

3. The method according to claim 1, characterized in that, The step of determining a reference angle based on the angle between the target first straight edge and the target second straight edge that have a matching relationship among the at least two first straight edge edges and the at least two second straight edge edges includes: Determine the actual length of each of the at least two first straight line edges in the indoor area, and determine the actual length of each of the at least two second straight line edges in the indoor area; Determine the relative positional relationship between each of the first straight line edges and the relative positional relationship between each of the second straight line edges; Based on the actual length and relative positional relationship, target first straight line edges and target second straight line edges with matching relationship are determined from the at least two first straight line edges and the at least two second straight line edges; A reference angle is determined based on the angle between the first straight edge and the second straight edge of the target.

4. The method according to claim 3, characterized in that, Determining the actual length of each of the at least two first straight line edges in the indoor area includes: The pixel diameter of the robot is determined from the camera image, and the diameter ratio between the pixel diameter and the actual diameter of the robot is determined. For each of the first straight line edges, the actual length of the first straight line edge in the indoor area is determined based on the pixel length of the first straight line edge in the camera image and the diameter ratio.

5. The method according to claim 1, characterized in that, Determining the robot's orientation in the global map based on the robot's orientation in the point cloud image and the reference angle includes: Based on the robot's orientation in the point cloud image and the reference angle, the robot's first orientation in the image coordinate system of the camera image is determined; The robot's second orientation in the camera coordinate system is determined based on the first orientation; Based on the extrinsic matrix of the camera and the second orientation, the orientation of the robot in the global map is determined.

6. The method according to any one of claims 1, 3 to 5, characterized in that, Determining the robot's position in the global map includes: Determine the actual vertical distance between the robot's body center and each edge of the first straight line, and determine the shortest actual vertical distance from among the actual vertical distances; The robot's relocation area is determined from the local map corresponding to the indoor area based on the shortest actual vertical distance; wherein the area of ​​the relocation area is smaller than the area of ​​the local map, and the distance between the boundary of the relocation area and the boundary of the local map is less than or equal to the shortest actual vertical distance. The position of the robot in the global map is determined from multiple locations in the relocation area.

7. The method according to claim 6, characterized in that, Determining the actual vertical distance between the robot's body center and each of the first straight line edges includes: The pixel diameter of the robot is determined from the camera image, and the diameter ratio between the pixel diameter and the actual diameter of the robot is determined. The center pixel of the robot is determined from the camera image, and the vertical distance between the center pixel and the edge of each of the first straight lines is determined. For each edge of the first straight line, the actual vertical distance between the robot's body center and the edge of the first straight line is determined based on the diameter ratio and the pixel vertical distance.

8. The method according to claim 6, characterized in that, Determining the robot's position in the global map from multiple locations within the relocation region includes: The position of the robot in the local map is determined from multiple locations within the relocation area; The position of the robot in the global map is determined based on the position of the robot in the local map and the position of the local map in the global map.

9. The method according to claim 6, characterized in that, Before determining the robot's relocation area from the local map corresponding to the indoor area based on the shortest actual vertical distance, the method further includes: Based on the association between the camera and the local map, the local map corresponding to the indoor area is determined.

10. A home service robot, characterized in that, The home service robots include: The image acquisition module is used to acquire, during the repositioning process, the current point cloud data collected by the lidar in the robot from the current indoor area, convert the current point cloud data into a point cloud image, and acquire camera images collected by the camera installed in the indoor area. An edge determination module is used to perform edge recognition on objects with straight edges in the camera image, obtain at least two first straight edges, and determine at least two second straight edges contained in the point cloud image; An orientation determination module is used to determine a reference angle based on the angle between the target first straight line edge and the target second straight line edge that have a matching relationship among the at least two first straight line edges and the at least two second straight line edges, and to determine the orientation of the robot in the global map based on the robot's orientation in the point cloud image and the reference angle, wherein the matching relationship refers to the same edge in the indoor area; A location determination module is used to determine the robot's position in the global map; The pose determination module is used to determine the pose information of the robot in the global map based on the robot's orientation and position in the global map.

11. A home service robot system, characterized in that, The system includes the home service robot of claim 10.

Images