Working boundary construction method for robot, and robot

Abstract

Embodiments of the present application provide a working boundary construction method for a robot, and a robot. The method comprises: acquiring M frame images collected along an expected boundary, and determining an initial pose of each image among the M frame images, wherein M is a positive integer greater than 1; selecting n key frame images from among the M frame images on the basis of the initial pose of each image, and using the n key frame images to correct the corresponding initial poses to obtain optimized poses, wherein n is an integer and 1<n≤M; and constructing the working boundary of the robot on the basis of the optimized poses of the key frame images, thereby quickly and accurately setting the working boundary.

Description

A method for constructing a working boundary of a robot and a robot

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 30, 2023, with application number 202311427617.7, entitled “A method for constructing a working boundary of a robot and a robot”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of robots, and in particular to a method for constructing a working boundary of a robot and the robot. Background Art

[0003] With the development of intelligent technology, autonomous equipment, such as robotic lawn mowers and sweepers, is becoming increasingly widespread. One key aspect of these autonomous devices is the definition of work boundaries. For example, when a robotic lawn mower is mowing, accurately defining the work boundaries is essential for efficient mowing.

[0004] In the existing technology, one method is to bury wires at the edge of the lawn and pass alternating current through the wires. Then, a device for detecting the alternating current is installed on the robot chassis to detect and determine the working boundary. However, this method requires physical wiring, which is very troublesome, and the wires need to be powered on all the time, resulting in energy waste. Another method is to use a global camera to obtain an image of the entire lawn scene, and manually select and draw the working boundary on the image. However, this solution is only applicable to open scenes. Obstructions such as houses and trees will affect the confirmation of the working boundary. Furthermore, images can be obtained through a camera and semantic segmentation methods can be used to segment the lawn and non-grass areas. However, this requires the robot to perform semantic analysis of the image in real time, which increases the amount of calculation and the computing cost of the equipment.

[0005] Therefore, there is an urgent need for a technical solution that can quickly set the working boundary at a low computational cost.

[0006] Summary of the Invention

[0007] The present application provides a method for constructing a working boundary of a robot and a robot, which can quickly and accurately set the working boundary.

[0008] In a first aspect, a method for constructing a robot's working boundary is provided, the method comprising: acquiring M frames of images collected along an expected boundary, determining an initial pose of each image in the M frames, where M is a positive integer greater than 1; screening n frames of key frame images from the M frames of images according to the initial pose of each image, and using the n frames of key frame images to correct the corresponding initial poses to obtain optimized poses, where n is an integer and 1<n≤M; and constructing the robot's working boundary based on the optimized poses of each frame of key frame images.

[0009] This application uses the user's movement trajectory as the robot's working boundary, and the user can select the customized robot working area boundary according to their needs, so that the method provided by this application has high compatibility. Regardless of whether the expected boundary is a simple two-dimensional boundary or a complex three-dimensional boundary, the corresponding working boundary can be quickly obtained by obtaining an image collected along the expected boundary. This application reduces the robot's requirements for its working environment and improves the user experience.

[0010] In addition, the method of constructing the working boundary in the present application does not require the identification and processing of grass and non-grass in the image. It only needs to determine the posture of the image when it is captured based on the matching results of feature points in adjacent images. The posture connection method of multiple frames of images is consistent with the working boundary, which can achieve rapid setting of the working boundary at a lower computational cost.

[0011] In a second aspect, a robot is provided, comprising: an acquisition device for acquiring M frames of images of an expected boundary, and a processor for processing the M frames of images according to the method of the first aspect and any possible implementation method of the first aspect to obtain a working boundary of the robot's working area.

[0012] In a third aspect, a computer-readable storage medium is provided, which stores a program code for execution by a device, wherein the program code is used to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 is a schematic diagram of a robot application system provided by an embodiment of the present application.

[0014] FIG2 is a flowchart of a method for constructing a robot working boundary according to an embodiment of the present application.

[0015] FIG3 is a schematic diagram of a path for collecting trajectories in constructing a global map according to an embodiment of the present application.

[0016] FIG4 is a schematic diagram of a robot according to an embodiment of the present application.

[0017] FIG5 shows a schematic diagram of a working boundary generating device provided in the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0019] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0020] Figure 1 is a schematic diagram of an application scenario 100 of a robotic system according to an embodiment of the present application. In the robotic application scenario 100 shown in Figure 1 , robot 110 is an autonomous operating device, such as a lawn mower, cleaning robot, or other type of robot. To ensure the safe operation of robot 110, a work boundary 120 must be set for robot 110. Specifically, robot 110 must be confined within work boundary 120 to perform autonomous operations.

[0021] The robot 110 may be equipped with a first acquisition device 102 for collecting images or videos of the robot 110's surroundings. The robot 110 may analyze the images or videos collected by the first acquisition device 102, for example, to confirm the robot 110's current location, confirm whether the robot 110 has reached the work boundary 120, confirm the robot 110's current automatic operation mode (e.g., analyzing the height of surrounding grass to determine how to mow it), etc. Aside from the analysis and processing related to identifying the work boundary 101, the present embodiment of the application does not limit other analysis and processing performed after the robot 110 collects images or videos.

[0022] As a possible implementation, the robot 110 can carry the first acquisition device 102 to shoot along a specified route (such as a predetermined boundary). For example, the shooting route of the robot 110 can be controlled by the user 103, or the robot 110 can automatically shoot according to a specified method.

[0023] As another possible implementation, the first acquisition device 102 is detachable. For example, the user 103 can remove the first acquisition device 102 from the robot 110 and then hold the first acquisition device 102 to capture images along the working boundary 120 of the robot 110, thereby obtaining multiple frames of images.

[0024] As another possible implementation method, the first acquisition device 102 may also have a motion parameter acquisition function, and the motion parameters corresponding to the robot 110 can be obtained when acquiring multiple frames of images; wherein the motion parameters can be one or more parameters such as motion direction, motion speed, motion acceleration, offset angle, etc.

[0025] In addition, the user 103 may also use other types of second acquisition devices 104 installed outside the robot 110 body to acquire images. For example, the user 103 may use a mobile terminal with a camera function to acquire images of the working boundary 120 .

[0026] After capturing the image, the first capture device 102 or the second capture device 104 can independently process the image data. For example, the captured image and motion parameters can be processed to output the pose of each frame of the image, thereby constructing the work boundary 120. As another implementation, the first capture device 102 or the second capture device 104 can also not process the image data, but instead transmit the captured image and motion parameters to the processing device of the robot 110 or other processing device (such as another computing device) via a wired or wireless method. The processing device of the robot 110 or other processing device processes the image data to construct the work boundary 120. The embodiment of the present application does not limit the processing subject of the above-mentioned image data. For example, the processing process can also be distributed to different devices for execution.

[0027] The robot 110 can also improve the global map based on the work boundary 120. Once the work boundary is determined, the robot can autonomously capture images within the work boundary 120 to improve the construction of the global map. Alternatively, the user 103 can also independently capture images of the global map. The robot 110 can also be equipped with a processor. During operation, the processor repositions the robot based on the work boundary or the global map, obtaining the robot's coordinates and boundary information in the global map. This allows the robot to accurately locate itself during operation, thereby providing more precise service to the work area.

[0028] For example, the first acquisition device 102 or the second acquisition device 104 may acquire images using a binocular or multi-camera, or a monocular camera. As a possible scenario, the acquisition device may also include a motion parameter recording module to record the motion parameters of the robot 110 at the time of capturing the image.

[0029] It should be understood that whether the first acquisition device 102 or the second acquisition device 104 includes a motion parameter recording module may depend on whether the first acquisition device 102 or the second acquisition device 104 can directly obtain absolute scale, that is, the proportional relationship between the actual size of the scene captured by the first acquisition device 102 or the second acquisition device 104 and the size of the scene in the image captured including the scene. For binocular cameras, due to their characteristics, the binocular camera has a baseline, and the distance between the baselines is known. Therefore, the binocular camera baseline can be used as reference information to directly obtain the absolute scale of the captured image, and thus the relative change in the position of the object in the image can be calculated based on the absolute scale. For monocular cameras, due to the lack of a reference baseline, the monocular camera cannot directly obtain the absolute scale, and therefore requires the use of a motion parameter recording module to assist in determining the relative change in the position of the object in the image.

[0030] It should be understood that the first acquisition device 102 or the second acquisition device 104 in the present application can be configured with not only a camera or a motion parameter recording module, but also a global positioning system (GPS) / real-time kinematic (RTK) / ultra-wide band (UWB) / lidar and inertial measurement unit (IMU). The first acquisition device 102 or the second acquisition device 104 may not even include a camera or IMU, such as only a lidar, and the present application does not impose any restrictions on this.

[0031] FIG2 is a flow chart of a method for constructing a robot working boundary according to an embodiment of the present application. The method of FIG2 can be implemented by the robot 110 of FIG1 , or by a processing device independent of the robot 110 .

[0032] As shown in FIG2 , the method for constructing the robot working boundary can be performed simultaneously with the acquisition process or after all images are acquired. The method for setting the working boundary can include:

[0033] S210 , acquiring M frames of images collected along the expected boundary and determining an initial pose of each image in the M frames of images, where M is an integer greater than or equal to 1.

[0034] In one embodiment, acquiring M frames of images along a desired boundary includes: a user moving a handheld acquisition device along the desired boundary, and when the user begins moving, the acquisition device is simultaneously activated to collect data. When the acquisition device begins operating, the current position is simultaneously marked as the starting position of the boundary of the work area, where the desired boundary is the boundary of the robot's custom working area. Furthermore, each time the user moves, multiple frames of images are continuously captured at the current point of the desired boundary to obtain M frames of images. After the handheld acquisition device continuously moves along the boundary of the custom robot's working area for a period of time, capturing multiple frames of images, and then returns to the starting position or near the starting position, marking the end of the boundary, the acquisition device can be deactivated, completing acquisition of the current working area.

[0035] It should be noted that when the process of setting the working boundary can be carried out synchronously with the acquisition process, the M-frame image is the multiple-frame image acquired at the current position; when the process of setting the working boundary is carried out after all images are acquired along the expected boundary, the M-frame image is all images.

[0036] Furthermore, in the process of using the acquisition device to collect data, when the user moves to a position close to the starting position, the image similarity between the image collected at the starting position and the image collected when close to the starting position can be compared to determine whether the collected data has completed a closed loop, thereby determining whether the collection work can be completed.

[0037] It should be noted that when the acquisition device includes a motion parameter recording module, each time the acquisition device captures a frame of image of the working area, the motion parameters of the acquisition device when capturing the local image of the current frame will also be synchronously recorded for subsequent image processing to obtain a global map and the boundaries of the working area.

[0038] This application uses the user's movement trajectory as the robot's working boundary, and the user can select the customized robot working area boundary according to their needs, so that the method provided by this application has high compatibility. Regardless of whether the expected boundary is a simple two-dimensional boundary or a complex three-dimensional boundary, the corresponding working boundary can be quickly obtained by obtaining an image collected along the expected boundary. This application reduces the robot's requirements for its working environment and improves the user experience.

[0039] It should be understood that the M frames of images collected can be ground images of the boundary of the working area, or environmental images around the boundary of the working area. The ground images or environmental images can be two-dimensional or three-dimensional images such as color images, depth images, point cloud images, etc. The collection method can be to obtain images frame by frame by taking photos, or to obtain images continuously by shooting videos. This application does not limit this.

[0040] There are multiple ways to determine the initial pose of each image in the M-frame image. As a possible implementation method, when the acquisition device includes a motion parameter recording module, determining the initial pose of each image in the M-frame image includes: obtaining the motion parameters corresponding to each image in the M-frame image, and determining the predicted pose of the m-frame image based on the motion parameters of the m-1-th frame image and the m-1-th frame image in the M-frame image, where m is an integer and 1<m≤M; obtaining matching features of the m-1-th frame image and the m-1-th frame image, and using the matching features as observation constraints to update the predicted pose to obtain the initial pose of the m-frame image; traversing all images in the M-frame image to obtain the initial pose of each image, and the initial pose can be a translation matrix and / or a rotation matrix; when the m-th frame image is the first frame, its corresponding m-th frame image can be the first frame image, and the same first frame image can be used to obtain the corresponding initial pose, which is not limited here.

[0041] In one embodiment, there are multiple methods for determining the predicted pose of the m-th frame image using the motion parameters of the m-1-th frame image. As one possible example, the corresponding motion parameters of the m-1-th frame image are input into a preset position estimation model, and the preset position estimation model is used to predict the predicted pose of the first acquisition device 102 or the second acquisition device 104 when acquiring the m-th frame image. It should be understood that the predicted pose is an initial estimate of the m-th frame image. However, due to errors caused by factors such as drift of the motion parameter recording module or large changes in pose, the predicted pose may be inaccurate, and therefore needs to be optimized to obtain the initial pose.

[0042] In one embodiment, obtaining matching features between the m-th frame image and the m-1-th frame image, and using the matching features of the two adjacent frame images as observation constraints to update the predicted pose to obtain the initial pose of the m-th frame image includes: extracting features from the m-th frame image, where the extracted features may be corner features, or the images may be input into a preset neural network model to extract features from each image to obtain corresponding key feature points; calculating descriptors of the key feature points in the m-th frame image and matching the descriptors of the key feature points in the m-th frame image with the descriptors of the key feature points in the m-1-th frame image to obtain matching features between the m-th frame image and the m-1-th frame image; wherein the descriptor can be expressed as a feature vector for the key feature points; and using the position information of the matching features as observation constraints to update the predicted pose to obtain the initial pose of the m-th frame image.

[0043] In another embodiment, matching features between the m-th frame image and the m-1-th frame image are obtained, and the matching features of the two adjacent frame images are used as observation constraints to update the predicted pose to obtain the initial pose of the m-th frame image, including: taking the key feature point obtained in the m-1-th frame image as a reference point, tracking the position of the reference point in the m-th frame image, because the key feature points between different frames are the same reference point, the same reference point between different frames can be regarded as a matching feature; based on the position of the reference point between the m-1-th frame image and the m-th frame image as an observation constraint, the predicted pose is updated to obtain the initial pose of the m-th frame image.

[0044] In one embodiment, the predicted pose is updated using matching features as observation constraints to obtain an initial pose of the m-th frame image, including: using chi-square distribution to check whether there are outliers between the feature matching between the m-th frame image and the m-1-th frame image features, so as to eliminate outliers in the predicted pose and obtain a more accurate initial pose of the m-th frame image.

[0045] Specifically, the three-dimensional point corresponding to the captured object in the world coordinate system is projected onto the imaging plane of the acquisition device using preset camera internal and external parameters. The predicted position value of the feature point of the three-dimensional point on the mth and m-1th frames of the image is obtained. The chi-square value is constructed based on the difference between the predicted position value of the predicted feature point and the observed position value of the corresponding observed feature point in the mth and m-1th frames of the image captured by the acquisition device. The chi-square value is used to measure the degree of difference between the predicted position value of the predicted feature point and the observed position value of the corresponding observed feature point in the image. The corresponding critical value is found in the chi-square distribution table based on the degrees of freedom and significance level, and the calculated chi-square value is compared with the critical value. If the chi-square value is less than the critical value, it indicates that there are no outliers in the data. If the chi-square value is greater than the critical value, it indicates that there are outliers in the data and the corresponding data needs to be eliminated. The degrees of freedom are determined by the number of different groups in the data subjected to the chi-square test; the significance level is a custom preset threshold that represents the acceptable level of error during the test; the values ​​in the chi-square distribution table are statistically calculated and pre-calibrated, and different degrees of freedom and significance levels correspond to different critical values.

[0046] Furthermore, if the chi-square value is greater than the critical value, it means that there are anomalies in the data. In this case, it is necessary to eliminate the abnormal observation values ​​of the three-dimensional feature points of the collected object in the corresponding image, and only use the normal observation values ​​corresponding to the three-dimensional feature points in two adjacent frames of images to calculate the relative pose, so as to ensure a more accurate initial pose.

[0047] It should be understood that in the above process, in addition to using the chi-square distribution, the T distribution or the F distribution can also be used to eliminate abnormal data, and this application does not limit this.

[0048] In the embodiments provided in the present application, due to the drift of motion parameters or image calculation errors, etc., there may be abnormal values ​​in the determined predicted pose, which will lead to a decrease in boundary quality. By constructing a test value to constrain the predicted pose, bad values ​​can be eliminated, thereby ensuring the rationality of the estimation and improving the quality of the determined working boundary.

[0049] As another implementation method, when the acquisition device does not include a motion parameter recording module, determining the initial pose of the mth frame image in the M frame images includes: obtaining matching features of the mth frame image and the m-1th frame image in the M frame images, and calculating the calculated pose of the mth frame image based on the matching features; projecting the three-dimensional feature points corresponding to the acquisition object in the world coordinate system onto the imaging plane of the acquisition device through preset camera internal and external parameters, obtaining the predicted position values ​​of the three-dimensional points corresponding to the predicted feature points on the m-1th frame image and the m-1th frame image, using the observed feature points of the m-1th frame image and the m-1th frame image acquired by the acquisition device that match the predicted feature points as observation constraints, calculating the difference between the predicted position values ​​of the predicted feature points in the m-1th frame image and the m-1th frame image and the observed position values ​​of the observed feature points in the m-1th frame image and the m-1th frame image, and processing the calculated pose based on the difference to obtain the initial pose of the m-th frame image.

[0050] In one embodiment, calculating the difference between the predicted position value of the predicted feature point in the m-th frame image and the m-1-th frame image and the observed position value of the observed feature point in the m-th frame image and the m-1-th frame image, and processing the calculated pose based on the difference to obtain the initial pose of the m-th frame image includes: calculating the difference between the predicted position value of the predicted feature point in the m-th frame image and the m-1-th frame image and the observed position value of the observed feature point in the m-th frame image and the m-1-th frame image using a preset optimization function to obtain an error value; determining whether the error value is less than a preset threshold, and if so, using the calculated pose as the initial pose of the m-th frame image; if the error value is greater than the preset threshold, eliminating abnormal observation values ​​of the observed feature point, and iteratively optimizing the calculated pose of the m-th frame partial image using normal observation values, repeating the above steps until the obtained error value is less than the preset threshold, and using the calculated pose corresponding to the error value as the initial pose of the m-th frame partial image. The same process is repeated for other images, and the initial pose of each image is obtained by traversing each image.

[0051] In the embodiment provided in the present application, iterative optimization of a preset optimization function can make the predicted position calculated by the function increasingly close to the actual observed position, making the estimation more accurate, thereby obtaining an accurate posture, and using the posture to construct an accurate boundary, thereby improving the reliability of the constructed boundary.

[0052] S220 , selecting n key frame images from M frames according to the initial poses of each image, and using the n key frame images to correct the corresponding initial poses to obtain optimized poses, where n is an integer and 1<n≤M.

[0053] In one embodiment, selecting n key frame images from M frames of images based on the initial pose of each image includes: when a user moves a handheld acquisition device along the boundary of a customized lawn mowing robot working area, multiple frames of images are acquired at each acquisition point on the boundary. In order to make the images relevant and facilitate the construction of a high-precision map, the images acquired based on the acquisition points need to be filtered to obtain key frame images with overlapping areas.

[0054] Specifically, any frame image captured at a certain acquisition point on the boundary is used as a reference frame, and the initial pose of the frame image and the image obtained based on the adjacent acquisition point (the previous acquisition point or the next acquisition point) are calculated to obtain the disparity between the two adjacent frames of image captured by the acquisition device; the size relationship between the disparity and the preset value is compared, and if the disparity is less than the preset value, it is filtered as a key frame image, and M frame images are traversed to filter n key frames with overlapping areas.

[0055] In the embodiment provided in the present application, by screening key frames, the image used to obtain the boundary can have matching features and can also ensure sufficient variation values, thereby achieving high-precision boundary construction with fewer samples.

[0056] In one embodiment, using n frames of key frame images to correct the corresponding initial posture to obtain an optimized posture includes: taking any frame among the n frames of key frame images as the current key frame image, calculating the similarity between the current key frame image and the historical key frame images; judging whether the current key frame image and the historical key frame images form a loop based on the similarity, and using the relative transformation relationship between the images that construct the loop to correct the initial posture corresponding to the current key frame image to obtain the optimized posture.

[0057] In one embodiment, the relative transformation relationship between the images that construct the loop is used to correct the initial posture corresponding to the current key frame image to obtain an optimized posture, including: if the similarity is greater than a preset similarity threshold, it means that the current key frame image and the historical key frame image may have common view to form a loop; based on the coordinate information of each feature point of the two key frame images that may form a loop, the relative transformation relationship between the two key frame images is calculated. If the relative transformation relationship can be successfully calculated, it means that the two key frame images form a loop, then the initial posture corresponding to the current key frame image needs to be corrected based on the relative transformation relationship to further constrain the posture of the current key frame and obtain the optimized posture.

[0058] It should be noted that the successful calculation of the relative transformation relationship between two key frame images requires the following conditions to be met: the two key frame images that constitute the loop need to have enough matching feature points, the matching feature points have depth and the relative transformation relationship between the two frames can be calculated using the PNP solution algorithm.

[0059] In one embodiment, calculating the similarity between the current key frame image and the historical key frame images includes: performing feature extraction on the current key frame image to obtain feature points of the current key frame image and descriptors corresponding to the feature points; inputting the descriptors into a preset bag-of-words model to obtain a bag-of-words vector representing the current key frame image; calculating the similarity between the current key frame and each historical key frame image based on the bag-of-words vector of the current key frame and the bag-of-words vector of each historical key frame image in a bag-of-words model database; wherein the preset bag-of-words model database is composed of bag-of-words vectors of historical key frames.

[0060] It should be noted that if the descriptor of the key frame image has been obtained in step S210, then in this embodiment, the descriptor of the current key frame can be directly input into the preset bag-of-words model for processing to obtain the similarity between the current key frame and the historical key frames.

[0061] This embodiment performs loop optimization on the pose of each key frame image to obtain an optimized pose, which is beneficial to improving the accuracy of the boundary and reducing the amount of calculation.

[0062] S230: Construct a working boundary based on the optimized pose of each key frame image.

[0063] The optimized poses of each key frame image are linearly or nonlinearly fitted to obtain the boundary of the robot's working area. This embodiment can improve the accuracy of the working boundary by constructing the working boundary using the optimized poses after global optimization.

[0064] In summary, the method of constructing the working boundary in this application does not require the identification and processing of grass and non-grass in the image. It only needs to determine the posture of the image when it is captured based on the matching results of feature points in adjacent images. The posture connection method of multiple frames of images is consistent with the working boundary, which can achieve rapid setting of the working boundary at a lower computational cost.

[0065] After obtaining the working boundary, the present application also provides a method for constructing a global map, which includes: generating a collection trajectory based on the working boundary, controlling the robot to move on the collection trajectory to collect images within the working area, and obtaining a global map of the working area based on the images collected within the working area.

[0066] To construct a global map, images can be captured within the work boundary constructed in step S230 to complete the map of the entire work area. Specifically, after obtaining the work boundary in step S230, a capture trajectory can be generated based on the work boundary to match the work area, such as the "bow"-shaped capture trajectory shown in Figure 4. By controlling the robot to move along the capture trajectory to capture multiple frames of images within the work area, the multiple frames are stitched together to generate a map of the entire work area.

[0067] It should be understood that in addition to collecting the "bow"-shaped collection trajectory as shown in Figure 3, the present application can also use a "return"-shaped collection trajectory to collect multiple frames of images of the working area along the working boundary inward, and the present application is not limited to this. In addition, in the process of using images to obtain the working boundary of the robot, if the image used is an image within the working area and the image covers the entire area of ​​the working area, the optimized poses of each key frame image in the image can be used to stitch the images together to obtain a global map of the working area.

[0068] In the embodiment provided in the present application, a map is constructed according to an established working boundary. When the working boundary has been determined, a global map can be constructed accurately and automatically without the need for additional effort to improve the map image along the map.

[0069] After obtaining a global map of the work area, the robot can reposition itself within the work area based on the global map and perform operations based on the repositioning results. Specifically, since the robot is not directly positioned when placed in the work area, it is necessary to reposition the robot's current position. As one possible implementation method, based on the obtained global map, while the robot is performing its work tasks, an image of the robot's current position is captured by a capture device located in the robot's top / front view. This image is then matched with the global map to complete repositioning, obtaining the robot's position within the work area and its boundary information. This information can then be used to control the robot's operations.

[0070] In one embodiment, matching the current image with the global map through the bag-of-words vector to complete repositioning includes: extracting feature points of the current image and obtaining descriptors corresponding to the feature points; inputting the descriptors of the feature points of the current image into a preset bag-of-words model to obtain the bag-of-words vector of the current image; calculating the similarity between the bag-of-words vector of the current image and the bag-of-words vectors of each frame image in the global map; and obtaining the position of the robot in the working area based on the similarity, thereby achieving repositioning.

[0071] In this embodiment, when the robot is working, the current image of the current position is collected by the collection device set on the robot. By comparing the current map with the stored global map, the position of the robot when working can be determined, and the robot's actions can be controlled more accurately and quickly according to the map information, thereby more efficiently controlling the robot to complete the task.

[0072] Figure 4 is a schematic diagram of a robot according to an embodiment of the present application. As shown in Figure 4 , robot 110 includes a data acquisition device 102 and a body 105. When setting a work boundary 120, user 103 uses data acquisition device 104 or a mobile terminal to capture images along the boundary. After capturing the images, an offline work boundary 120 is constructed based on the images using the method described in the above embodiment. Furthermore, the global map can be improved based on work boundary 120.

[0073] The robot 110 may also be configured with a processor 106. When the robot is working, the processor 106 is repositioned according to the working boundary 120 or according to the global map to obtain the position and boundary information of the robot in the global map.

[0074] The robot may also be configured with a controller, and when performing an operation, the controller may control the main body 105 to perform the operation according to the acquired position information.

[0075] It should be understood that the method for the robot to obtain the working boundary and global map and achieve relocalization provided in this application can be obtained according to any of the above embodiments. For details not described in this embodiment, please refer to the relevant description of the above method embodiments, which will not be repeated here.

[0076] Fig. 5 shows a schematic diagram of a working boundary generation device provided by the present application. The working boundary generation device shown in Fig. 5 includes a processor 510, and the processor 510 is used to implement any of the methods in the above embodiments.

[0077] The processor 510 mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), neural network chips, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. It should be understood that when the processor is a neural network chip, the device may not include a memory.

[0078] In one embodiment, the apparatus further includes a memory 520 , which can store the images collected in the above embodiments and the computer program products required to execute the methods in the above embodiments.

[0079] The memory 520 and the processor 510 can be coupled via a bus. The memory 520 is used to store computer program instructions or data. The processor 510 reads the computer instructions stored in the memory 520 or reads the data stored in the memory 520 to execute the method in the above embodiment.

[0080] The processor 510 and the memory 520 may be placed separately or integrated. For example, the processor 510 may be a processing device in a user's mobile terminal with independent data processing capabilities, a processing device in a collection device, or other processing devices.

[0081] It should also be understood that the memory 520 mentioned in the embodiments of the present application may be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM). For example, RAM can be used as an external cache. By way of example and not limitation, RAM includes the following forms: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0082] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0083] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the various steps in the above method embodiment can be implemented.

[0084] It should be understood that the above-mentioned specific embodiments of the present application are exemplary, and those skilled in the art can implement them individually or combine the methods between the embodiments to implement them.

[0085] The explanation of the relevant contents and beneficial effects of any of the above-mentioned devices can be referred to the corresponding method embodiments provided above, which will not be repeated here.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. For example, the computer can be a personal computer, a server, or a network device, etc. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)). For example, the aforementioned available medium includes, but is not limited to, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0088] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for constructing a robot's working boundary, characterized in that: The method comprises: Obtain M frames of images collected along the expected boundary, and determine the initial pose of each image in the M frames of images, where M is a positive integer greater than 1; According to the initial poses of the images, n key frame images are selected from the M frames of images, and the corresponding initial poses are corrected using the n key frame images to obtain an optimized pose, wherein n is an integer and 1<n≤M; Based on the optimized posture of each key frame image, the working boundary of the robot is constructed.

2. The method according to claim 1, characterized in that: The expected boundary is a boundary of a user-defined robot working area, wherein acquiring M frames of images collected along the expected boundary includes: The handheld acquisition device is moved along the expected boundary, and each time it moves, multiple frames of images are continuously acquired at the current acquisition point of the expected boundary to obtain the M frames of images.

3. The method according to claim 1, characterized in that: The determining of the initial position and posture of each image in the M frames of images includes: Obtain motion parameters corresponding to each image of the M frames of images, and determine a predicted position and posture of the m-th frame of image according to the motion parameters of the m-1th frame of image and the m-1th frame of image in the M frames of image, where m is an integer and 1<m≤M; Acquire matching features between the m-th frame image and the m-1-th frame image, and use the matching features as observation constraints to update the predicted pose to obtain an initial pose of the m-th frame image; All images in the M frames are traversed to obtain the initial poses of the images.

4. The method according to claim 3, characterized in that The obtaining matching features between the m-th frame image and the m-1-th frame image, and using the matching features as observation constraints to update the predicted pose to obtain an initial pose of the m-th frame image, includes: Extracting features from the m-th frame image to obtain key feature points, and calculating descriptors of the key feature points in the m-th frame image; Based on matching the descriptor of the key feature point in the m-th frame image with the descriptor of the key feature point in the m-1-th frame image, a matching feature between the m-th frame image and the m-1-th frame image is obtained; wherein the descriptor can be expressed as a feature vector of the key feature point; The predicted pose is updated using the position information of the matching feature as an observation constraint to obtain an initial pose of the m-th frame image.

5. The method according to claim 3, characterized in that: The obtaining matching features between the m-th frame image and the m-1-th frame image, and using the matching features as observation constraints to update the predicted pose to obtain an initial pose of the m-th frame image, includes: Taking the key feature point obtained in the m-1th frame image as a reference point, tracking the position of the reference point in the mth frame image, and the same reference point between different frames is the matching feature; The predicted pose is updated based on the position of the reference point between the m-1th frame image and the mth frame image as an observation constraint to obtain an initial pose of the mth frame image.

6. The method according to claim 1, characterized in that The determining of the initial position and posture of each image in the M frames of images includes: Acquire matching features between an m-th frame image and an m-1-th frame image in the M frames of images, and calculate a calculated pose of the m-th frame image based on the matching features, where m is an integer and 1<m≤M; Projecting the three-dimensional feature points corresponding to the acquisition object in the world coordinate system onto the imaging plane of the acquisition device to obtain predicted position values ​​of the three-dimensional points corresponding to the predicted feature points on the m-1th frame image and the mth frame image; Obtaining observed feature points of the m-th image frame and the m-1-th image frame that match the predicted feature points, and calculating the difference between the predicted position value of the predicted feature point in the m-th image frame and the m-1-th image frame and the observed position value of the observed feature point in the m-th image frame and the m-1-th image frame; The calculated posture is processed based on the difference to obtain the initial posture of the m-th frame image, and all images in the M frames of images are traversed to obtain the initial posture of each image.

7. The method according to claim 1, characterized in that The step of selecting n key frame images from M frames of images according to the initial positions of the images includes: Taking any frame of image collected at a certain collection point on the expected boundary as a reference frame, calculating the initial pose of the reference frame and the image obtained based on adjacent collection points to obtain the disparity between two frames of image collected by the collection device; The magnitude relationship between the disparity and a preset value is compared, and if the disparity is smaller than the preset value, the image is selected as a key frame, and the M frames of images are traversed to select n key frames with overlapping areas.

8. The method according to claim 1, characterized in that The method of using n frames of key frame images to correct the corresponding initial posture to obtain the optimized posture includes: Taking any frame of the n key frame images as the current key frame image, calculating the similarity between the current key frame image and the historical key frame images; It is determined whether the current key frame image and the historical key frame image form a loop according to the similarity, and the initial posture corresponding to the current key frame image is corrected by using the relative transformation relationship between the images that construct the loop to obtain the optimized posture.

9. The method according to claim 8, characterized in that The calculating the similarity between the current key frame image and the historical key frame image includes: Performing feature extraction on the current key frame image to obtain feature points of the current key frame image and descriptors corresponding to the feature points; Inputting the descriptor into a preset bag-of-words model to obtain a bag-of-words vector representing the current key frame image; The similarity between the current key frame and each historical key frame image is calculated based on the bag-of-words vector of the current key frame and the bag-of-words vector of each historical key frame image in a bag-of-words model database; wherein the preset bag-of-words model database is composed of bag-of-words vectors of historical key frames.

10. The method according to any one of claims 1 to 9, characterized in that: In the process of obtaining the working boundary by using an image, if the image used is an image within the working area and the image covers the entire area of ​​the working area, the key frame images in the M frame images can also be spliced ​​using the optimized postures of the key frame images to obtain a global map of the working area; or Generate a collection track according to the working boundary, control the robot to move on the collection track to collect images in the working area, and acquire a global map of the working area based on the images collected in the working area.

11. The method according to claim 10, characterized in that Also includes: The image collected by the robot during the execution of its work is matched with the global map to obtain the position of the robot in the work area to achieve repositioning.

12. A robot, characterized in that: The robot comprises: A collection device, used for collecting M frames of images of expected boundaries; A processor, configured to process the M frame images according to the method described in any one of claims 1 to 9 to obtain a working boundary of a working area of ​​the robot, and / or process the M frame images according to the method described in claim 10 to obtain a global map of the working area of ​​the robot, and / or realize repositioning of the robot according to the method described in claim 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 11 is implemented.

Images