Method for guiding mobile robot back to station, apparatus, base station and household appliance

By installing image acquisition equipment at the base station location, acquiring environmental images and planning paths, the obstacles of mobile robots returning to the base station in complex environments are solved, and an efficient and accurate return process is achieved.

WO2024230316A1PCT designated stage expired Publication Date: 2024-11-14MIDEA ROBOZONE TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/081719
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-03-14
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Mobile robots are difficult to efficiently avoid obstacles and return to the base station in complex base station working environments. Existing infrared or laser sensors cannot detect all obstacles, resulting in inaccurate return paths or collisions.

Method used

By installing image acquisition equipment at locations higher than the base station, we can obtain higher and wider viewing angle environment images, identify obstacle locations and plan back to the station paths, and use the internal and external parameter data of the image acquisition equipment to convert coordinates and path planning, so as to achieve efficient mobile robots Return to the site.

Benefits of technology

Effectively avoid obstacle collisions, ensure that mobile robots accurately return to the base station in complex environments, and reduce the risks and complexity during the return process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024081719_14112024_PF_FP_ABST
    Figure CN2024081719_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A method for guiding a mobile robot back to a station, an apparatus, a base station, and a household appliance, relating to the technical field of mobile robots. The method comprises: first using an image acquisition device at a position higher than the height of a base station to photograph the working environment of the base station so as to obtain an environment image; and then, at least in part on the basis of the environment image, identifying the mobile robot and / or guiding the mobile robot back to the base station.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device, base station and household appliance for guiding mobile robot back to station

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202310523927.2, filed on May 10, 2023, entitled “Method, device, base station and household appliance for guiding a mobile robot back to the station,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of mobile robots, and in particular to a method, device, base station, and household appliance for guiding a mobile robot back to a station. Background Art

[0004] There may be some shoes, toys, etc. in front of the base station that are obstacles for the mobile robot. As can be seen, the environment in front of the mobile robot is relatively complex, so how to efficiently guide the mobile robot back to the base station is a problem that needs to be solved.

[0005] Public content

[0006] The embodiments of the present disclosure aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of the present disclosure provide a method, apparatus, base station, and home appliance for guiding a mobile robot back to a station.

[0007] The present disclosure provides a method for guiding a mobile robot to return to a station, the method comprising:

[0008] Acquire an environmental image; wherein the environmental image is obtained by photographing the base station working environment through an image acquisition device; the height of the image acquisition device is greater than the height of the top of the base station;

[0009] Based at least in part on the image of the environment, a mobile robot is identified and / or directed to return to the base station.

[0010] In one embodiment, guiding the mobile robot back to the base station includes:

[0011] Determining first position data of an obstacle in the base station working environment based on the environment image;

[0012] Performing path planning based on the first position data and the second position data of the mobile robot to obtain a first return path for the mobile robot to return to the base station;

[0013] The mobile robot is guided to return to the base station according to the first return path.

[0014] In one embodiment, guiding the mobile robot back to the base station includes:

[0015] determining second position data of the mobile robot based on the environment image;

[0016] performing path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station;

[0017] The mobile robot is guided to return to the base station according to the first return path.

[0018] In one embodiment, the method for determining the first position data includes:

[0019] receiving the environment image sent by the image acquisition device;

[0020] Identify the environment image to obtain the image coordinate system coordinates of the obstacle in the environment image;

[0021] Coordinate conversion is performed according to the image coordinate system coordinates of the obstacle and the internal and external parameter data of the image acquisition device to obtain the first position data of the obstacle in the operation map of the mobile robot.

[0022] In one embodiment, directing the mobile robot back to the base station based at least in part on the image of the environment includes:

[0023] Determine second position data of the mobile robot in the operation map based on the environmental image and the internal and external parameter data of the image acquisition device; determine the return reference point of the base station based on the external parameter data of the image acquisition device and the relative position data between the image acquisition device and the return reference point; guide the mobile robot back to the base station using the second position data and the return reference point; or

[0024] Based on the environmental image and the internal and external parameter data of the image acquisition device, the second position data of the mobile robot in the operation map is determined; and path planning is performed according to the second position data and the position data of the base station to guide the mobile robot back to the base station.

[0025] In one embodiment, the method comprises:

[0026] If the position of the obstacle changes, obtaining third position data of the obstacle after the change;

[0027] Path planning is performed again based on the third position data and the real-time position data of the mobile robot to obtain a second return path for the mobile robot to return to the base station.

[0028] In one embodiment, if the position of the obstacle changes, obtaining third position data of the obstacle after the change includes:

[0029] Receiving a real-time image of the environment after the position of the obstacle changes, which is sent in real time by the image acquisition device;

[0030] The third position data is obtained by performing calculation based on the real-time image of the environment and the internal and external parameter data of the image acquisition device.

[0031] In one embodiment, the return-to-station reference line of the base station is fixed relative to the image acquisition device; the position of the return-to-station reference line is determined by external parameters of the image acquisition device and relative position data between the return-to-station reference line and the image acquisition device.

[0032] In one embodiment, performing path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station includes:

[0033] Path planning is performed at least in part based on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path.

[0034] In one embodiment, before performing path planning based at least in part on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path, the method further includes:

[0035] Determining first position data of an obstacle in the base station working environment based on the environment image;

[0036] The performing path planning based at least in part on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path includes:

[0037] Path planning is performed based on the first position data of the obstacle, the second position data of the mobile robot, and the position of the return-to-station reference line to obtain the first return-to-station path.

[0038] In one embodiment, the base station is arranged at the bottom of the home appliance or as a part of the home appliance, and the image acquisition device is arranged on the home appliance; or

[0039] The base station is arranged at the bottom of the furniture or as a part of the furniture, and the image acquisition device is arranged on the furniture.

[0040] In one embodiment, the household appliance includes a clothes processing appliance.

[0041] An embodiment of the present disclosure provides a base station, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method described in any one of the above embodiments when running the computer program.

[0042] An embodiment of the present disclosure provides a home appliance, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method described in any one of the above embodiments when running the computer program.

[0043] In one embodiment, the household appliance includes a clothes processing appliance.

[0044] The present disclosure provides a device for guiding a mobile robot to return to a station, the device comprising:

[0045] An environmental image acquisition module is used to acquire an environmental image; wherein the environmental image is obtained by photographing the base station working environment through an image acquisition device; the height of the image acquisition device is greater than the height of the top of the base station;

[0046] An identification and guidance module is configured to identify the mobile robot and / or guide the mobile robot back to the base station based at least in part on the environment image.

[0047] In the disclosed embodiments, an image capturing device positioned at a height greater than the base station captures the base station's operating environment to obtain an environmental image. The mobile robot is then identified and / or guided back to the base station based, at least in part, on the environmental image. By installing the image capturing device at a height greater than the base station, an image of the environment in front of the base station is captured from a higher and wider perspective. The mobile robot can be identified and / or guided back to the base station based, at least in part, on the environmental image, eliminating the need for the mobile robot to search for the environment and efficiently guiding the mobile robot back to the base station. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0049] FIG1 is a schematic flow chart of a method for guiding a mobile robot to return to a station according to an embodiment of the present disclosure;

[0050] FIG2 is a schematic diagram of a process for guiding a mobile robot to return to a base station according to an embodiment of the present disclosure;

[0051] FIG3 is a schematic diagram of a process for guiding a mobile robot to return to a base station according to an embodiment of the present disclosure;

[0052] FIG4 is a schematic diagram of a process for determining first position data according to an embodiment of the present disclosure;

[0053] FIG5 is a schematic flow chart of a method for guiding a mobile robot to return to a station according to an embodiment of the present disclosure;

[0054] FIG6 is a schematic diagram of a process for determining third position data according to an embodiment of the present disclosure;

[0055] FIG7 is a schematic diagram of a base station return reference line provided by an embodiment of the present disclosure;

[0056] FIG8 is a schematic diagram of a process for determining a first return path to a station according to an embodiment of the present disclosure;

[0057] FIG9 is a schematic diagram of a base station and an image acquisition device provided in an embodiment of the present disclosure;

[0058] FIG10 is a schematic flow chart of a method for guiding a mobile robot to return to a station according to an embodiment of the present disclosure;

[0059] FIG11 is a schematic diagram of an apparatus for guiding a mobile robot to return to a station according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0061] The present disclosure provides a method for guiding a mobile robot to return to a station. Referring to FIG1 , the method for guiding a mobile robot to return to a station may include the following steps:

[0062] S110: Acquire an environment image.

[0063] S120 : Identify the mobile robot and / or guide the mobile robot back to the base station based at least in part on the environment image.

[0064] The environmental image is captured by an image acquisition device capturing the base station's operating environment. The image acquisition device is located at a height greater than the top of the base station. The base station's operating environment may include the layout of objects or obstacles surrounding the base station.

[0065] In some cases, an image acquisition device is installed at a position higher than the base station, and the image acquisition device interacts with the mobile robot to guide the mobile robot back to the base station with a higher and wider perspective.

[0066] Specifically, an image capture device is installed at a location higher than the base station. The image capture device then captures images of the base station's operating environment to generate an environmental image. Based on at least a portion of the captured environmental image, the mobile robot can identify its own location or, based on at least a portion of the environmental image, guide the mobile robot back to the base station.

[0067] In the above-mentioned method for guiding a mobile robot back to a base station, an image capture device located at a height greater than the base station captures the base station's operating environment to obtain an environmental image. The mobile robot is then identified and / or guided back to the base station based, at least in part, on the environmental image. By installing the image capture device at a height greater than the base station, an image of the environment in front of the base station is captured from a higher and wider perspective. This, at least in part, allows the mobile robot to be identified and / or guided back to the base station based on the environmental image, eliminating the need for the mobile robot to search for the environment and achieving efficient guidance of the mobile robot back to the base station.

[0068] With the continuous development of artificial intelligence (AI) technology, a series of AI-based products are gradually entering people's lives, providing numerous conveniences. For example, cleaning robots can automatically identify obstacles and avoid them to perform cleaning tasks, reducing cleaning workloads for users. Cleaning robots are typically equipped with a base station to assist them in their work. For example, the base station can provide charging services for the cleaning robot, mop cleaning services, and garbage collection services. In front of the base station, there may be items such as shoes and toys that are obstacles for mobile robots. Therefore, the operating environment of the mobile robot base station is relatively complex. Therefore, guiding the mobile robot to effectively avoid obstacles and efficiently return to the base station is a problem that needs to be solved.

[0069] In related technologies, mobile robots usually use infrared, laser and other sensors to locate the positions of base stations and obstacles.

[0070] However, due to the limitations of infrared or laser methods, there is a possibility that not all obstacles in the base station working environment can be detected, and obstacles that are not detected may affect the mobile robot's return to the base station.

[0071] Based on this, an embodiment of the present disclosure provides a method for guiding a mobile robot to return to the station. Specifically, an image acquisition device at a height greater than the base station is used to capture the base station working environment to obtain an environmental image, and then, based at least in part on the environmental image, the mobile robot is identified and / or the mobile robot is guided to return to the base station. By installing an image acquisition device at a position higher than the base station, the mobile robot can be guided back to the base station with a higher and wider viewing angle, and the environmental image in front of the base station can be captured. Based at least in part on the environmental image, the mobile robot can be identified and / or prevented from colliding with obstacles during the return process, and the mobile robot can be effectively guided to the position of the return reference line in a complex base station working environment, and the mobile robot can be guided back to the base station.

[0072] The method for guiding a mobile robot back to a base station provided in the embodiments of the present disclosure can be applied to a mobile robot corresponding to a base station integrated below a washing machine. The image acquisition device can be mounted on the washing machine at a height higher than the base station (e.g., the top of the washing machine). Obstacles can be items such as toys, shoes, and stools that the mobile robot needs to avoid during movement. The mobile robot can be a sweeping robot, a mopping robot, or other self-moving device equipped with a base station. Exemplarily, the image acquisition device captures an image of the base station's operating environment and sends it to a cloud server. The cloud server can identify the received image and obtain the image coordinates of the obstacle in the image. The cloud server then sends the obtained image coordinates of the obstacle to the mobile robot. The mobile robot can then plan a path back to the base station based on the received image coordinates of the obstacle and the mobile robot's positioning data. Exemplarily, the image acquisition device captures an image of the base station's operating environment and sends it to the mobile robot. The mobile robot can identify the received image and obtain the image coordinates of the obstacle in the image. The mobile robot can then plan a path back to the base station based on the received image coordinates of the obstacle and the mobile robot's positioning data.

[0073] A cloud server can be an electronic device with certain computing and processing capabilities. It may include a network communication module, a processor, and memory. A cloud server can be a distributed server, a system with multiple processors, memories, network communication modules, and other components operating in coordination. Alternatively, a cloud server can be a server cluster consisting of multiple servers. Alternatively, with the advancement of science and technology, a cloud server can also be a new technological means capable of implementing the functions of the disclosed embodiments. For example, it can be a new form of "server" based on quantum computing.

[0074] In some embodiments, referring to FIG. 2 , guiding the mobile robot back to the base station may include the following steps:

[0075] S210: Determine first position data of an obstacle in the base station working environment based on the environment image.

[0076] Among them, the first position data is calculated based on the environmental image and the internal and external parameter data of the image acquisition device. The environmental image is obtained by shooting the base station working environment with the image acquisition device. The internal and external parameter data may include the internal parameter data and the external parameter data of the image acquisition device. The height of the image acquisition device is greater than the height of the top of the base station. The internal parameter data of the image acquisition device may be parameters related to the characteristics of the image acquisition device itself. For example, the focal length and pixel size of the camera. The external parameter data of the image acquisition device may be the position of the image acquisition device in the world coordinate system, which can be understood as converting the position information of the image acquisition device from the image acquisition device coordinate system to the world coordinate system.

[0077] In some cases, image acquisition equipment is installed at a position higher than the base station. The image acquisition equipment can be used to guide the mobile robot with a higher and wider perspective, interact with the mobile robot, achieve efficient return to the station, and more accurately and comprehensively determine the location of obstacles in the base station working environment.

[0078] Specifically, the position data of the obstacle in the working environment of the base station, ie, the first position data, can be obtained by calculating the environment image and the internal parameter data and the external parameter data of the image acquisition device.

[0079] In some embodiments, the environmental image can be uploaded to a cloud server, where the location data of obstacles in the base station working environment, i.e., first location data, can be obtained by calculation based on the environmental image and the internal and external parameter data of the image acquisition device.

[0080] In some embodiments, the environmental image can be sent to a mobile robot, in which the position data of obstacles in the base station working environment, i.e., first position data, can be obtained by calculation based on the environmental image and the internal and external parameter data of the image acquisition device.

[0081] S220: Perform path planning based on the first position data and the second position data of the mobile robot to obtain a first return path for the mobile robot to return to the base station.

[0082] S230: Guide the mobile robot back to the base station according to the first return path.

[0083] The first return path may be a path for the mobile robot to return to the base station, and this path can avoid collisions with obstacles. The second position data of the mobile robot may be position data calculated based on an image of the environment and intrinsic and extrinsic parameter data of an image acquisition device. The second position data of the mobile robot may also be position data obtained by the mobile robot through positioning and obtaining its own position.

[0084] In some cases, there are generally requirements for the placement of the base station. Generally, a distance of 30 to 50 centimeters is required on the left and right sides of the base station, and a space of 1 to 1.5 meters is required in front of the base station. However, in reality, not all base station placements can perfectly meet the above requirements. Therefore, it is necessary to plan a path back to the base station. This path allows the mobile robot to avoid obstacles during the return process and guide the mobile robot back to the base station. The method of installing image acquisition equipment and guiding the mobile robot back to the base station through the first return path can replace or supplement the step of searching for the return reference line in front of the base station by infrared or other means.

[0085] Specifically, the mobile robot can combine the obstacle's location and the mobile robot's location on a single map based on the obstacle's first location data. Based on the mobile robot's own location and the obstacle's location on the map, the mobile robot can plan a path back to the base station, known as a first return path. The planned first return path can then be used to guide the mobile robot back to the base station.

[0086] In some embodiments, the cloud server transmits the obtained first obstacle location data to the mobile robot. Based on the first obstacle location data, the mobile robot can combine the obstacle's location and the mobile robot's location on a single operation map. Based on the mobile robot's own location and the obstacle's location on the operation map, the mobile robot can plan a path back to the base station, i.e., the first return path.

[0087] In the above-mentioned method for guiding a mobile robot back to the base station, first position data of obstacles in the base station's operating environment are calculated based on an image of the environment and internal and external parameter data of an image acquisition device located at a height greater than the base station. Path planning is then performed based on the first position data of the obstacles and the second position data of the mobile robot to obtain a first return path for the mobile robot to return to the base station, and the mobile robot is guided back to the base station according to the first return path. By installing an image acquisition device at a height higher than the base station, the mobile robot is guided and images of the environment in front of the base station are captured from a higher and wider perspective. The planned first return path allows the mobile robot to avoid collisions with obstacles during its return to the base station, effectively guides the mobile robot to the position of a return reference line in the complex base station's operating environment, and guides the mobile robot back to the base station.

[0088] In some embodiments, referring to FIG3 , guiding the mobile robot back to the base station may include the following steps:

[0089] S310: Determine second position data of the mobile robot based on the environment image.

[0090] The second position data may be the position data of the mobile robot in the operation map, or the second position data may be the position data calculated based on the environmental image and the internal and external parameter data of the image acquisition device.

[0091] In some cases, an image acquisition device installed at a higher position than the base station can be used to determine the mobile robot's position from a higher and wider perspective, thereby enabling interaction with the mobile robot and efficiently guiding it back to the base station. Specifically, the mobile robot's position data within the base station's operating environment, i.e., second position data, can be obtained by calculating the environmental image and the intrinsic and extrinsic parameter data of the image acquisition device.

[0092] In some embodiments, the environmental image can be uploaded to a cloud server. In the cloud server, calculations are performed based on the environmental image and the internal and external parameter data of the image acquisition device to obtain the position data of the mobile robot in the base station working environment, i.e., the second position data.

[0093] In some other embodiments, the environmental image can be sent to a mobile robot. In the mobile robot, calculations are performed based on the environmental image and the internal and external parameter data of the image acquisition device to obtain the position data of the mobile robot in the base station working environment, i.e., the second position data.

[0094] S320: Perform path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station.

[0095] S330: Guide the mobile robot back to the base station according to the first return path.

[0096] Specifically, if the mobile robot's position is known, a path back to the base station can be planned based on the second position data, resulting in a first return path for the mobile robot to return to the base station. In some embodiments, the mobile robot can combine the location of obstacles and the location of the mobile robot on the same map based on the second position data. Based on its own position and the location of obstacles on the map, the mobile robot can plan a path back to the base station, namely the first return path. The mobile robot can then be guided back to the base station based on the planned first return path.

[0097] In some embodiments, the cloud server transmits the obtained second position data of the mobile robot to the mobile robot. The mobile robot, based on the second position data, can combine the location of the obstacle with the location of the mobile robot on a single operation map. The mobile robot can then plan a path back to the base station, i.e., a first return path, based on its own location and the location of the obstacle on the operation map.

[0098] In the above-mentioned method for guiding the mobile robot back to the station, calculations are performed based on the environmental image and the internal and external parameter data of the image acquisition device at a height greater than the base station, and second position data for determining the mobile robot in the base station working environment can be obtained. Then, path planning is performed at least partially based on the second position data, and a first return path for the mobile robot to return to the base station can be obtained. The mobile robot is then guided back to the base station according to the first return path. By installing the image acquisition device at a position higher than the base station, the mobile robot is guided with a higher and wider perspective, and the planned first return path allows the mobile robot to return to the station efficiently and conveniently. Furthermore, the probability of collision with obstacles is reduced, and the mobile robot can be effectively guided to reach the position of the return reference line in a complex base station working environment, as well as to guide the mobile robot back to the base station.

[0099] In some embodiments, referring to FIG. 4 , a method for determining the first position data may include the following steps:

[0100] S410: Receive an environment image sent by an image acquisition device.

[0101] S420: Detect obstacles based on the environment image to obtain the image coordinate system coordinates of the obstacles in the environment image.

[0102] An image is composed of pixels, and the image coordinate system coordinates are the positions of pixels in the image. The image acquisition device can be any device with image acquisition capabilities, such as a monocular camera, a binocular camera, or a webcam.

[0103] Specifically, the image acquisition device uploads the captured environment image to a designated receiving device. The designated receiving device receives the environment image sent by the image acquisition device and then performs obstacle detection on the environment image within the designated device. By performing obstacle detection on the environment image, obstacles, mobile robots, and background (floor, wall, etc.) can be determined. If an obstacle is present, the obstacle coordinates in the image coordinate system of the environment image can be obtained by performing obstacle detection on the environment image.

[0104] In some embodiments, an obstacle detection model can be trained and deployed on a cloud server. An environmental image captured by an image acquisition device is uploaded to the cloud server, which then inputs the environmental image into the trained obstacle detection model. The cloud server then determines whether the feature information of the obstacle extracted by the obstacle detection model matches the trained obstacle detection model. If the extracted feature information does not match the trained obstacle detection model, then no obstacle is present in the captured environmental image. If the extracted feature information does match the trained obstacle detection model, then an obstacle is present in the captured environmental image. The obstacle detection model can then identify the obstacle category and detect the obstacle's position bounding box in the environmental image and the coordinates of the position bounding box. The coordinates of the position bounding box are the image coordinate system coordinates of the obstacle in the environmental image. Furthermore, the environmental image contains a number of pixels, and the detected coordinates of the position bounding box can be considered as the pixel coordinates of a vertex of the position bounding box. The image coordinate system coordinates of the obstacle can be represented as a position bounding box, with a certain number of pixels distributed within the position bounding box. It should be noted that the position border can be used to mark the area where the obstacle is located in the environment image, such as using a rectangular frame to frame the obstacle in the environment image. The position border can be the bounding box determined when the obstacle is detected by the obstacle detection model. Based on the position border, the image coordinate system coordinates of the obstacle corresponding to the position border can be determined. The obstacle detection model can use a deep learning network. The obstacle detection model can also use an SSD (Single Shot MultiBox Detector) model. Deploying the obstacle detection model on a cloud server can reduce the consumption of mobile robot resources.

[0105] In other implementations, an obstacle detection model can be deployed on a mobile robot, and an image acquisition device can upload captured environmental images to the mobile robot. The mobile robot then inputs the environmental images into the obstacle detection model, which then detects obstacles in the environmental images and obtains the obstacle's positional bounding box within the environmental images and the obstacle's coordinates in the image coordinate system.

[0106] S430 , performing coordinate conversion based on the image coordinate system coordinates of the obstacle and the internal and external parameter data of the image acquisition device to obtain first position data of the obstacle in the operation map of the mobile robot.

[0107] The operation map of the mobile robot may be marked with the position of the mobile robot, the position of obstacles, and the relative position relationship between the position of the mobile robot and the position of obstacles.

[0108] In some cases, displaying the mobile robot's position and obstacle positions on the same map can facilitate the mobile robot's path planning to return to the base station.

[0109] Specifically, according to the internal and external parameter data of the image acquisition device, the image coordinate system coordinates of the obstacle can be converted into the operation map of the mobile robot, and the position data of the obstacle in the operation map of the mobile robot is the first position data.

[0110] In some embodiments, if the image acquisition device is a binocular camera, the depth information of the obstacle can be directly calculated. Therefore, the image coordinate system coordinates of the obstacle can be converted into the working map of the mobile robot more conveniently through coordinate conversion.

[0111] In other embodiments, if the image acquisition device is a monocular camera, the coordinate conversion requires a certain algorithm to complete. The coordinate conversion can be performed through PnP (Perspective-n-Point), which is a method for 3D-2D mapping. The basic formula is ωp = KP^C = K(R_CW×P^W+t_CW^C). Through this formula, the image coordinate system coordinates of the obstacle can be converted to the world map. Based on the world map, the operation map of the mobile robot can be obtained. The position of the obstacle in the operation map of the mobile robot is the first position data. Among them, P is the coordinate of the obstacle in the image coordinate system, P^C is the coordinate of the obstacle in the camera coordinate system, P^W can be the external parameter data of the image acquisition device, ω is the depth of the obstacle, K is the internal parameter data of the image acquisition device, R_CW and t_CW^C are the pose conversion from the world coordinate system to the camera coordinate system.

[0112] In the above-described method for guiding a mobile robot back to a base station, an image of an environment is received from an image acquisition device. Obstacle detection is performed on the environment image to obtain the image coordinate system coordinates of the obstacle in the environment image. A coordinate conversion is performed based on the image coordinate system coordinates of the obstacle and the internal and external parameters of the image acquisition device to obtain first position data of the obstacle in the mobile robot's operation map. By obtaining the first position data of the obstacle in the mobile robot's operation map, a path back to the base station can be more accurately planned.

[0113] In some embodiments, directing the mobile robot back to the base station based at least in part on the image of the environment may include:

[0114] Determine second position data of the mobile robot in the operation map based on the environmental image and the internal and external parameter data of the image acquisition device; determine the return reference point of the base station based on the external parameter data of the image acquisition device and the relative position data of the image acquisition device and the return reference point; guide the mobile robot back to the base station using the second position data and the return reference point; or

[0115] Based on the environmental image and the internal and external parameter data of the image acquisition device, the second position data of the mobile robot in the operation map is determined; the path is planned according to the second position data and the position data of the base station to guide the mobile robot back to the base station.

[0116] The return reference point may be a designated location that the mobile robot needs to reach when returning to the base station. After the mobile robot reaches the return reference point, it can return to the base station more accurately, thereby reducing collisions between the mobile robot and the base station.

[0117] In some embodiments, the second position data of the mobile robot within the operation map can be calculated based on the environmental image and the internal and external parameter data of the image acquisition device. The return reference point of the base station can be obtained by calculating the relative position data between the image acquisition device and the return reference point and the external parameter data of the image acquisition device. Path planning based on the second position data and the return reference point can determine the path for the mobile robot to reach the return reference point and return to the base station. The mobile robot can then be guided back to the base station based on the determined return path.

[0118] In other embodiments, second position data of the mobile robot within the operation map can be calculated based on the environmental image and internal and external parameter data of the image acquisition device. Path planning can be performed based on the second position data and the base station's position data to determine a path for the mobile robot to return to the base station. The mobile robot can then be guided back to the base station based on the determined return path.

[0119] In the above method for guiding the mobile robot to return to the base station, the path for the mobile robot to return to the base station can be flexibly determined in the above manner, and the return path that best meets the user's requirements can be selected.

[0120] In some embodiments, referring to FIG5 , the method for guiding the mobile robot back to the station may include the following steps:

[0121] S510: If the position of the obstacle changes, obtain third position data of the obstacle after the change.

[0122] The third position data may be position data after the obstacle position changes, and the third position data is calculated based on the real-time image of the environment and the internal and external parameter data of the image acquisition device.

[0123] In some cases, not all obstacles in a mobile robot's working environment are static; there may be movable obstacles. For example, obstacles may be pets at home, other mobile robots other than the robot itself, or movable toys.

[0124] Specifically, an image acquisition device can capture images of the base station's operating environment in real time, producing a series of multiple frames of real-time environmental images, with a very short interval between adjacent frames. Because the movement of an obstacle causes the obstacle's position to change, a calculation is performed based on each frame of the real-time environmental image and the intrinsic and extrinsic reference data of the image acquisition device to obtain the position data for the obstacle after the change in position, i.e., the third position data.

[0125] In some embodiments, each frame of the real-time environmental image can be uploaded to a cloud server. In the cloud server, calculations can be performed based on each frame of the real-time environmental image and the internal and external parameter data of the image acquisition device to obtain the position data of the obstacle in the real-time environmental image after the position changes, that is, the third position data.

[0126] In some other embodiments, each frame of the real-time image of the environment can be uploaded to a mobile robot. In the mobile robot, calculations can be performed based on each frame of the real-time image of the environment and the internal and external parameter data of the image acquisition device to obtain the position data of the obstacle in the real-time image of the environment after the position changes, that is, the third position data.

[0127] S520: Re-plan the path based on the third position data and the real-time position data of the mobile robot to obtain a second return path for the mobile robot to return to the base station.

[0128] The second return path is the path that the mobile robot replans to return to the base station after the position of the obstacle changes, and this path can avoid collision with the obstacle.

[0129] Specifically, the mobile robot can combine the obstacle's location and the robot's real-time location on a single operation map based on the obstacle's third position data. The mobile robot can then replan a path back to the base station, i.e., a second return path, based on its real-time location and the obstacle's location on the operation map.

[0130] In some embodiments, the cloud server transmits the obtained third position data of the obstacle to the mobile robot. Based on the third position data, the mobile robot can combine the obstacle's location with the mobile robot's real-time location on a single operation map. Based on the mobile robot's real-time location and the obstacle's location on the operation map, the mobile robot can replan a path back to the base station, i.e., a second return route.

[0131] In the above-mentioned method for guiding the mobile robot back to the base station, if the position of the obstacle changes, third position data of the obstacle after the change is obtained. Based on the third position data and the real-time position data of the mobile robot, the path is replanned to obtain a second return path for the mobile robot to return to the base station. For dynamic scenarios, an image acquisition device captures the working environment in front of the base station in real time. If the position of the obstacle changes, the environmental image is also updated in real time. The real-time position of the obstacle can be determined from the updated environmental image, and the path of the mobile robot returning to the base station can be replanned based on the real-time position of the obstacle.

[0132] In some embodiments, referring to FIG. 6 , if the position of the obstacle changes, obtaining the third position data after the obstacle changes may include the following steps:

[0133] S610: Receive a real-time image of the environment after the obstacle position changes, which is sent in real time by an image acquisition device.

[0134] S620: Calculate based on the real-time image of the environment and the internal and external parameter data of the image acquisition device to obtain third position data.

[0135] Specifically, due to changes in the position of obstacles, the image acquisition device needs to capture the base station's working environment in real time, obtain a real-time image of the environment, and upload the captured real-time image to a designated receiving device. The designated receiving device receives the real-time image of the environment and then, within the designated device, identifies the real-time image of the environment. By performing obstacle detection on the real-time image of the environment, the obstacles, mobile robots, and background (floor, wall, etc.) present in the real-time image of the environment can be determined. If an obstacle exists, the image coordinate system coordinates of the obstacle in the real-time image of the environment can be obtained by performing obstacle detection on the real-time image of the environment. Coordinate conversion is then performed based on the image coordinate system coordinates of the obstacle in the real-time image of the environment and the internal and external parameter data of the image acquisition device. The image coordinate system coordinates of the obstacle can be converted into the mobile robot's operation map. The position data after the obstacle position changes in the mobile robot's operation map is the third position data.

[0136] In the above-mentioned method for guiding the mobile robot back to the base station, a real-time image of the environment after the obstacle's position has changed, transmitted in real time by an image acquisition device, is received. A calculation is performed based on the real-time image of the environment and the internal and external parameters of the image acquisition device to obtain third position data. Obtaining this third position data allows the position of the obstacle after it has moved to be determined in real time, enabling the mobile robot to plan a path back to the base station in real time.

[0137] In some embodiments, the return-to-station reference line of the base station is fixed relative to the image acquisition device; the position of the return-to-station reference line is determined by external parameters of the image acquisition device and relative position data between the return-to-station reference line and the image acquisition device.

[0138] Among them, the return-to-station reference line can be the center line of the base station. For example, please refer to Figure 7. The line segment 710 in Figure 7 is the center line of the base station, which can be used as the return-to-station reference line of the base station. An image acquisition device is set at a position higher than the base station. The image acquisition device can also be a camera, and the mobile robot is located outside the base station.

[0139] In some cases, the base station's return reference line can enable the mobile robot to return to the base station more accurately, reducing the deviation between the position where the mobile robot stays in the base station and the position where the mobile robot should stay, and allowing the mobile robot to be cleaned more comprehensively.

[0140] In some embodiments, performing path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station includes:

[0141] Path planning is performed at least partially based on the second position data of the mobile robot and the position of the return-to-station reference line to obtain a first return-to-station path.

[0142] In some embodiments, the mobile robot can plan a path to return to the base station based on the second position data of the mobile robot and the position of the return reference line, and select an obstacle avoidance path from the multiple planned paths to return to the base station. The obstacle avoidance path can not only ensure the safety factor of obstacle avoidance, but also ensure that it can still return to the base station after avoiding the obstacle, and ensure that the turning time during obstacle avoidance is the shortest. Of course, the mobile robot can also send the multiple planned paths to return to the base station to the terminal device, and the terminal device specifies the corresponding path to return to the base station. The path to return to the base station can be regarded as the first return path. After specifying the first return path, the mobile robot is controlled to perform the operation of returning to the base station according to the path to return to the base station specified by the user on the terminal device side.

[0143] In the above-mentioned method for guiding the mobile robot back to the base station, path planning is performed at least in part based on the second position data of the mobile robot and the position of the return reference line to obtain a first return path. The first return path can effectively guide the mobile robot back to the base station in a complex pre-base station environment.

[0144] In some embodiments, referring to FIG. 8 , before performing path planning based at least in part on the second position data of the mobile robot and the position of the return reference line to obtain the first return path, the method may further include:

[0145] S810: Determine first position data of an obstacle in a base station working environment based on the environment image.

[0146] Performing path planning based at least in part on the second position data of the mobile robot and the position of the return-to-station reference line to obtain a first return-to-station path includes:

[0147] S820: Perform path planning based on the first position data of the obstacle, the second position data of the mobile robot, and the position of the return-to-station reference line to obtain a first return-to-station path.

[0148] In some embodiments, because the return reference line of the base station is relatively fixed to the image acquisition device, the position of the return reference line of the base station can be obtained through the external parameter data of the image acquisition device and the relative position data between the return reference line of the base station and the image acquisition device. Based on the principle of the shortest line segment between two points, the mobile robot can plan the path to return to the base station through its own positioning data and the position of the return reference line. In this path, if there is an obstacle, the path can be adjusted at the position where the obstacle exists in the path to avoid the obstacle, and the adjusted path to return to the base station can be obtained. If there is an obstacle in the adjusted path, the path at the location of the obstacle is also adjusted until there is no obstacle on the path back to the base station. The path finally determined to return to the base station can be regarded as the first return path, and the mobile robot can return to the base station according to the first return path.

[0149] In other embodiments, the planning of the return path can be based on the safety of obstacle avoidance, obstacle avoidance time, obstacle avoidance path, time to return to the base station, etc. The mobile robot can plan the path to return to the base station based on the first position data of the obstacle, the positioning data of the mobile robot, and the position of the reference line for returning to the station, and select an obstacle avoidance path from the multiple planned paths to return to the base station that can not only ensure the safety factor of obstacle avoidance, but also ensure that it can still return to the base station after avoiding the obstacle and has the shortest turning time when avoiding the obstacle. Of course, the mobile robot can also send the multiple planned paths back to the base station to the terminal device, and the terminal device specifies the corresponding path back to the base station. The path back to the base station can be regarded as the first return path. After specifying the first return path, the mobile robot is controlled to perform the operation of returning to the base station according to the path back to the base station specified by the user on the terminal device side.

[0150] It should be noted that obstacles can be categorized into various types, such as those that do not require avoidance (e.g., low steps that are lower than the mobile robot can traverse) and those that do require avoidance (e.g., scales, shoes, etc.). The obstacle can be first determined based on its category, location, and depth to determine whether it requires avoidance. If it does not, the robot continues its return to the base station in the direction of the current obstacle. If it does, the robot determines whether it is an avoidable obstacle. If so, the robot is controlled to perform automatic obstacle avoidance. If the obstacle is an avoidable obstacle, the robot can further determine whether it requires clearing. If so, the robot is cleared without performing automatic obstacle avoidance. If not, the robot is controlled to perform automatic obstacle avoidance. If the obstacle is unavoidable, the robot replans the path back to the base station without performing automatic obstacle avoidance. When it is determined that the robot vacuum cleaner needs to avoid an obstacle and is able to avoid it, a path back to the base station can be planned for the robot vacuum cleaner based on the location and depth information corresponding to the obstacle type. A TOF (Time of Flight) camera can be used to further obtain depth information about the obstacle.

[0151] In the above method for guiding a mobile robot back to the base station, path planning is performed based on the first position data of the obstacle, the positioning data of the mobile robot, and the position of the return reference line to obtain a first return path. This first return path can effectively guide the mobile robot back to the base station in a complex pre-station environment.

[0152] In some embodiments, the base station is set at the bottom of the household appliance or as a part of the household appliance, and the image acquisition device is set on the household appliance; or the base station is set at the bottom of the furniture or as a part of the furniture, and the image acquisition device is set on the furniture.

[0153] Home appliances may include laundry machines, such as washing machines, dryers, and washer-dryers. They may also include refrigerators, dishwashers, televisions, air purifiers, water purifiers, air conditioners, steamers, ovens, sinks, and integrated stoves. Furniture may include shoe cabinets, TV cabinets, bookcases, bathtubs, fish tanks, cabinets, toilets, fixed tables and chairs, bar counters, and sinks.

[0154] For example, please refer to Figure 9, where 910 is the location of the image acquisition device. Specifically, 910 can be a camera. 920 is the location of the base station. Because the clothing processing device is equipped with an independent socket for power supply and automatic water supply and drainage devices, the clothing processing device can be placed on the balcony floor. The base station can automatically drain water through the drain port shared with the clothing processing device. Therefore, it is very convenient to install the base station next to the clothing processing device. Installing an image acquisition device on the clothing processing device can collect information about the base station's working environment so that the mobile robot can plan a path back to the base station.

[0155] The present disclosure also provides a method for guiding a mobile robot back to a station. A base station is disposed at the bottom of a household appliance, and an image acquisition device is disposed on the household appliance. The household appliance includes a laundry processing device. For example, referring to FIG. 10 , the method for guiding a mobile robot back to a station may include the following steps:

[0156] S1002: Receive an environment image sent by an image acquisition device.

[0157] The environmental image is obtained by photographing the base station working environment through image acquisition equipment.

[0158] S1004: Perform obstacle detection on the environment image to obtain the image coordinate system coordinates of the obstacle in the environment image.

[0159] S1006 , performing coordinate conversion based on the image coordinate system coordinates of the obstacle and the internal and external parameter data of the image acquisition device to obtain first position data of the obstacle in the operation map of the mobile robot.

[0160] The first position data is calculated based on the environmental image and the internal and external parameter data of the image acquisition device at the preset position; the height of the image acquisition device is greater than the height of the base station.

[0161] S1008. Determine second position data of the mobile robot in the operation map based on the environmental image and the internal and external parameter data of the image acquisition device.

[0162] S1010: If the position of the obstacle does not change, perform path planning based on the first position data of the obstacle, the second position data of the mobile robot, and the position of the return-to-station reference line to obtain a first return-to-station path.

[0163] The return-to-station reference line of the base station is relatively fixed to the image acquisition device; the position of the return-to-station reference line is determined by the external parameters of the image acquisition device and the relative position data between the return-to-station reference line and the image acquisition device.

[0164] S1012: If the position of the obstacle changes, receive a real-time image of the environment after the position of the obstacle changes, which is sent in real time by the image acquisition device.

[0165] S1014: Calculate based on the real-time image of the environment and the internal and external parameter data of the image acquisition device to obtain third position data.

[0166] S1016: Re-plan the path based on the third position data and the real-time position data of the mobile robot to obtain a second return path for the mobile robot to return to the base station.

[0167] An embodiment of the present disclosure provides an apparatus 1100 for guiding a mobile robot to return to a station. Referring to FIG. 11 , the apparatus 1100 for guiding a mobile robot to return to a station includes: an environment image acquisition module 1110 and an identification and guidance module 1120 .

[0168] The environmental image acquisition module 1110 is configured to acquire an environmental image, wherein the environmental image is obtained by photographing the base station working environment using an image acquisition device; the image acquisition device is located at a height greater than the height of the top of the base station;

[0169] The identification and guidance module 1120 is configured to identify the mobile robot and / or guide the mobile robot back to the base station based at least in part on the environment image.

[0170] For a detailed description of the device for guiding the mobile robot to return to the station, please refer to the description of the method for guiding the mobile robot to return to the station above, which will not be repeated here.

[0171] The present disclosure provides a base station including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the above embodiments. The household appliance includes a clothes processing device.

[0172] The present disclosure provides a household appliance comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the above embodiments. The household appliance comprises a clothes processing device.

[0173] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0174] One embodiment of the present disclosure provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0175] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this disclosure, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

Claims

1. A method for guiding a mobile robot to return to a station, the method comprising: Acquire an environmental image; wherein the environmental image is obtained by photographing the base station working environment through an image acquisition device; the height of the image acquisition device is greater than the height of the top of the base station; Based at least in part on the image of the environment, a mobile robot is identified and / or directed to return to the base station.

2. The method according to claim 1, wherein: The step of guiding the mobile robot to return to the base station comprises: Determining first position data of an obstacle in the working environment of the base station based on the environment image; Performing path planning based on the first position data and the second position data of the mobile robot to obtain a first return path for the mobile robot to return to the base station; The mobile robot is guided to return to the base station according to the first return path.

3. The method according to claim 1, wherein: The step of guiding the mobile robot to return to the base station comprises: Based on the environment image, determining second position data of the mobile robot; Performing path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station; The mobile robot is guided to return to the base station according to the first return path.

4. The method according to claim 2, wherein: The method for determining the first position data includes: Receiving the environment image sent by the image acquisition device; Perform obstacle detection based on the environment image to obtain the image coordinate system coordinates of the obstacle in the environment image; Coordinate conversion is performed according to the image coordinate system coordinates of the obstacle and the internal and external parameter data of the image acquisition device to obtain the first position data of the obstacle in the operation map.

5. The method according to claim 1, wherein: Based at least in part on the image of the environment, directing the mobile robot back to the base station includes: Determine the second position data of the mobile robot in the operation map based on the environmental image and the internal and external parameter data of the image acquisition device; determine the return reference point of the base station according to the external parameter data of the image acquisition device and the relative position data between the image acquisition device and the return reference point; guide the mobile robot back to the base station through the second position data and the return reference point; or Based on the environmental image and the internal and external parameter data of the image acquisition device, the second position data of the mobile robot in the operation map is determined; according to the second position data and the position data of the base station, path planning is performed to guide the mobile robot back to the base station.

6. The method according to claim 2, wherein: The method comprises: If the position of the obstacle changes, obtaining third position data of the obstacle after the change; The path is re-planned based on the third position data and the real-time position data of the mobile robot to obtain a second return path for the mobile robot to return to the base station.

7. The method according to claim 6, wherein: If the position of the obstacle changes, obtaining third position data of the obstacle after the change includes: Receiving a real-time image of the environment after the position of the obstacle changes, which is sent in real time by the image acquisition device; The third position data is obtained by performing calculation based on the real-time image of the environment and the internal and external parameter data of the image acquisition device.

8. The method according to claim 3, wherein: The return-to-station reference line of the base station is relatively fixed to the image acquisition device; the position of the return-to-station reference line is determined by external parameters of the image acquisition device and relative position data between the return-to-station reference line and the image acquisition device.

9. The method according to claim 8, wherein: Performing path planning based at least in part on the second position data to obtain a first return path for the mobile robot to return to the base station includes: Path planning is performed at least partially based on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path.

10. The method according to claim 9, wherein: Before performing path planning based at least in part on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path, the method further includes: Determining first position data of an obstacle in the working environment of the base station based on the environment image; The performing path planning at least partially based on the second position data of the mobile robot and the position of the return-to-station reference line to obtain the first return-to-station path includes: Path planning is performed based on the first position data of the obstacle, the second position data of the mobile robot, and the position of the return-to-station reference line to obtain the first return-to-station path.

11. The method according to any one of claims 1 to 10, wherein: The base station is arranged at the bottom of the household appliance or is a part of the household appliance, and the image acquisition device is arranged on the household appliance; or The base station is arranged at the bottom of the furniture or as a part of the furniture, and the image acquisition device is arranged on the furniture.

12. The method according to claim 11, wherein: The household appliance includes a clothes treating device.

13. A base station, comprising: A processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to perform the steps of the method according to any one of claims 1 to 12 when running the computer program.

14. A household appliance, comprising: A processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to perform the steps of the method according to any one of claims 1 to 12 when running the computer program.

15. The device according to claim 14, wherein: The household appliance includes a clothes treating device.

16. A device for guiding a mobile robot to return to a station, the device comprising: An environmental image acquisition module, used to acquire an environmental image; wherein the environmental image is obtained by photographing the base station working environment through an image acquisition device; the height of the image acquisition device is greater than the height of the top of the base station; An identification and guidance module is used to identify the mobile robot and / or guide the mobile robot back to the base station based at least in part on the environment image.

Citation Information

Patent Citations

  • Mobile robot path planning method and device based on image vision

    CN105955279A

  • Autonomous navigation method of robot based on video monitoring

    CN107368074A

  • Robot control system and control method thereof

    CN111872935A

  • Target area monitoring method and system

    CN113676696A

  • Map generation method and device, equipment and storage medium

    CN114705191A