Cleaning method, cleaning display method, cleaning device, and computer program product

Abstract

The application discloses a cleaning method, a cleaning display method, a cleaning device and a computer program product. The cleaning method comprises the following steps: acquiring a scene image containing a target object; determining first position information of a first key point of the target object and second position information of a second key point of the target object in the scene image, and determining a pointing position according to the first position information and the second position information, wherein the first key point and the second key point are located at a target part of the target object; and moving to the pointing position to clean a to-be-cleaned area corresponding to the pointing position. By acquiring the image coordinates of the target part of the user in the scene image containing the user, the position information of the target part can be calculated, the pointing position of the user is determined according to the position information, and then the cleaning device is controlled to the pointing position to clean. Compared with the cleaning device which can only move to the pointing position according to a preset path to clean, the cleaning device can realize the function of cleaning where the user points, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of cleaning technology, and more specifically, to a cleaning method, a cleaning demonstration method, a cleaning device, and a computer program product. Background Technology

[0002] As people's pace of life accelerates, robotic vacuum cleaners are playing an increasingly important role in daily life. Traditional robotic vacuum cleaners clean according to the user's preset cleaning path. However, when the user needs to clean a specific area, the robotic vacuum cleaner cannot clean according to the user's needs and can only clean the specific area by moving to the preset path, resulting in a poor user experience. Summary of the Invention

[0003] This application provides a cleaning method, a cleaning display method, a cleaning device, and a computer program product.

[0004] The cleaning method of this application includes acquiring a scene image containing a target object; determining first position information of a first key point of the target object and second position information of a second key point of the target object in the scene image, and determining a pointing position based on the first position information and the second position information, wherein the first key point and the second key point are located at a target part of the target object; moving to the pointing position to clean the area to be cleaned corresponding to the pointing position.

[0005] The cleaning display method of this application includes displaying the current position and posture of the cleaning device in a map corresponding to a scene image, as well as the pointing position of the target object of the cleaning device. The current position and posture are determined based on the current pose information of the cleaning device. The pointing position is determined based on the first position information of the first key point of the target object and the second position information of the second key point of the target object in the scene image collected by the visual sensor of the cleaning device. The first key point and the second key point are located at the target part of the target object.

[0006] The cleaning device of this application includes a processor, a memory, and a computer program, wherein the computer program is stored in the memory and executed by the processor, and the computer program includes instructions for performing the cleaning method. The cleaning method includes acquiring a scene image containing a target object; determining first position information of a first key point of the target object and second position information of a second key point of the target object in the scene image, and determining a pointing position based on the first position information and the second position information, wherein the first key point and the second key point are located at a target part of the target object; moving to the pointing position to clean the area to be cleaned corresponding to the pointing position.

[0007] The computer program product of this application includes a computer program comprising instructions for executing the cleaning method and instructions for the cleaning display method. The cleaning method includes acquiring a scene image containing a target object; determining first position information of a finger keypoint and second position information of a shoulder keypoint of the target object in the scene image, and determining a finger pointing position based on the first and second position information, wherein the first and second keypoints are located at a target part of the target object; and moving to the finger pointing position to clean the area to be cleaned corresponding to the finger pointing position. The cleaning display method includes displaying the current position and posture of the cleaning device in a map corresponding to the scene image, and the pointing position of the cleaning device at the target object, wherein the current position and posture are determined based on the current pose information of the cleaning device; wherein the pointing position is determined based on the first position information of a first keypoint and the second position information of a second keypoint of the target object in the scene image acquired by the visual sensor of the cleaning device, wherein the first and second keypoints are located at a target part of the target object.

[0008] The cleaning method, cleaning demonstration method, cleaning equipment, and computer program product of this application acquire scene images containing the user by capturing images of the environment where the target object is located using a camera mounted on the cleaning equipment. Then, by recognizing the scene image, the position information of the user's target body part (such as the head, arm, etc.) in the scene image can be determined, thereby determining the position information of the user's first key point and second key point (such as the user's first key point being a finger key point and the second key point being a shoulder key point).

[0009] Next, after accurately identifying the location information of the user's finger key points and shoulder key points from the scene image, it can accurately determine the finger pointing position (such as the intersection of the straight line passing through the finger key point and the shoulder key point on the ground), thereby accurately determining the area to be cleaned specified by the user.

[0010] Finally, the cleaning device is controlled to move to the area to be cleaned corresponding to the finger's pointing position. Thus, compared to a robotic vacuum cleaner that can only clean by moving to a specific area according to a preset path, the cleaning device of this application can quickly and conveniently move to a specific area to be cleaned by recognizing the user's pointing position, thereby enabling the cleaning device to achieve the function of "point-and-sweep," improving the user experience.

[0011] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0013] Figure 1 This is a plan view of a cleaning method according to certain embodiments of this application;

[0014] Figure 2 This is a schematic diagram illustrating an application scenario of the cleaning method according to certain embodiments of this application;

[0015] Figure 3 This is a plan view of a sweeping robot according to certain embodiments of this application;

[0016] Figure 4 This is a schematic diagram of a cleaning method according to certain embodiments of this application;

[0017] Figure 5 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0018] Figure 6 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0019] Figure 7 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0020] Figure 8 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0021] Figure 9This is a schematic diagram of a cleaning method according to certain embodiments of this application;

[0022] Figure 10 This is a schematic diagram of a cleaning method according to certain embodiments of this application;

[0023] Figure 11 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0024] Figure 12 This is a schematic diagram of a cleaning method according to certain embodiments of this application;

[0025] Figure 13 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0026] Figure 14 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0027] Figure 15 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0028] Figure 16 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0029] Figure 17 This is a flowchart illustrating a cleaning method according to certain embodiments of this application;

[0030] Figure 18 This is a schematic diagram of a cleaning method according to certain embodiments of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0032] To facilitate understanding of this application, the following explanations are provided for the terms used in this application:

[0033] A robotic vacuum cleaner, also known as an automatic cleaning robot, smart vacuum cleaner, or robotic vacuum cleaner, is a type of smart home appliance that uses artificial intelligence to automatically clean floors in a room. It typically uses a combination of brushing and vacuuming to collect debris into its own dustbin, thus completing the cleaning process. Generally, robots that perform sweeping, vacuuming, and mopping are collectively categorized as robotic vacuum cleaners.

[0034] The self-moving robot of this application can be a vacuum cleaner robot, or a sweeping / mopping / brushing / sweeping and mopping robot, etc. For simplicity, this application uses a sweeping robot as an example for explanation. The principle of other types of self-moving robots is similar and will not be elaborated here.

[0035] When users suddenly decide they need to clean a specific area, traditional robot vacuums can only clean according to a preset cleaning path. They cannot clean specific areas according to the user's wishes, resulting in a poor user experience.

[0036] To address the aforementioned technical problems, this application provides a cleaning method.

[0037] The following section will first introduce one application scenario of the technical solution of this application, such as... Figure 1 As shown, the cleaning method provided in this application can be applied to, for example... Figure 1 In the application scenario shown, the cleaning method is applied to a cleaning system 1000, which includes cleaning equipment.

[0038] Please see Figure 2 The cleaning equipment in this application may include only the sweeping robot 100, or the cleaning equipment may include the sweeping robot 100 and the base station 200 (or dust collection station). The sweeping robot 100 and the base station 200 can be connected via a network to determine the current status of the other end (e.g., battery status, working status, location information, etc.).

[0039] The robotic vacuum cleaner 100 includes a vision sensor 10, a radar sensor 20, a processor 30, a memory 40, and a body 50. The processor 30 communicates with the vision sensor 10 and the radar sensor 200 via a network. The vision sensor 10, the radar sensor 20, and the processor 30 are mounted on the body 50 of the robotic vacuum cleaner 100.

[0040] The vision sensor 10 is used to acquire scene images; the vision sensor 10 can be a visible light camera (Red-Green-Blue, RGB), a visible light depth camera (Red-Green-Blue-Depth, RGBD), an infrared camera, a thermal imaging camera, a depth camera, etc. The RGB camera and RGBD camera can acquire visible light images of the scene, the infrared camera can acquire infrared images of the scene, the thermal imaging camera can acquire thermal imaging images of the scene, and the depth camera can acquire depth images.

[0041] Optionally, the vision sensor 10 may include one or more. The vision sensor 10 may be disposed on the side wall of the body 50, such as when the vision sensor 10 is disposed in the front direction of the robot vacuum cleaner 100 to acquire scene images in front of the robot vacuum cleaner 100, or when the vision sensor 10 is disposed on both sides of the robot vacuum cleaner 100 to acquire scene images on both sides as the robot vacuum cleaner 100 moves forward.

[0042] Radar sensor 20 is used to collect point cloud information of objects in the scene. Radar sensor 20 can be a laser radar (LDS), such as a TOF radar based on the time-of-flight (TOF) principle, or a structured light radar based on the structured light principle.

[0043] The radar sensor 20 is mounted on the top wall of the robotic vacuum cleaner 100. The radar sensor 20 may protrude from the top wall, or it may be mounted inside the body 50 without protruding from it. In other words, the height of the radar sensor 20 may be lower than the height of the top wall.

[0044] When the cleaning equipment receives a targeted cleaning instruction from the target object M, the processor 30 obtains information containing the target object M (e.g., ...). Figure 3 The system uses a scene image (as shown) to determine the first position information of the finger key point M1 and the second position information of the shoulder key point M2 of the target object M. Based on the first and second position information, it determines the finger-pointing position P, thereby controlling the robotic vacuum cleaner 100 to move to the finger-pointing position P to clean the area to be cleaned corresponding to the finger-pointing position P. The fixed-point cleaning command can be issued via voice summoning information from the target object M, via a terminal application, via buttons or touch operations on the robotic vacuum cleaner, or via buttons or touch operations on a base station.

[0045] In one embodiment, the robot vacuum cleaner 100 further includes a memory 40, which stores scene images captured by the vision sensor 10, as well as position information of finger key points M1, shoulder key points M2, and finger pointing position P.

[0046] In one embodiment, the cleaning system 1000 further includes a server 400, which communicates with the cleaning device via a network. The server 400 is used to receive scene images collected by the visual sensor 10 sent by the robot vacuum cleaner 100 or the base station 200, obtain the first position information of the finger key point M1 of the target object M and the second position information of the shoulder key point M2 of the target object M in the scene image, and determine the finger pointing position P according to the first position information and the second position information, thereby controlling the robot vacuum cleaner 100 to move to the finger pointing position P to clean the area to be cleaned corresponding to the finger pointing position P.

[0047] In one embodiment, server 400 can be a standalone physical server 400, a server cluster or distributed system composed of multiple physical servers 400, or a cloud server 400 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application embodiment does not impose any limitations on these aspects.

[0048] The base station 200 may include a display screen 201. The base station 200 can communicate with the robot vacuum cleaner 100 to obtain the data transmitted by the robot vacuum cleaner 100, and can use the processing capability of the base station 200 to process the data, thereby realizing the control of the robot vacuum cleaner 100 (such as controlling the robot vacuum cleaner 100 to move to the target location for cleaning), display of relevant content of the robot vacuum cleaner 100, and other functions.

[0049] In one embodiment, the system further includes a terminal 300, which includes a display 301. The terminal 300 can communicate with the robotic vacuum cleaner 100 to obtain data transmitted by the robotic vacuum cleaner 100, and can use the processing power of the terminal 300 to process the data, thereby realizing functions such as controlling the robotic vacuum cleaner 100 (e.g., controlling the robotic vacuum cleaner 100 to move to a target location for cleaning), and displaying relevant content of the robotic vacuum cleaner 100.

[0050] In one embodiment, terminal 300 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0051] For example, at least one of the display screen 201 of the base station 200 and the display 301 of the terminal 300 can process the location information transmitted by the robotic vacuum cleaner 100 to determine the location of the robotic vacuum cleaner 100 on the map, and then display the current location of the robotic vacuum cleaner 100 on the map and the movement trajectory of the robotic vacuum cleaner 100 in real time. As another example, the status information transmitted by the robotic vacuum cleaner 100 can be processed to determine the current status of the robotic vacuum cleaner 100, and then the current status of the robotic vacuum cleaner 100 can be displayed in real time.

[0052] In one embodiment, the robotic vacuum cleaner 100, base station 200, terminal 300, and server 400 all communicate via a network. For example, any two of the robotic vacuum cleaner 100, base station 200, terminal 300, and server 400 can communicate wirelessly (e.g., via Wireless Fidelity (Wi-Fi), Bluetooth, infrared, etc.). It is understood that the communication between the robotic vacuum cleaner 100, base station 200, terminal 300, and server 400 is not limited to the above-mentioned communication methods, and no limitation is imposed.

[0053] When communicating via Wi-Fi, the robot vacuum cleaner 100 and the base station 200 communicate with the cloud server 400 respectively, and then the cloud server 400 enables communication between the robot vacuum cleaner 100 and the base station 200 (or the terminal 300); when communicating via Bluetooth or infrared, the robot vacuum cleaner 100 and the base station 200 (or the terminal 300) are each equipped with a corresponding communication module to directly enable communication between the two.

[0054] In one embodiment, the cleaning method can be implemented by at least one of the following three components: a robotic vacuum cleaner 100, a base station 200, a terminal 300, and a server 400. For example, it can be implemented by the robotic vacuum cleaner 100 in conjunction with the base station 200, the terminal 300, and the server 400; or it can be implemented by the robotic vacuum cleaner 100 in conjunction with the base station 200; or it can be implemented by the robotic vacuum cleaner 100 in conjunction with the terminal 300, etc.

[0055] Please see Figure 3 , Figure 4 and Figure 5 This cleaning method is applied to Figure 1 Taking the cleaning equipment in the example, the cleaning method of this application includes:

[0056] Step 011: Obtain the scene image containing the target object M;

[0057] Specifically, when the robot vacuum cleaner 100 receives a voice call message, a call message sent by the application of the terminal 300, a call message sent by the button or touch operation on the robot vacuum cleaner 100, or a call message sent by the button or touch operation on the base station 200, and the finger key point M1 of the target object M points to the area that needs to be cleaned, the processor 30 can control the vision sensor 10 to take a picture of the target object M, so that the robot vacuum cleaner 100 can generate a scene image containing the target object M and store it in the memory 40.

[0058] For example, when a user discovers an area that needs cleaning, they can issue a voice command such as "This needs cleaning" or "Start cleaning" and point their finger at the area that needs cleaning. The processor 30 can then control the visual sensor 10 installed on the robot vacuum cleaner 100 body 50 to take a picture of the user, thereby generating a scene image containing the user and storing the scene image in the memory 40.

[0059] Step 012: Determine the first position information of the first key point M1 of the target object M and the second position information of the second key point M2 of the target object M in the scene image, and determine the pointing position P based on the first position information and the second position information. The first key point M1 and the second key point M2 are located at the target part of the target object M.

[0060] The target body part of the target object M can be any part of the body, such as the head, arms, and legs. The first key point M1 and the second key point M2 can be located at any position of the target body part, such as the first key point M1 being located at one end of the target body part and the second key point being located at the other end of the target body part.

[0061] For example, when the target area is the head, the first key point M1 can be the tip of the nose, and the second key point M2 can be the chin; when the target area is the arm, the first key point M1 can be the finger, and the second key point M2 can be the shoulder; when the target area is the leg, the first key point M1 can be the toe, and the second key point M2 can be the knee. This application uses the arm as an example of the target area in its implementation. The cleaning method of this application also applies to implementations targeting the head and legs, and for simplicity, will not be elaborated upon here.

[0062] When the target area is the head, the first key point M1 can be the tip of the nose, and the second key point M2 can be the chin. When the target area is the arm, the first key point M1 can be the finger, and the second key point M2 can be the shoulder. When the target area is the leg, the first key point M1 can be the toe, and the second key point M2 can be the knee. This application uses the arm as an example of the target area in its implementation. The cleaning method of this application also applies to implementations targeting the head and legs, and for simplicity, will not be elaborated upon here.

[0063] Specifically, after the processor 30 acquires the scene image generated by the vision sensor 10, the processor 30 needs to determine the first key point M1 and the second key point M2 of the target object M in the scene image. The first key point M1 of the target object M can be a finger key point, a toe key point, or a chin key point, etc., and the second key point M2 of the target object M can be a shoulder key point, a knee key point, or a nose tip key point, etc. The processor 30 also determines the first position information of the first key point M1 and the second position information of the second key point M2 of the target object M (such as coordinate values, direction, and distance, etc.). Based on the acquired first position information of the first key point M1 and the second position information of the second key point M2 of the target object M, the processor 30 can determine the pointing position P of the target object M, which can be a finger pointing position, a toe pointing position, etc.

[0064] Step 013: Move to the indicated position P to clean the area to be cleaned corresponding to the indicated position P.

[0065] Specifically, taking the finger pointing to position P as an example, after determining the finger pointing to position P, the processor 30 can plan the path from the current position of the robot vacuum cleaner 100 to the finger pointing to position P. Thus, the processor 30 can control the robot vacuum cleaner 100 to move to the finger pointing to position P and clean the area to be cleaned corresponding to the finger pointing to position P. The area to be cleaned can be a circular area with the target position as the center, or the area to be cleaned can be a rectangular area with the target position as the center of symmetry.

[0066] Thus, by using the vision sensor 10 installed on the robot vacuum cleaner 100 body 50 to capture images of the environment in which the target object M is located, a scene image containing the target object M can be obtained. Then, by recognizing the scene image, the position information of the first key point M1 of the target object M in the scene image (such as determining the coordinates of the user's finger) and the position information of the second key point M2 of the target object M (such as determining the coordinates of the user's shoulder) can be determined.

[0067] Next, based on the location information of the user's first key point M1 and the location information of the user's second key point M2, the pointing position P of the target object M can be determined (e.g., the pointing position of the finger is the intersection of the straight line passing through the finger key point and the shoulder key point on the ground). The pointing position P of the target object M is the cleaning position that the user specifies needs to be cleaned.

[0068] Finally, the robot vacuum cleaner 100 is controlled to move to the area to be cleaned corresponding to the finger's pointing position P. Thus, compared to the robot vacuum cleaner 100 which can only clean when it moves to a specific area according to a preset path, the cleaning device of this application can quickly and conveniently go to the specific area to be cleaned by recognizing the user's pointing position P, thereby enabling the cleaning device to achieve the function of "point-and-sweep," improving the user experience.

[0069] Please see Figure 6 In some embodiments, the cleaning method further includes:

[0070] Step 014: Identify the current pose of the target object M;

[0071] Step 015: Based on the current posture, determine the first key point M1 and the second key point M2 in the target part that match the current posture.

[0072] Specifically, the processor 30 can identify the current pose of the target object M by recognizing the scene image including the target object M. The current pose of the target object M can be the relative positional relationship between target parts of the target object M. Based on the current pose of the target object M, the processor 30 can determine the first key point M1 and the second key point M2 in the target parts that match the current pose.

[0073] The target part determined by the processor 30 in the scene image can be set in advance. Alternatively, the processor 30 can determine the target part based on the current posture of the target object M in the scene image. For example, when the current posture of the target object M is an outstretched arm posture or a bent arm posture, the processor 30 can determine that the target part is the arm; when the current posture of the target object M is an outstretched leg posture or a bent leg posture, the processor 30 can determine that the target part is the leg; and when the current posture of the target object M is a head-up posture, the processor 30 can determine that the target part is the head.

[0074] Then, the processor 30 can determine the first key point M1 and the second key point M2 in the target part that match the current posture, based on the current posture.

[0075] For example, when the current posture is an extended arm posture, the processor 30 can determine that the first keypoint M1 is one of the finger keypoint, elbow keypoint, and shoulder keypoint, and the second keypoint M2 is the other one of the finger keypoint, elbow keypoint, and shoulder keypoint.

[0076] When the current posture is a bent arm posture, the processor 30 can determine that the first key point M1 is one of the finger key point and the elbow key point, and the second key point M2 is the other of the elbow key point and the shoulder key point.

[0077] When the current posture is an extended leg posture, the processor 30 can determine that the first key point M1 is one of the toe key point, the knee key point, and the hip joint key point, and the second key point M2 is the other one of the toe key point, the knee key point, and the hip joint key point.

[0078] When the current posture is a bent-leg posture, the processor 30 can determine that the first key point M1 is one of the toe key point and the knee key point, and the second key point M2 is the other of the knee key point and the hip joint key point.

[0079] When the current posture is a head-up posture, the processor 30 can determine that the first key point M1 is one of the chin key point, the tip of the nose key point, and the forehead key point, and the second key point M2 is the other one of the chin key point, the tip of the nose key point, and the forehead key point.

[0080] Thus, by recognizing the current posture of the target object M in the scene image, the first key point M1 and the second key point M2 matching the current posture in the target part can be determined. This eliminates the need for the target object M to consciously adjust its posture, thereby improving the accuracy of the robot vacuum cleaner 100 in recognizing the first key point M1 and the second key point M2.

[0081] Please see Figure 7 In some embodiments, the cleaning method is applied to a cleaning device, the radar sensor 20 of which is configured to collect point cloud information of objects within a first preset height, the first preset height being less than the height of the target part. Step 012: Determine the first position information of the first key point M1 of the target object M and the second position information of the second key point M2 of the target object M in the scene image, and determine the pointing position P based on the first and second position information, including:

[0082] Step 0121: Based on the preset key point detection model, identify the first position information and the second position information of the target object M in the scene image. The first position information is the first image coordinate of the first key point M1 in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point M2 in the image coordinate system of the scene image.

[0083] Step 0122: Based on the preset calibration parameters of the visual sensor 10 and radar sensor 20 of the cleaning equipment, obtain the target point cloud information corresponding to the second position information from all point cloud information of the radar sensor 20;

[0084] Step 0123: Determine the finger pointing position P based on the first position information, the second position information, and the target point cloud information.

[0085] Specifically, the processor 30 controls the radar sensor 20 of the robotic vacuum cleaner 100 to collect point cloud information of objects within a preset first preset height (e.g., the first preset height can be 40 cm, 50 cm, etc.) of the radar sensor 20. The first preset height is less than the height of the target part of the target object M; for example, if the target part of the target object M is a finger or shoulder, the first preset height is less than the height of the shoulder of the target object M.

[0086] After acquiring the scene image generated by the vision sensor 10, the processor 30 transmits the scene image to the key point detection model preset in the robot vacuum cleaner 100, so as to identify the first position information of the finger key point M1 and the second position information of the shoulder key point M2 of the outstretched arm of the target object M in the scene image. It should be noted that an image coordinate system can be established for the scene image in the key point detection model. The first position information of the finger key point M1 can be the first image coordinate in the image coordinate system of the scene image, and the second position information of the shoulder key point M2 can be the second image coordinate in the image coordinate system of the scene image.

[0087] After acquiring the first image coordinates of the finger key point M1 and the second image coordinates of the shoulder key point M2, the processor 30 can determine the target point cloud information corresponding to the second image coordinates of the shoulder key point M2 from all point cloud information of the environment where the target object M is located (such as the point cloud information being a set of point coordinates of the object's surface detected by the radar sensor 20) collected by the radar sensor 20, based on the preset calibration parameters of the visual sensor 10 and the radar sensor 20 of the robot vacuum cleaner 100.

[0088] Finally, the processor 30 can determine the finger pointing position P based on the first image coordinates of the acquired finger key point M1, the second image coordinates of the shoulder key point M2, and the target point cloud information.

[0089] Optionally, if the first key point is the toe key point, the second key point is the knee key point, and the target object M is pointing at the toe key point, the method for determining the toe pointing position is basically similar to the method for determining the finger pointing position P, and will not be elaborated here for the sake of simplicity.

[0090] In this way, by setting the preset height of the radar sensor 20 to be less than the height of the shoulder, the height requirement for the radar sensor 20 to collect point clouds can be reduced. It is only necessary to collect the target point cloud of the foot near the ground corresponding to the shoulder key point M2, thereby reducing the cost of using the radar sensor 20 and reducing the amount of computation required by the cleaning equipment.

[0091] Please see Figure 8 In some implementations, in the scene image, step 0122: based on the preset calibration parameters of the visual sensor 10 and radar sensor 20 of the cleaning equipment, the target point cloud information corresponding to the second position information is obtained from all point cloud information of the radar sensor 20, including:

[0092] Step 01221: Determine the target region in the scene image where the second keypoint M2 is located. The target region includes one or more columns of pixels.

[0093] Step 01222: Based on the preset calibration parameters, obtain the set of point cloud information located within the target area from all point cloud information of the radar sensor 20;

[0094] Step 01223: Based on the point cloud information set, determine the target point cloud information corresponding to the second key point M2;

[0095] Step 0123: Determine the pointing location P based on the first location information, the second location information, and the target point cloud information, including:

[0096] Step 01231: Determine the pointing position P based on the first position information, the second position information, the target point cloud information, and the height and length of the target part.

[0097] Specifically, the processor 30 obtains the target point cloud information corresponding to the second position information of the shoulder key point M2. This can be achieved by determining the target area where the shoulder key point M2 is located in the scene image (e.g., the target area can be a circular area centered on the shoulder key point M2, or a rectangular area symmetrically centered on the shoulder key point M2), and obtaining one or more columns of pixels included in the target area.

[0098] After acquiring the pixels of the target area, the processor 30 determines the set of point cloud information corresponding to the pixels of the target area from all the point cloud information collected by the radar sensor 20 based on the extrinsic parameters between the visual sensor 10 and the radar sensor 20 of the robot vacuum cleaner 100. By calculating the average point cloud information in the point cloud information set, the processor 30 can determine the target point cloud information corresponding to the shoulder key point M2 as the average point cloud information.

[0099] The processor 30 can calculate the shoulder height and arm length using the pre-inputted height of the target object M, the pre-inputted ratio of shoulder height to body height as a first preset ratio (e.g., the first preset ratio could be 4:5), and the ratio of arm length to body height as a second preset ratio (e.g., the second preset ratio could be 1:2). Finally, the processor 30 can determine the finger pointing position P based on the first finger position information, the second shoulder position information, the shoulder height, the arm length, and the target point cloud information.

[0100] Thus, by using one or more columns of pixels in the target area where the shoulder key point M2 is located, and according to preset calibration parameters, the point cloud set corresponding to the target area in the point cloud information collected by the radar sensor 20 is determined. Based on the point cloud set, the target point cloud information corresponding to the shoulder key point M2 (such as the point cloud corresponding to the foot near the ground corresponding to the shoulder) is determined. Finally, the finger pointing position P is determined based on the position information of the finger key point M1 and the shoulder key point M2, the arm length, the shoulder height, and the target point cloud information, which can improve the accuracy of determining the finger pointing position P.

[0101] Please see Figure 9 , Figure 10 and Figure 11 In some implementations, step 0123: determining the pointing position P based on the first position information, the second position information, the target point cloud information, and the height and length of the target part, includes:

[0102] Step 01232: Determine the projected length and pitch angle of the target part based on the first and second position information;

[0103] Step 01233: Determine the yaw angle of the target part based on the projected length and the length of the target part;

[0104] Step 01234: Determine the third position information based on the height and pitch angle of the target part, and determine the fourth position information based on the height, pitch angle and yaw angle of the target part. The third position information and the fourth position information are the position of the pointing position P relative to the target point cloud information.

[0105] Step 01235: Determine the pointing position P based on the third position information, the fourth position information, and the target point cloud information.

[0106] Specifically, an image coordinate system is established with the target point cloud information as the origin, the direction directly in front of the target object M as the Y-axis, and the direction parallel to the ground and perpendicular to the Y-axis as the X-axis. After acquiring the first position information of the finger keypoint M1 and the second position information of the shoulder keypoint M2, the processor 30 can obtain the coordinates of the finger keypoint M1 and the shoulder keypoint M2, thereby calculating the distance between the finger keypoint M1 and the shoulder keypoint M2, and then obtaining the projected length z of the arm. Based on the projected length x of the arm on the X-axis and the projected length y of the arm on the Y-axis, the pitch angle θ of the arm can be calculated using trigonometric functions.

[0107] Then, based on the projected length z of the arm and the length L of the arm, the yaw angle φ of the arm can be calculated using trigonometric relationships.

[0108] Next, the processor 30 determines the third position information X1 based on the shoulder height H and pitch angle θ using trigonometric functions, and determines the fourth position information Y1 based on the shoulder height H, pitch angle θ, and yaw angle φ using trigonometric functions. It should be noted that the third position information X1 and the fourth position information Y1 represent the position of the finger pointing at the point P relative to the target point cloud information.

[0109] Finally, the finger's pointing position P can be obtained based on the third position information X1 and the fourth position information Y1.

[0110] Thus, by using the positional information of the finger keypoint M1 and the shoulder keypoint M2, the projected length of the arm, the pitch angle θ of the arm, and the yaw angle φ of the arm can be determined. Then, based on the shoulder height H, the pitch angle θ, and the yaw angle φ, the position of the finger pointing position P relative to the target point cloud information can be determined. Finally, based on the position of the finger pointing position P relative to the target point cloud information and the target point cloud information, the finger pointing position P is determined. The target point cloud information is determined using the point cloud information of the radar sensor 20. Compared with using only the extrinsic parameters of a monocular camera to determine the actual three-dimensional position corresponding to the image coordinates of the shoulder keypoint M2 on the ground, the accuracy of the target point cloud information corresponding to the shoulder keypoint M2 can be improved, thereby improving the accuracy of the finger pointing position P.

[0111] Please see Figure 4 , Figure 12 and Figure 13 In some implementations, before determining the first position information of the finger keypoint M1 of the target object M and the second position information of the shoulder keypoint M2 of the target object M in the scene image, the cleaning method further includes:

[0112] Step 016: Adjust the position of the cleaning equipment so that the vision sensor 10 is facing the target object M.

[0113] Specifically, before determining the first position information of the finger key point M1 of the target object M and the second position information of the shoulder key point M2 of the target object M in the scene image, the processor 30 needs to adjust the pose of the robot vacuum cleaner 100 so that the visual sensor 10 is facing the target object M, that is, the optical axis of the camera is parallel to the front direction of the target object M.

[0114] For example, when the target object M is 30° to the right of the direction directly facing the vision sensor 10, or when the target object M is 30° to the left of the direction directly facing the vision sensor 10, the robot vacuum cleaner 100 adjusts the posture of its body 50, such as rotating the body 50 30° to the left or 30° to the right, so that the vision sensor 10 on the body 50 is directly facing the target object M. That is, after the vision sensor 10 is turned to the target object M, the field of view of the vision sensor 10 can include the user.

[0115] Optionally, when the robot vacuum cleaner 100 adjusts its posture, it needs to ensure that the entire target object M or the part below the shoulders is within the scene image, so as to ensure that the position information of the finger key point M1 and the shoulder key point M2 can be obtained.

[0116] In this way, by adjusting the pose of the robot vacuum cleaner 100, the visual sensor 10 of the robot vacuum cleaner 100 can be directly facing the target object M, thereby improving the accuracy of the obtained arm projection length in the scene image generated by recognition.

[0117] Optionally, when acquiring a scene image containing the target object M, it is necessary to determine whether the target object M's arm is straight, and if the target object M's arm is straight, the finger pointing position P is determined based on the first position information and the second position information.

[0118] Specifically, after the visual sensor 10 of the robotic vacuum cleaner 100 captures a scene image of the target object M, the processor 30 needs to determine whether the arm of the target object M in the scene image is straight. If it is determined that the arm of the target object M is straight, the processor 30 can obtain accurate first position information and second position information, and thus determine the finger pointing position P based on the first position information and the second position information.

[0119] Therefore, when determining the finger pointing position P based on the position information of the finger key point M1 and the shoulder key point M2, it is necessary to ensure that the target object M's arm is straight, which can improve the accuracy of obtaining the arm projection length, thereby improving the accuracy of obtaining the finger pointing position P.

[0120] Optionally, the method for determining whether the arm of the target object M is straight can be to determine the seventh position information of the elbow key point M3 of the target object M in the scene image. The seventh position information is the fourth image coordinate of the elbow key point M3 in the image coordinate system of the scene image. Based on the first position information, the second position information and the seventh position information, it is possible to determine whether the arm of the target object M is straight.

[0121] Specifically, the processor 30 determines whether the arm of the target object M in the scene image is straight by determining the seventh position information of the elbow keypoint M3 of the target object M in the scene image. It should be noted that the seventh position information can be the fourth image coordinate of the elbow keypoint M3 in the image coordinate system of the scene image. Then, based on the first position information, the second position information, and the seventh position information, the processor 30 can determine whether the arm of the target object M is straight, such as whether the position information of the finger keypoint M1, the elbow keypoint M3, and the shoulder keypoint M2 are collinear in the same plane.

[0122] Thus, by determining whether the positional information of the finger key point M1, elbow key point M3, and shoulder key point M2 of the target object M is collinear in the same plane, it is possible to determine whether the arm is straight.

[0123] Please see Figure 14 In some embodiments, the cleaning method is applied to a cleaning device. Step 012: Determine the first position information of the first key point M1 of the target object M and the second position information of the second key point M2 of the target object M in the scene image, and determine the pointing position P based on the first and second position information, including:

[0124] Step 0121: Based on the preset key point detection model, identify the first position information of the first key point M1 and the second position information of the second key point M2 of the target object M in the scene image. The first position information is the first image coordinate of the first key point M1 in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point M2 in the image coordinate system of the scene image.

[0125] Step 0124: Determine the fifth position information on the ground corresponding to the second position information in the scene image. The fifth position information is the third image coordinate in the image coordinate system of the scene image.

[0126] Step 0125: Based on the extrinsic parameters of the visual sensor 10 of the cleaning equipment, obtain the sixth position information corresponding to the fifth position information. The sixth position information is the world coordinate of the world coordinate system of the cleaning equipment.

[0127] Step 0126: Determine the pointing position P based on the first position information, the second position information, the height of the target part, the length of the target part, and the sixth position information.

[0128] Specifically, after acquiring the scene image generated by the vision sensor 10, the processor 30 transmits the scene image to the key point detection model preset in the robot vacuum cleaner 100, so as to identify the first position information of the finger key point M1 and the second position information of the shoulder key point M2 of the outstretched arm of the target object M in the scene image. It should be noted that an image coordinate system can be established for the scene image in the key point detection model. The first position information of the finger key point M1 can be the first image coordinate in the image coordinate system of the scene image, and the second position information of the shoulder key point M2 can be the second image coordinate in the image coordinate system of the scene image.

[0129] After obtaining the second position information of the shoulder keypoint M2, the fifth position information of the foot of the shoulder keypoint M2 perpendicular to the ground in the scene image can be determined based on the second position information. The fifth position information can be the third image coordinate in the image coordinate system of the scene image.

[0130] Then, based on the extrinsic parameters between the visual sensor 10 and the radar sensor 20, the processor 30 is able to obtain the sixth position information in the world coordinate system of the sweeping robot 100 corresponding to the fifth position information.

[0131] Finally, based on the first position information, the second position information, the shoulder height, the arm length, and the sixth position information, the finger pointing position P can be determined.

[0132] Thus, by identifying the position information of finger keypoint M1 and shoulder keypoint M2 in the scene image through the preset keypoint detection model, the image coordinates of the finger and shoulder in the image coordinate system of the scene image can be obtained. Then, based on the position information of shoulder keypoint M2, the image coordinates of shoulder keypoint M2 on the ground can be determined. Finally, based on the extrinsic parameters of the vision sensor 10, the world coordinates of the image coordinates of shoulder keypoint M2 on the ground in the world coordinate system of the sweeping robot 100 can be determined. Therefore, based on the position information of finger keypoint M1 and shoulder keypoint M2, shoulder height, arm length and world coordinates, the accuracy of finger pointing position P can be improved, and the radar sensor 20 is not required, which can reduce the cost required to purchase the radar sensor 20.

[0133] Please see Figure 9 , Figure 10 and Figure 15In some implementations, step 0126: determining the pointing position P based on the first position information, the second position information, the height of the target part, the length of the target part, and the sixth position information, includes:

[0134] Step 01261: Determine the projected length of the target part and the pitch angle θ of the target part based on the first position information and the second position information;

[0135] Step 01262: Determine the yaw angle φ of the target part based on the projected length of the target part and the length of the target part;

[0136] Step 01263: Determine the third position information based on the height and pitch angle of the target part, and determine the fourth position information based on the height H, pitch angle θ and yaw angle φ of the target part. The third and fourth position information are the positions of the finger pointing position P relative to the sixth position information.

[0137] Step 01264: Determine the pointing position P based on the third position information, the fourth position information, and the sixth position information.

[0138] Specifically, after acquiring the first position information of the finger key point M1 and the second position information of the shoulder key point M2, the processor 30 can obtain the coordinates of the finger key point M1 and the shoulder key point M2, thereby calculating the distance between the finger key point M1 and the shoulder key point M2, and then obtaining the projection length Z1 of the arm. Based on the projection length x of the arm on the X-axis and the projection length y of the arm on the Y-axis, the pitch angle θ of the arm can be calculated through trigonometric functions.

[0139] Then, based on the length Z1 of the arm projection and the length L of the arm, the yaw angle φ of the arm can be calculated using trigonometric relationships.

[0140] Next, the processor 30 determines the third position information X2 based on the shoulder height H and pitch angle θ using trigonometric functions, and determines the fourth position information Y2 based on the shoulder height H, pitch angle θ, and yaw angle φ using trigonometric functions. It should be noted that the third position information X2 and the fourth position information Y2 represent the positions of the finger pointing position P relative to the sixth position information.

[0141] Finally, the finger's pointing position P can be obtained based on the third position information X2 and the fourth position information Y2.

[0142] Thus, by using the extrinsic parameters of the camera to identify the image coordinates of the shoulder key point M2, the sixth position information can be obtained by mapping the image coordinates based on the extrinsic parameters. Then, the finger pointing position is calculated based on the sixth position information. Compared with obtaining target point cloud information through the point cloud of radar sensor 20, the calculation is simpler and does not require radar sensor 20, resulting in lower cost.

[0143] Please see Figure 16 In some embodiments, the cleaning method is applied to a cleaning device, whose radar sensor 20 can collect point cloud information of objects within a second preset height. The second preset height is greater than or equal to the height of a preset target part of the target object M. Step 012: Determine the first position information of the first key point M1 of the target object M and the second position information of the second key point M2 of the target object M in the scene image, and determine the finger pointing position P based on the first and second position information, including:

[0144] Step 0121: Based on the preset key point detection model, identify the first position information of the first key point M1 and the second position information of the second key point M2 in the scene image. The first position information is the first image coordinate of the first key point M1 in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point M2 in the image coordinate system of the scene image.

[0145] Step 0127: Based on the preset calibration parameters of the visual sensor 10 and radar sensor 20 of the cleaning equipment and the first position information, determine the first point cloud information corresponding to the first position information, and based on the preset calibration parameters and the second position information, determine the second point cloud information corresponding to the second position information;

[0146] Step 0128: Determine the pointing position P based on the first point cloud information and the second point cloud information.

[0147] Specifically, the processor 30 controls the radar sensor 20 of the robotic vacuum cleaner 100 to collect point cloud information of objects within a preset second preset height (e.g., 170 cm, 180 cm, etc.) of the radar sensor 20. The second preset height must be greater than or equal to the shoulder height of the target object M.

[0148] After acquiring the scene image generated by the vision sensor 10, the processor 30 transmits the scene image to the key point detection model preset in the robot vacuum cleaner 100, so as to identify the first position information of the finger key point M1 and the second position information of the shoulder key point M2 of the outstretched arm of the target object M in the scene image. It should be noted that an image coordinate system can be established for the scene image in the key point detection model. The first position information of the finger key point M1 can be the first image coordinate in the image coordinate system of the scene image, and the second position information of the shoulder key point M2 can be the second image coordinate in the image coordinate system of the scene image.

[0149] Then, the processor 30 will acquire the first position information of the finger key point M1 and the second position information of the shoulder key point M2. Based on the extrinsic parameters between the vision sensor 10 and the radar sensor 20, it can determine the first point cloud information corresponding to the first position information and the second point cloud information corresponding to the second position information. It should be noted that the resolutions of the radar sensor 20 and the vision sensor 10 may be different. Generally, the resolution of the vision sensor 10 is larger than that of the radar sensor 20. Therefore, when determining the first point cloud information corresponding to the first position information and the second point cloud information corresponding to the second position information, the point cloud set may lack corresponding point cloud information. Thus, the first point cloud information corresponding to the first position information and the second point cloud information corresponding to the second position information need to be obtained from the point cloud closest to the first point cloud information and the second point cloud information.

[0150] Finally, the processor 30 can determine the finger's pointing position P based on the determined first point cloud information and second point cloud information. For example, the processor 30 can determine the finger's pointing position P by using the intersection of the line connecting the first and second point cloud coordinates and the ground.

[0151] Thus, when the preset height of the radar sensor 20 is greater than the height of the shoulder, the position information of the finger key point M1 and the shoulder key point M2 in the scene image can be identified by the preset key point detection model. The image coordinates of the finger key point M1 and the shoulder key point M2 in the image coordinate system of the scene image can be obtained. Then, through the preset calibration parameters between the vision sensor 10 and the radar sensor 20, the point cloud information corresponding to the position information of the finger key point M1 and the shoulder key point M2 can be obtained. Based on the point cloud information of the finger key point M1 and the shoulder key point M2, such as the intersection of the lines connecting the corresponding point clouds of the finger key point M1 and the shoulder key point M2 on the ground, the finger pointing position P can be used as the finger pointing position P. Thus, the finger pointing position P can be accurately and quickly determined. Moreover, when determining the finger pointing position P, compared with the radar sensor 20 being lower than the shoulder height for shooting the target object M, it is not necessary to adjust the posture of the vision sensor 10 so that the target object M is facing the vision sensor 10, which can improve the efficiency of obtaining the finger pointing position P.

[0152] Please see Figure 17 and Figure 18 The cleaning demonstration method in this application includes:

[0153] Step 021: Display the current position and posture of the cleaning equipment on the map, as well as the pointing position P of the target object M in the visual sensor 10 of the cleaning equipment. The current position and posture are determined based on the current pose information of the cleaning equipment. The pointing position P is determined based on the first position information of the first key point M1 of the target object M and the second position information of the second key point M2 of the target object M in the scene image collected by the visual sensor 10. The first key point M1 and the second key point M2 are located at the target part of the target object M.

[0154] The entity executing the cleaning demonstration method is a computer program product corresponding to the robot vacuum cleaner 100. The computer program product can be set in the terminal 300 or the base station 200; for example, the computer program product is an application installed in the terminal 300 or the base station 200.

[0155] For example, when a computer program runs on base station 200 to implement a cleaning demonstration method, the sweeping robot 100 can transmit information such as its current position and posture on the map, and the finger pointing position P of the target object M in the vision sensor 10 of the sweeping robot 100 to base station 200. After processing the information, base station 200 can display the information such as the current position and posture of the sweeping robot 100 on the map, and the finger pointing position P of the target object M in the vision sensor 10 of the sweeping robot 100 on the display screen of base station 200.

[0156] For example, when a computer program runs on terminal 300 to implement a cleaning demonstration method, the robot vacuum cleaner 100 can upload information such as its current position and posture on the map, and the finger pointing position P of the target object M in the vision sensor 10 of the robot vacuum cleaner 100 to server 400. Then, terminal 300 obtains information from server 400 to display information such as the current position and posture of the robot vacuum cleaner 100 on the map, and the finger pointing position P of the target object M in the vision sensor 10 of the robot vacuum cleaner 100.

[0157] Specifically, the robot vacuum cleaner 100 is equipped with a wireless communication module, which enables it to communicate wirelessly with the user's terminal 300 (such as via Bluetooth or Wi-Fi). The user can then view the cleaning process of the robot vacuum cleaner 100 on the display screen of the terminal 300.

[0158] Based on the current pose information of the robotic vacuum cleaner 100, the current position of the robotic vacuum cleaner 100 on the map, the current posture of the robotic vacuum cleaner 100, and the finger pointing position P of the target object M in the visual sensor 10 of the robotic vacuum cleaner 100 can be displayed on the user's terminal 300 display screen.

[0159] For example, the robot vacuum cleaner 100 can communicate with the user's smartphone via Bluetooth, and the user can see the current location of the robot vacuum cleaner 100 on the map on the smartphone screen; after the robot vacuum cleaner 100 receives a voice call, the smartphone screen can display the current location of the robot vacuum cleaner 100, the robot vacuum cleaner 100's posture (such as the robot vacuum cleaner 100's posture being directly in front of the user, or the robot vacuum cleaner 100's posture being deviated from the user's) and the position P pointed to by the user's finger.

[0160] Thus, by displaying the current position and posture of the robot vacuum cleaner 100 on the map, as well as the finger pointing position P of the target object M in the visual sensor 10 of the robot vacuum cleaner 100, the position of the robot vacuum cleaner 100 and the finger pointing position P, the posture of the robot vacuum cleaner 100 and the relative position of the target object M can be clearly understood through the display screen.

[0161] Optionally, the real-time cleaning path of the robotic vacuum cleaner 100 as it cleans the area to be cleaned is displayed, the area to be cleaned being determined based on the finger pointing to the location P. The processor 30 can determine the area to be cleaned based on the finger pointing to the location P of the target object M, and after controlling the robotic vacuum cleaner 100 to move to the area to be cleaned, the terminal 300 communicating with the robotic vacuum cleaner 100 can display the planned cleaning path of the robotic vacuum cleaner 100 moving in the area to be cleaned, as well as the real-time cleaning path during cleaning, on its display screen.

[0162] For example, by recognizing the position P where the target object M is pointing, the processor 30 can determine that the area to be cleaned is a rectangular area symmetrically centered on the sound source object, and display the planned cleaning path of the robot vacuum cleaner 100 in the rectangular area (e.g., the planned cleaning path is in a bow shape) on the smartphone display. When the robot vacuum cleaner 100 starts cleaning along the planned cleaning path, the smartphone display can show the real-time cleaning path of the robot vacuum cleaner 100, that is, the user can see the area that the robot vacuum cleaner 100 has already cleaned.

[0163] In this way, by displaying the real-time cleaning path of the robot vacuum cleaner 100 as it cleans the area to be cleaned on the screen, users can clearly know the cleaning progress of the robot vacuum cleaner 100.

[0164] Please refer to it again. Figure 3 The cleaning device of this application includes a processor 30, a memory 40 and a computer program 41. The computer program 41 is stored in the memory 40 and executed by the processor 30. The computer program 41 includes instructions for performing the cleaning method of any of the above embodiments. For the sake of brevity, it will not be described in detail here.

[0165] In the description of this specification, the references to terms such as "some embodiments," "in one example," "exemplarily," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0166] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0167] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A cleaning method, characterized in that, The method, applied to cleaning equipment, wherein the radar sensor of the cleaning equipment is configured to collect point cloud information of objects within a first preset height, wherein the first preset height is less than the height of the target location, includes: Get the scene image containing the target object; Based on a preset key point detection model, the first position information and the second position information of the target object in the scene image are identified. The first position information is the first image coordinate of the first key point in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point in the image coordinate system of the scene image. The first key point and the second key point are located at the target part of the target object. Determine the target region where the second key point is located in the scene image, wherein the target region includes one or more columns of pixels; Based on the preset calibration parameters of the visual sensor and radar sensor of the cleaning equipment, the set of point cloud information located in the target area is obtained from all the point cloud information collected by the radar sensor. Based on the point cloud information set, determine the target point cloud information corresponding to the second key point; The projected length and pitch angle of the target part are determined based on the first position information and the second position information. The yaw angle of the target part is determined based on the projected length of the target part and the length of the target part. The third position information is determined based on the height of the target part and the pitch angle, and the fourth position information is determined based on the height of the target part, the pitch angle and the yaw angle. The third position information and the fourth position information are the positions of the pointing positions relative to the target point cloud information. The pointing location is determined based on the third location information, the fourth location information, and the target point cloud information; Move to the indicated position to clean the area to be cleaned corresponding to the indicated position.

2. The cleaning method according to claim 1, characterized in that, Also includes: Identify the current pose of the target object; Based on the current posture, determine the first key point and the second key point in the target part that match the current posture.

3. The cleaning method according to claim 1, characterized in that, Before determining the first position information of the first key point of the target object and the second position information of the second key point of the target object in the scene image, the method further includes: Adjust the position of the cleaning device so that the vision sensor is facing the target object.

4. The cleaning method according to claim 1, characterized in that, Applied to cleaning equipment, the step of determining the first position information of a first key point of the target object and the second position information of a second key point of the target object in the scene image, and determining the pointing position based on the first position information and the second position information, includes: Based on a preset key point detection model, the first position information of the first key point of the target object and the second position information of the second key point in the scene image are identified. The first position information is the first image coordinate of the first key point in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point in the image coordinate system of the scene image. In the scene image, the fifth position information corresponding to the second position information on the ground is determined, and the fifth position information is the third image coordinate in the image coordinate system of the scene image; Based on the extrinsic parameters of the visual sensor of the cleaning equipment, a sixth position information corresponding to the fifth position information is obtained, and the sixth position information is the world coordinate of the world coordinate system of the cleaning equipment. The pointing position is determined based on the first position information, the second position information, the height of the target part, the length of the target part, and the sixth position information.

5. The cleaning method according to claim 4, characterized in that, Determining the pointing position based on the first position information, the second position information, the height of the target part, the length of the target part, and the sixth position information includes: The projected length and pitch angle of the target part are determined based on the first position information and the second position information. The yaw angle of the target part is determined based on the projected length of the target part and the length of the target part. The third position information is determined based on the height of the target part and the pitch angle, and the fourth position information is determined based on the height of the target part, the pitch angle and the yaw angle. The third position information and the fourth position information are the positions of the pointing position relative to the sixth position information. According to the third location information, the No. The pointing position is determined by the four position information and the sixth position information.

6. The cleaning method according to claim 1, characterized in that, Applied to cleaning equipment, the radar sensor of the cleaning equipment is configured to collect point cloud information of objects within a second preset height, the second preset height being greater than or equal to the height of a preset target part of the target object. The step of determining the first position information of a first key point of the target object and the second position information of a second key point of the target object in the scene image, and determining the finger pointing position based on the first position information and the second position information, includes: Based on a preset key point detection model, the first position information of the first key point and the second position information of the second key point in the scene image are identified. The first position information is the first image coordinate of the first key point in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point in the image coordinate system of the scene image. Based on the preset calibration parameters of the visual sensor and radar sensor of the cleaning equipment and the first position information, the first point cloud information corresponding to the first position information is determined, and based on the preset calibration parameters and the second position information, the second point cloud information corresponding to the second position information is determined. The pointing location is determined based on the first point cloud information and the second point cloud information.

7. A cleaning display method, characterized in that, include: The display shows the current position and orientation of the cleaning equipment in the map corresponding to the scene image, as well as the pointing position of the target object of the cleaning equipment. The current position and orientation are determined based on the current pose information of the cleaning equipment. The pointing position is determined according to the following steps: Get the scene image containing the target object; Based on a preset key point detection model, the first position information and the second position information of the target object in the scene image are identified. The first position information is the first image coordinate of the first key point in the image coordinate system of the scene image, and the second position information is the second image coordinate of the second key point in the image coordinate system of the scene image. The first key point and the second key point are located at the target part of the target object. Determine the target region where the second key point is located in the scene image, wherein the target region includes one or more columns of pixels; Based on preset calibration parameters, the point cloud information set located within the target area is obtained from the point cloud information of objects within a first preset height collected by the radar sensor of the cleaning equipment. Based on the point cloud information set, determine the target point cloud information corresponding to the second key point; The projected length of the target part and the pitch angle of the target part are determined based on the first position information and the second position information. The yaw angle of the target part is determined based on the projected length of the target part and the length of the target part. The third position information is determined based on the height of the target part and the pitch angle, and the fourth position information is determined based on the height of the target part, the pitch angle and the yaw angle. The third position information and the fourth position information are the positions of the pointing position relative to the target point cloud information. The pointing location is determined based on the third location information, the fourth location information, and the target point cloud information.

8. A cleaning device, characterized in that, include: Processor, memory; and A computer program, wherein the computer program is stored in the memory and executed by the processor, the computer program comprising instructions for performing the cleaning method according to any one of claims 1 to 6.

9. A computer program product, characterized in that, It includes a computer program, the computer program comprising instructions for performing the cleaning method according to any one of claims 1 to 6 and the cleaning demonstration method according to claim 7.

Images