Floor sweeping robot and control method, device, equipment and product

Abstract

The invention provides a sweeping robot and a control method, device, equipment and product, and the method comprises the steps: obtaining an RGB image, a depth camera point cloud and a laser radar point cloud of a user in response to a sweeping instruction issued by a user through limb movement; according to the RGB image and the depth camera point cloud, determining a comparison result of a spacing distance between the user and the sweeping robot and a preset distance; and according to the comparison result, selecting the depth camera point cloud or the laser radar point cloud, and generating a sweeping control instruction for the sweeping robot. The method can enlarge the recognition range of the target region boundary, and improves the recognition precision of the target region boundary.

Description

Technical Field

[0001] The present application relates to the field of intelligent robots, and in particular to a sweeping robot and control methods, devices, equipment and products. Background Art

[0002] With the development of artificial intelligence technology, sweeping robots have become common equipment in daily life. By automating cleaning tasks, sweeping robots can significantly reduce the user's housework burden.

[0003] At present, sweeping robots can realize local cleaning tasks by identifying the target area indicated by the user, such as cleaning only a small specified area. However, the current target area recognition technology has limitations, especially when the target area is large, the recognition accuracy is low or even cannot be recognized, resulting in low cleaning efficiency. Summary of the invention

[0004] Based on the above-mentioned technical status, the present application provides a sweeping robot and a control method, device, equipment and product, which can expand the recognition range of the target area, improve the recognition accuracy of the target area, and thus improve the cleaning efficiency.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a control method for a sweeping robot is provided, comprising: in response to a cleaning instruction issued by a user through body movements, obtaining an RGB image, a depth camera point cloud, and a lidar point cloud including the user; determining, based on the RGB image and the depth camera point cloud, an interval distance between the user and the sweeping robot, and a comparison result between the interval distance and a preset distance; and, based on the comparison result between the interval distance and the preset distance, selecting to generate a cleaning control instruction for the sweeping robot based on the depth camera point cloud or the lidar point cloud.

[0007] In some implementations, determining the interval distance between the user and the sweeping robot based on the RGB image and the depth camera point cloud, and the comparison result between the interval distance and the preset distance includes: performing semantic segmentation based on the RGB image to obtain a human body mask image of the user; determining the depth camera point cloud located on the human body mask image based on the user's human body mask image and a mapping relationship between a depth camera coordinate system and an image coordinate system; and determining the comparison result between the interval distance and the preset distance based on the number of depth camera point clouds located on the human body mask image.

[0008] In some implementations, determining the comparison result between the interval distance and the preset distance based on the number of depth camera point clouds located on the human body mask image includes: if the number of depth camera point clouds located on the human body mask image is less than or equal to the preset number, determining that the interval distance is greater than the preset distance; if the number of depth camera point clouds located on the human body mask image is greater than the preset number, determining that the interval distance is less than or equal to the preset distance.

[0009] In some implementations, the determination of the interval distance between the user and the sweeping robot based on the RGB image and the depth camera point cloud, and the comparison result with the preset distance include: determining the user's body center of mass coordinates based on the RGB image, the depth camera point cloud, and the mapping relationship between the depth camera coordinate system and the image coordinate system, the body center of mass coordinates including z-axis values; comparing the z-axis values ​​in the body center of mass coordinates with the preset distance to obtain a comparison result between the interval distance and the preset distance.

[0010] In some implementations, determining the user's body center of mass coordinates based on the depth camera point cloud and the mapping relationship between the depth camera coordinate system and the image coordinate system includes: performing semantic segmentation based on the RGB image to obtain a body mask image of the user; determining a depth camera point cloud located on the body mask based on the user's body mask image and the mapping relationship between the depth camera coordinate system and the image coordinate system; determining the user's body center of mass coordinates based on the depth camera point cloud located on the body mask image.

[0011] In some implementations, based on the comparison result between the interval distance and the preset distance, the cleaning control instructions for the cleaning robot are generated based on the depth camera point cloud or the lidar point cloud, including: if the interval distance is less than or equal to the preset distance, the cleaning control instructions for the cleaning robot are generated based on the depth camera point cloud; if the interval distance is greater than the preset distance, the cleaning control instructions for the cleaning robot are generated based on the lidar point cloud.

[0012] In some implementations, generating cleaning control instructions for the sweeping robot based on the depth camera point cloud includes: determining the depth camera point cloud located at the user's limb joints based on the user's limb joints in the RGB image and the mapping relationship between the depth camera coordinate system and the image coordinate system; determining the cleaning action pointing point corresponding to the user's limb movement based on the depth camera point cloud located at the user's limb joints; and generating cleaning control instructions for the sweeping robot based on the cleaning action pointing point.

[0013] In some implementations, the limb joints include large joints and wrist joints, and determining the sweeping action pointing point corresponding to the user's limb movement based on the depth camera point cloud on the user's limb joints includes: determining the movement state of the wrist joint on the ground based on the depth camera point cloud on the user's wrist joint; determining the limb part corresponding to the user's limb movement based on the movement state of the wrist joint on the ground; connecting the depth camera point cloud coordinates corresponding to the large joints and wrist joints corresponding to the limb part and extending them to intersect with the ground, and determining the intersection point as the sweeping action pointing point corresponding to the user's limb movement.

[0014] In some implementations, the movement state of the wrist joint point on the ground includes the first wrist or the second wrist of the user moving on the ground, or the first wrist and the second wrist of the user both moving on the ground; wherein, determining the limb part corresponding to the user's limb movement based on the movement state of the wrist joint point on the ground includes: if the first wrist or the second wrist of the user moves, determining the limb part to which the first wrist or the second wrist belongs as the limb part corresponding to the user's limb movement; if both the first wrist and the second wrist move, determining the distance between each of the wrist joint points of the first wrist and the second wrist and the user's body center of mass, and selecting the limb part to which the wrist corresponding to the maximum distance belongs as the limb part corresponding to the user's limb movement.

[0015] In some implementations, the limb joints include wrist joints, and generating cleaning control instructions for the sweeping robot based on the lidar point cloud includes: determining the lidar point cloud located on the wrist joint of the user based on the wrist joint of the user in the RGB image and the mapping relationship between the lidar coordinate system and the image coordinate system; determining the cleaning action pointing point corresponding to the user's limb movement based on the lidar point cloud located on the wrist joint of the user; and generating cleaning control instructions for the sweeping robot based on the cleaning action pointing point.

[0016] In some implementations, determining the direction point of the sweeping action of the user's limb movement based on the laser radar point cloud located on the wrist joint of the user includes: determining the moving state of the wrist joint on the ground based on the laser radar point cloud on the user's wrist joint; determining the limb part corresponding to the user's limb movement based on the moving state of the wrist joint on the ground; determining the sweeping action direction point corresponding to the limb movement based on the laser radar point cloud on the limb part corresponding to the user's limb movement.

[0017] According to the second aspect of an embodiment of the present application, a control device for a sweeping robot is provided, comprising: an acquisition unit, for acquiring, in response to a cleaning instruction issued by a user through body movements, an RGB image, a depth camera point cloud, and a lidar point cloud of the user; a determination unit, for determining, based on the RGB image and the depth camera point cloud, an interval distance between the user and the sweeping robot, and a comparison result between the interval distance and a preset distance; and a generation unit, for selecting, based on the comparison result between the interval distance and the preset distance, either the depth camera point cloud or the lidar point cloud to generate a cleaning control instruction for the sweeping robot.

[0018] According to a third aspect of an embodiment of the present application, a control device of a sweeping robot is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the control method of the sweeping robot as described in the first aspect by running the program in the memory.

[0019] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the control method of the sweeping robot as described in the first aspect is implemented.

[0020] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, cause the processor to execute: the control method of the sweeping robot as described in the first aspect.

[0021] According to a sixth aspect of the embodiments of the present application, a sweeping robot is provided, comprising: a sweeping robot control device as described in the third aspect, and a sweeping robot body.

[0022] The embodiment of the present application provides a sweeping robot and control method, device, equipment and product. When receiving a cleaning instruction issued by a user through body movements, an RGB image, a depth camera point cloud and a laser radar point cloud including the user are obtained, and the distance between the user and the sweeping robot is determined based on the RGB image and the depth camera point cloud, and the comparison result between the distance and the preset distance, and the depth camera point cloud or the laser radar point cloud is selected according to the comparison result to generate a cleaning control instruction for the sweeping robot. Among them, the user issues a cleaning instruction through simple body movements, which can improve the interaction experience between the user and the robot. In addition, since different methods can be selected to generate cleaning control instructions according to different comparison results, no matter whether the human body is close or far from the robot, it can be conveniently controlled by simple body movements to clean the designated area. This can not only expand the recognition range of the target area, improve the recognition accuracy of the target area, but also further improve the cleaning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0024] Figure 1 A flow chart of a control method of a sweeping robot provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of an implementation method of step S102 provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of semantic segmentation provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of determining a depth camera point cloud located on a human body mask image provided by an embodiment of the present application;

[0028] Figure 5 A flowchart of another implementation method of step S102 provided in an embodiment of the present application;

[0029] Figure 6 A flowchart of generating cleaning control instructions for a sweeping robot according to a depth camera point cloud provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of determining the direction point of a cleaning action according to a limb joint point provided in an embodiment of the present application;

[0031] Figure 8 A flowchart of generating a cleaning control instruction according to a laser radar point cloud provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of the structure of a control device for a sweeping robot provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] Before introducing this application solution, the relevant technologies are first introduced:

[0037] Sweeping robots have become a standard feature of many households. When the user issues a cleaning command, the robot can automatically clean the entire house, freeing up the user's time and energy, significantly reducing the user's housework burden and improving their quality of life.

[0038] However, existing sweeping robots have obvious deficiencies in terms of user interaction. Most sweeping robots have relatively simple functions and lack the ability to flexibly respond to user commands. For example, when users want the robot to clean only a small area instead of the entire room, existing robots often cannot meet this demand well. This is mainly reflected in the following aspects:

[0039] When the distance between the boundary of the target area and the robot exceeds a certain range, the robot's recognition accuracy will drop significantly. For example, if the boundary of the target area is far away from the robot, the robot may not be able to accurately recognize the boundary of the area, causing the robot to miss certain areas when cleaning a specific area.

[0040] In summary, existing sweeping robots have obvious deficiencies in target area recognition, which limits their application effect in household cleaning. Therefore, developing sweeping robots with higher target area recognition accuracy has become an urgent problem to be solved. This can not only improve the user experience, but also further improve the intelligence level of household robots.

[0041] In view of this, the embodiments of the present application are dedicated to providing a cleaning robot and a control method, device, equipment and product, by determining the interval distance between the robot and the boundary of the target area indicated by the user, and comparing the result with the preset distance, selecting different ways to generate cleaning control instructions, and then controlling the robot to clean the target area, thereby expanding the recognition range of the boundary of the target area and improving the recognition accuracy of the boundary of the target area. The following embodiments are described in detail one by one.

[0042] Exemplary Methods

[0043] Figure 1 Flow chart of a control method of a robot sweeper provided in an embodiment of the present application. Figure 1 As shown, the control method of the sweeping robot provided in this embodiment includes steps S101-S103:

[0044] S101. In response to a cleaning instruction issued by a user through body movements, an RGB image, a depth camera point cloud, and a lidar point cloud of the user are obtained.

[0045] When the user wants to issue cleaning instructions to the cleaning robot through limbs (such as toes and fingers), the user needs to make predefined body movements within the robot's field of view (such as marking the boundary range of the target area on the ground with toes or fingers). At this time, the time-series RGB camera installed on the robot will capture the user's body contour and generate a high-resolution color image, namely an RGB image; the depth camera (such as the iToF depth camera) measures the actual distance from each pixel to the RGB camera by emitting and receiving infrared light or laser pulses, and constructs three-dimensional point cloud data of the user and its surrounding environment, hereinafter referred to as the depth camera point cloud depth camera. Depth information plays an important role in determining the specific position of the user relative to the robot. At the same time, LiDAR uses laser beam scanning technology to further enhance the understanding of environmental structure, especially the precise position determination of distant objects, thereby obtaining a LiDAR point cloud.

[0046] When the user draws the boundary range of the target area on the ground with the toes or fingers, the toes or fingers may or may not be in contact with the ground. When the toes or fingers of the user are in contact with the ground, the user can draw a circular or rectangular area on the ground with the toes. When the toes or fingers of the user are not in contact with the ground, the user can draw a virtual area boundary in the air with the toes or fingers.

[0047] S102: Determine the interval distance between the user and the sweeping robot according to the RGB image and the depth camera point cloud, and compare the interval distance with the preset distance.

[0048] In step S102, the cleaning robot aims to accurately measure the distance between the user and the cleaning robot by combining the RGB image and the depth camera point cloud data, and compare the distance with the preset operation distance threshold to determine the subsequent user command recognition method.

[0049] There are many ways to implement step S102, which can be specifically as follows:

[0050] Figure 2 This is a flowchart of an implementation method of step S102 provided in an embodiment of the present application. Figure 2 As shown, step S102 includes the following steps S201 to S203:

[0051] S201. Perform semantic segmentation based on the RGB image to obtain a human body mask image of the user.

[0052] Figure 3 This is a schematic diagram of semantic segmentation provided in the embodiment of the present application. Figure 3 As shown in the figure, the RGB image is input into the trained semantic segmentation model, and the semantic segmentation model can perform semantic segmentation based on the RGB image to obtain the user's human body mask image. The semantic segmentation model can distinguish objects of different categories in the image and mark the position of the human body. Its output is a binary image (i.e., human body mask image), in which the part belonging to the human body is marked as white (or 1), and the part not belonging to the human body is marked as black (or 0).

[0053] S202: Determine a depth camera point cloud located on the human body mask image according to the human body mask image of the user and a mapping relationship between the depth camera coordinate system and the image coordinate system.

[0054] The user's human body mask image provides image data of the user's body contour in the image coordinate system (i.e., the RGB camera coordinate system). Through the mapping relationship between the depth camera coordinate system and the image coordinate system, the depth camera point cloud collected in step S101 can be converted to the RGB camera perspective, so as to facilitate the screening of the depth camera point cloud located on the human body mask image. These depth camera point clouds represent the depth information of the user's body part and can be used to calculate the actual distance between the user and the robot.

[0055] Figure 4 A schematic diagram of determining a depth camera point cloud located on a human body mask image provided by an embodiment of the present application. Figure 4 As shown, by mapping the depth camera point cloud to the RGB camera perspective, a mapping image can be obtained, and the gray dots in the mapping image are the mapped depth camera point cloud.

[0056] Furthermore, by combining the human body mask image, the depth camera point cloud located on the human body mask can be filtered out. The gray dots in the figure are the depth camera point clouds located on the human body mask.

[0057] Specifically, the depth camera point cloud collected in step S101 can be converted to the RGB camera perspective through the conversion matrix between the depth camera coordinate system and the image coordinate system (usually including the rigid transformation matrix H and the RGB camera intrinsic parameter matrix M). The specific formula is as follows:

[0058]

[0059] In the formula, H=[R 3*3 T 3*1 ]; γ is the scale factor; the matrix of the equal sign coordinates is the representation of the RGB camera coordinate system; the matrix on the right side of the equal sign is the representation of the depth camera coordinate system.

[0060] By converting the depth camera point cloud collected in step S101 to the RGB camera perspective, the depth camera point cloud set falling on the human body mask can be screened out, denoted as S human_mask ={(x1,y1,z1),(x2,y2,z2)…(x n ,y n ,z n )}.

[0061] S203: Determine a comparison result between the interval distance and a preset distance according to the number of depth camera point clouds located on the human body mask image.

[0062] The distance between the user and the robot is negatively correlated with the number of depth camera point clouds. Specifically, the closer the user is to the robot, the more depth camera point clouds there are; conversely, the farther the user is from the robot, the fewer depth camera point clouds there are. Therefore, the comparison result between the distance and the preset distance can be determined by counting the number of depth camera point clouds located on the human mask image.

[0063] Specifically, step S203 includes: if the number of depth camera point clouds located on the human body mask image is less than or equal to a preset number, determining that the interval distance is greater than a preset distance; if the number of depth camera point clouds located on the human body mask image is greater than a preset number, determining that the interval distance is less than or equal to a preset distance.

[0064] The preset number can be 0. That is, if the depth camera point cloud set S located on the human body mask human_mask If the depth camera point cloud set S located on the human body mask is empty, it is determined that the interval distance is greater than the preset distance; human_mask is a non-empty set, it is determined that the interval distance is less than or equal to the preset distance.

[0065] Figure 5 FIG. 1 is a flowchart of another implementation method of step S102 provided in an embodiment of the present application. Figure 5 As shown, step S102 includes the following steps S501 and S502:

[0066] S501. Determine the user's body center of mass coordinates according to the RGB image, the depth camera point cloud, and the mapping relationship between the depth camera coordinate system and the image coordinate system, where the body center of mass coordinates include a z-axis value.

[0067] In step S501, the sweeping robot determines the coordinates of the user's body center of mass by combining the RGB image, the depth camera point cloud, and the mapping relationship between the depth camera coordinate system and the image coordinate system. The coordinates of the body center of mass include the z-axis value, which is used to determine the distance between the user and the robot. The specific implementation process includes the following steps a1 to a3:

[0068] Step a1: perform semantic segmentation based on the RGB image to obtain the user's body mask image.

[0069] Among them, the implementation method of step a1 is similar to the implementation method of step S201. For details, please refer to the introduction of the implementation method of step S201, which will not be repeated here.

[0070] Step a2: determine the depth camera point cloud located on the human body mask image according to the user's human body mask image and the mapping relationship between the depth camera coordinate system and the image coordinate system.

[0071] Among them, the implementation method of step a2 is similar to the implementation method of step S202. For details, please refer to the introduction of the implementation method of step S202, which will not be repeated here.

[0072] Step a3: Determine the coordinates of the user's body center of mass based on the depth camera point cloud located on the body mask image.

[0073] Specifically, by performing averaging operations on the coordinate values ​​of the depth camera point cloud falling on the human body mask on the X, Y, and Z axes, the coordinates of the user's human body center of mass can be obtained. The specific formula for the averaging operation is as follows:

[0074] human_center xyz =(x human_center ,y human_center ,z human_center );

[0075] The averaging operation can be expressed as the following formula:

[0076] Where n is the number of depth camera point clouds located on the human mask, and (xi, yi, zi) is the coordinate of the i-th point.

[0077] After obtaining the user's body center of mass coordinates, the z-axis value z in the body center of mass coordinates can be human_center Determined as the separation distance between the user and the robot.

[0078] S502: Compare the z-axis value in the human body center of mass coordinates with the preset distance to obtain a comparison result between the interval distance and the preset distance.

[0079] In step S502, the cleaning robot determines the distance between the user and the robot by comparing the z-axis value in the human body mass center coordinates with the preset distance, and selects an appropriate user command recognition mode accordingly. The specific implementation process is as follows:

[0080] Through the above step S501, the user's body center of mass coordinates (x human_center ,y humax_center ,z human_center ). The z-axis value z in the human body center of mass coordinates humax_center Represents the distance between the user and the robot.

[0081] By comparing the interval distance with a preset operation distance threshold, a comparison result between the two can be obtained, and then according to different comparison results, a close distance interaction mode or a long distance interaction mode is selected to identify the user's cleaning instructions.

[0082] Continue reading Figure 1 , the control method of the sweeping robot in this embodiment further includes step S103 after step S102.

[0083] S103: According to the comparison result between the interval distance and the preset distance, select to generate a cleaning control instruction for the sweeping robot according to the depth camera point cloud or the laser radar point cloud.

[0084] In step S103, a method for generating a cleaning control instruction for the cleaning robot is selected according to the comparison result between the interval distance and the preset distance. Specifically, step S103 includes: if the interval distance is less than or equal to the preset distance, generating a cleaning control instruction for the cleaning robot according to the depth camera point cloud; if the interval distance is greater than the preset distance, generating a cleaning control instruction for the cleaning robot according to the laser radar point cloud.

[0085] Specifically, when the human body is within 3 meters of the robot, the number of points on the robot's depth camera point cloud that fall on the human mask can be used to calculate the 3D coordinates of the human joints for subsequent cleaning point calculations. However, when the human body is more than 3 meters away from the robot, the depth camera point cloud that falls on the human mask is very small, and the ability to obtain depth information of distant objects will be weakened or even ineffective. However, the signal capability of the lidar is quite strong and can capture depth information of objects within 5 meters. Therefore, when the human body is far away from the robot, the lidar point cloud on the robot can be used to determine the approximate position of the human body, and finally the final cleaning point is calculated based on the ankle lidar point cloud pointing to the foot.

[0086] Figure 6 A flowchart of generating cleaning control instructions for a sweeping robot according to a depth camera point cloud provided in an embodiment of the present application. Figure 6 As shown, generating a cleaning control instruction for a cleaning robot according to the depth camera point cloud includes the following steps S601 to S603:

[0087] S601, determining a depth camera point cloud located at the user's limb joints according to the user's limb joints in the RGB image and a mapping relationship between a depth camera coordinate system and an image coordinate system.

[0088] Continue reading Figure 3 , the RGB image is input into the semantic segmentation model, and the semantic segmentation model is used to detect the joint points according to the RGB image, so as to obtain the user's limb joint points. The gray dots in the figure are the limb joint points. The user's limbs can be legs or arms. When the limbs are legs, the limb joint points include knee joint points and ankle joint points; when the limbs are arms, the limb joint points include elbow joint points and wrist joint points.

[0089] The limb joint points output by the model are image data in the image coordinate system (i.e., RGB camera coordinate system). Through the mapping relationship between the depth camera coordinate system and the image coordinate system, the depth camera point cloud of the limb joint points output by the model is converted to the RGB camera perspective, and the depth camera point cloud located on the human limb joint points can be screened out.

[0090] The coordinate system conversion process between the depth camera coordinate system and the image coordinate system can be referred to the introduction in the aforementioned step S202, which will not be repeated here.

[0091] S602: Determine, based on the depth camera point cloud located at the user's limb joints, a cleaning action pointing point corresponding to the user's limb action.

[0092] Due to different personal habits of users, they may use their left foot or right foot to issue a cleaning command, or use their left hand or right hand to issue a cleaning command. Therefore, it is necessary to first determine the limb part from which the user issues the cleaning command, and then determine the pointing point according to the wrist joint point on the limb part. Specifically, the specific implementation method of step S602 includes the following steps b1 to b3:

[0093] Step b1: Determine the movement state of the wrist joint on the ground according to the depth camera point cloud on the wrist joint of the user.

[0094] In step b1, the movement state of the wrist joint on the ground includes the user's first wrist or second wrist moving on the ground, or the first wrist and the second wrist moving on the ground. Specifically, it can be determined according to the position change of the depth camera point cloud of the user's wrist joint between different frames.

[0095] Specifically, the wrist joint points of the user include various wrist joint points, and the position coordinates of each wrist joint point can be determined by the following steps: for each wrist joint point in each wrist joint point, a depth camera point cloud set s within a preset range around the wrist joint point is selected. skel ∈S human_mask , and perform the mean operation on the depth camera point cloud set corresponding to the wrist joint point to obtain the three-dimensional coordinate s of the wrist joint point in the depth camera coordinate system itof_skel_i (x, y, z). The preset interval range may be a range of m pixels around the wrist joint point. m may be a positive integer less than or equal to 10.

[0096] Next, the movement state of each wrist joint on the ground can be determined based on the position coordinates of each wrist joint in the depth camera coordinate system. Specifically, if the position coordinates of the first wrist joint or the second wrist joint of the user are identified to have changed between the t-th frame and the t+k-th frame based on the three-dimensional coordinates of each wrist joint in the depth camera coordinate system, then the first wrist or the second wrist of the user is determined to be moving on the ground; if the position coordinates of the first wrist joint and the second wrist joint of the user are identified to have changed between the t-th frame and the t+k-th frame based on the three-dimensional coordinates of each wrist joint in the depth camera coordinate system, then the first wrist and the second wrist of the user are determined to be moving on the ground. Wherein, k is a positive integer.

[0097] Step b2: Determine the limb part corresponding to the user's limb movement according to the movement state of the wrist joint on the ground.

[0098] This step aims to determine the limb part of the user that issued the cleaning command through the wrist movement of the limb. Specifically, step b2 includes: if the user's first wrist or second wrist moves, then determine the limb part to which the first wrist or second wrist belongs as the limb part corresponding to the user's limb movement; if both the first wrist and the second wrist move, then determine the distance between the wrist joint points of the first wrist and the second wrist and the user's body mass center, and select the limb part to which the wrist corresponding to the maximum distance belongs as the limb part corresponding to the user's limb movement.

[0099] As mentioned above, the user may have a single wrist movement or double wrist movement. In order to accurately identify the limb part of the user that issues the cleaning command, it is necessary to combine the movement status of the user's two wrists to determine. If one of the user's two wrists is displaced on the ground, the limb part to which the wrist belongs can be identified as the limb part to which the user issues the cleaning command; if both of the user's wrists are displaced on the ground, the limb part to which the wrist belongs is selected as the limb part to which the user issues the cleaning command by comparing the distance between the two wrist joints and the center of mass of the human body. This is because, in most cases, people tend to use the limb on the side away from the center of gravity to make pointing movements, which helps to maintain body balance and express directionality more naturally.

[0100] Step b3: connect the depth camera point cloud coordinates corresponding to the large joints and wrist joints of the limbs and extend them to intersect with the ground, and determine the intersection point as the cleaning action pointing point corresponding to the user's limb movement.

[0101] As mentioned above, the user's body part may be an arm or a leg. Therefore, step b3 includes the following implementations:

[0102] In some implementations, when the user's limbs include legs, the large joints and wrist joints corresponding to the limbs are knee joints and ankle joints, respectively. Then step b3 includes: connecting the depth camera point cloud coordinates corresponding to the knee joints and ankle joints and extending them to intersect with the ground, and determining the intersection point as the sweeping action pointing point corresponding to the user's limb action.

[0103] Figure 7 A schematic diagram of determining the direction of a cleaning action according to a limb joint point provided in an embodiment of the present application. Figure 7 As shown, point A represents the knee joint point, and point B represents the ankle joint point. By connecting point A and point B and extending them to the ground, an intersection point C with the ground can be obtained. Intersection point C is the direction point of the cleaning action.

[0104] In some implementations, when the user's limbs include arms, the large joint points and wrist joint points corresponding to the limbs are the elbow joint points and the wrist joint points, respectively. Step b3 includes: connecting the depth camera point cloud coordinates corresponding to the elbow joint points and the wrist joint points and extending them to intersect with the ground, and determining the intersection point as the sweeping action pointing point corresponding to the user's limb action.

[0105] Continue reading Figure 6 After step S602, a cleaning control instruction for the cleaning robot is generated according to the depth camera point cloud, and the following step S603 may also be included.

[0106] S603: Generate a cleaning control instruction for the sweeping robot according to the cleaning action pointing point.

[0107] By continuously detecting and tracking the user's cleaning action points in real time, and connecting the detected cleaning action points, the boundary of the area the user wants to clean is formed, thereby determining the target area. Then, according to the determined target area, the corresponding cleaning control instructions are generated to guide the sweeping robot to automatically clean the target area.

[0108] As mentioned above, when the distance between the user and the robot is less than or equal to the preset distance, the cleaning control instructions are generated based on the depth camera point cloud, and when the distance between the user and the robot is greater than the preset distance, the cleaning control instructions are generated based on the lidar point cloud. Figure 6 and Figure 7 The illustrated embodiment has introduced the specific implementation process of generating a cleaning control instruction for a cleaning robot according to the depth camera point cloud. The specific implementation process of generating a cleaning control instruction according to the laser radar point cloud is introduced below in conjunction with the accompanying drawings.

[0109] In this embodiment, the depth camera can obtain higher-resolution object height information than the laser radar. Therefore, when the human body is close to the robot, the point of the cleaning action can be determined based on the depth camera point cloud; when the human body is far away from the robot, it is difficult to accurately determine the point of the cleaning action because the point cloud obtained by the depth camera is relatively sparse. At this time, the laser radar point cloud can be used to determine the point of the cleaning action.

[0110] It should be noted that the laser radar can only obtain the depth information of objects with a height of 0.27 meters and below, which means that it cannot obtain the laser radar point cloud at the knee joint. Therefore, it is impossible to calculate the pointing point of the cleaning action by the method of step b3 above. However, the three-dimensional coordinates of the first ankle and the second ankle can be calculated by the laser radar point cloud to roughly determine the position of the human body. Then, the final pointing point is calculated based on the laser radar point cloud corresponding to the ankle of the pointing foot. The specific implementation process is as follows:

[0111] Figure 8 A flowchart of generating a cleaning control instruction based on a laser radar point cloud provided in an embodiment of the present application. Figure 8 As shown, generating a cleaning control instruction according to the laser radar point cloud includes the following steps S801 to S803:

[0112] S801. Determine a laser radar point cloud located on the user's wrist joint point according to the user's wrist joint point in the RGB image and a mapping relationship between the laser radar coordinate system and the image coordinate system.

[0113] Continue reading Figure 3 , the RGB image is input into the semantic segmentation model. The semantic segmentation model performs joint point detection based on the RGB image to obtain the user's limb joint points, including the wrist joint points.

[0114] The wrist joint points output by the model are image data in the image coordinate system (i.e., the RGB camera coordinate system). Through the mapping relationship between the depth camera coordinate system and the image coordinate system, the depth camera point cloud of the limb joint points output by the model is converted to the RGB camera perspective, and the depth camera point cloud located at the human limb joint points can be screened out.

[0115] Specifically, the laser radar point cloud collected in step S101 can be converted to the RGB camera perspective through the conversion matrix between the laser radar coordinate system and the image coordinate system (usually including the rigid transformation matrix H' and the RGB camera intrinsic parameter matrix M).

[0116] Among them, the coordinate system conversion process between the laser radar coordinate system and the image coordinate system can be found in the introduction of related technologies and will not be described in detail here.

[0117] S802: Determine the cleaning action pointing point corresponding to the user's body movement based on the laser radar point cloud located at the user's wrist joint point.

[0118] Specifically, step S802 includes the following steps c1 to c3:

[0119] Step c1: determine the movement state of the wrist joint on the ground according to the laser radar point cloud on the user's wrist joint.

[0120] In step c1, the movement state of the wrist joint point on the ground includes the user's first wrist or second wrist moving on the ground, or the first wrist and the second wrist moving on the ground. Specifically, it can be determined according to the position change of the laser radar point cloud of the user's wrist joint point between different frames.

[0121] Specifically, the wrist joint points of the user include various wrist joint points. First, the position coordinates of each wrist joint point in the laser radar coordinates are determined, specifically including: for each wrist joint point in each wrist joint point, a laser radar point cloud set s within a preset range around the wrist joint point is selected. skel ∈S human_mask , and perform the average operation on the laser radar point cloud set corresponding to the wrist joint point to obtain the three-dimensional coordinate s of the wrist joint point in the laser radar coordinate system itof_skel_i (x, y, z). The preset interval range may be a range of m pixels around the wrist joint point. m may be a positive integer less than or equal to 10.

[0122] Next, the movement state of each wrist joint point on the ground can be determined based on the position coordinates of each wrist joint point in the laser radar coordinate system. Specifically, if the position coordinates of the first wrist joint point or the second wrist joint point of the user are identified to have changed between the t-th frame and the t+k-th frame based on the three-dimensional coordinates of each wrist joint point in the laser radar coordinate system, then it is determined that the first wrist or the second wrist of the user is moving on the ground; if the position coordinates of the first wrist joint point and the second wrist joint point of the user are identified to have changed between the t-th frame and the t+k-th frame based on the three-dimensional coordinates of each wrist joint point in the laser radar coordinate system, then it is determined that the first wrist and the second wrist of the user are both moving on the ground. Wherein, k is a positive integer.

[0123] Step c2: Determine the limb part corresponding to the user's limb movement according to the movement state of the wrist joint on the ground.

[0124] This step aims to determine the limb part of the user that issued the cleaning command through the wrist movement of the limb. Specifically, step c2 includes: if the user's first wrist or second wrist moves, then determine the limb part to which the first wrist or second wrist belongs as the limb part corresponding to the user's limb movement; if both the first wrist and the second wrist move, then determine the distance between the wrist joint points of the first wrist and the second wrist and the user's body mass center, and select the limb part to which the wrist corresponding to the maximum distance belongs as the limb part corresponding to the user's limb movement.

[0125] As mentioned above, the user may have a single wrist movement or double wrist movement. In order to accurately identify the limb part of the user that issues the cleaning command, it is necessary to combine the movement status of the user's two wrists to determine. If one of the user's two wrists is displaced on the ground, the limb part to which the wrist belongs can be identified as the limb part to which the user issues the cleaning command; if both of the user's wrists are displaced on the ground, the limb part to which the wrist belongs is selected as the limb part to which the user issues the cleaning command by comparing the distance between the two wrist joints and the center of mass of the human body. This is because, in most cases, people tend to use the limb on the side away from the center of gravity to make pointing movements, which helps to maintain body balance and express directionality more naturally.

[0126] Step c3: determine the cleaning action pointing point corresponding to the user's limb movement according to the laser radar point cloud of the wrist of the limb corresponding to the user's limb movement.

[0127] Specifically, when the user's limb is a leg, the three-dimensional coordinates of the lidar point cloud on the user's ankle joint are directly determined as the cleaning action pointing point corresponding to the user's limb movement; when the user's limb is an arm, the three-dimensional coordinates of the lidar point cloud on the user's wrist joint are directly determined as the cleaning action pointing point corresponding to the user's limb movement.

[0128] Continue reading Figure 8 After step S802, a cleaning control instruction for the sweeping robot is generated according to the laser radar point cloud, and the following step S803 may also be included.

[0129] S803: Generate a cleaning control instruction for the cleaning robot according to the cleaning action pointing point.

[0130] By continuously detecting and tracking the user's cleaning action points in real time, and connecting the detected cleaning action points, the boundary of the area the user wants to clean is formed, thereby determining the target area. Then, according to the determined target area, a corresponding cleaning control instruction is generated to guide the sweeping robot to automatically clean the target area.

[0131] It should be noted that the cleaning action pointing point finally obtained in the above embodiment is a three-dimensional coordinate in the depth camera coordinate system or the laser radar coordinate system, and the sweeping robot works according to its own base. Therefore, when generating corresponding cleaning control instructions according to the determined target area and instructing the sweeping robot to automatically clean the target area, it is necessary to convert it to the robot's base coordinate system through a transformation matrix (such as a rigid transformation matrix and), and then make the robot clean the target area according to the position coordinates of the target area in the base coordinate system.

[0132] In summary, the control method of the sweeping robot provided in the embodiment of the present application can clean the designated dirty area according to the user's requirements, thereby enhancing the interactive experience between the user and the robot. In this process, in order to further expand the recognition range of the sweeping robot to the target area, the human body can issue cleaning instructions through body movements when it is close to and far from the robot. Specifically, when the user is close to the robot, the point cloud of the depth camera is used to accurately determine the pointing point of the cleaning action. When the user is far away from the robot, the laser radar point cloud is used to determine the pointing point of the cleaning action, which makes up for the shortcomings of the depth camera in long-distance recognition. In addition, by combining the respective advantages of the RGB camera, depth camera and laser radar already on the robot, complementary advantages are achieved, so that the robot can be controlled by body movements to clean the designated target area. This method can not only improve the accuracy and robustness of target area recognition, but also enhance the user experience, and can be deployed on low-power, low-computing edge devices to ensure its efficient operation and wide applicability.

[0133] Exemplary Devices

[0134] Corresponding to the above-mentioned control method of the sweeping robot, an embodiment of the present application also provides a control device of the sweeping robot. Fig. 9 Schematic diagram of the structure of a control device of a robot sweeper provided in an embodiment of the present application. Fig. 9 As shown, the control device of the sweeping robot provided in the embodiment of the present application includes: an acquisition unit 901, a determination unit 902 and a generation unit 903; wherein the acquisition unit 901 is used to respond to a cleaning instruction issued by a user through body movements, and acquire an RGB image, a depth camera point cloud and a lidar point cloud of the user; the determination unit 902 is used to determine the interval distance between the user and the sweeping robot according to the RGB image and the depth camera point cloud, and compare the result with the preset distance; the generation unit 903 is used to select the depth camera point cloud or the lidar point cloud according to the comparison result to generate a cleaning control instruction for the sweeping robot.

[0135] In some embodiments, the determination unit 902 determines the interval distance between the user and the sweeping robot based on the RGB image and the depth camera point cloud, and compares the result with the preset distance, including: performing semantic segmentation based on the RGB image to obtain a human mask image of the user; determining the depth camera point cloud located on the human mask image based on the human mask image of the user and a mapping relationship between the depth camera coordinate system and the image coordinate system; determining the comparison result between the interval distance and the preset distance based on the number of depth camera point clouds located on the human mask image.

[0136] In some embodiments, the determination unit 902 determines a comparison result between the interval distance and the preset distance based on the number of depth camera point clouds located on the human body mask image, including: if the number of depth camera point clouds located on the human body mask image is less than or equal to the preset number, determining that the interval distance is greater than the preset distance; if the number of depth camera point clouds located on the human body mask image is greater than the preset number, determining that the interval distance is less than or equal to the preset distance.

[0137] In some embodiments, the determination unit 902 determines the interval distance between the user and the sweeping robot based on the RGB image and the depth camera point cloud, and the comparison result with the preset distance includes: determining the user's body center of mass coordinates based on the RGB image, the depth camera point cloud, and the mapping relationship between the depth camera coordinate system and the image coordinate system, the body center of mass coordinates including the z-axis value; comparing the z-axis value in the body center of mass coordinates with the preset distance to obtain a comparison result between the interval distance and the preset distance.

[0138] In some embodiments, the determination unit 902 determines the coordinates of the user's body center of mass based on the depth camera point cloud and the mapping relationship between the depth camera coordinate system and the image coordinate system, including: performing semantic segmentation based on the RGB image to obtain a body mask image of the user; determining the depth camera point cloud located on the body mask based on the user's body mask image and the mapping relationship between the depth camera coordinate system and the image coordinate system; determining the coordinates of the user's body center of mass based on the depth camera point cloud located on the body mask image.

[0139] In some embodiments, the generation unit 903 selects to generate cleaning control instructions for the sweeping robot based on the depth camera point cloud or the lidar point cloud according to the comparison result between the interval distance and the preset distance, including: if the interval distance is less than or equal to the preset distance, then generating a cleaning control instruction for the sweeping robot according to the depth camera point cloud; if the interval distance is greater than the preset distance, then generating a cleaning control instruction for the sweeping robot according to the lidar point cloud.

[0140] In some embodiments, the generation unit 903 generates cleaning control instructions for the sweeping robot based on the depth camera point cloud, including: determining the depth camera point cloud located at the user's limb joints based on the user's limb joints in the RGB image and the mapping relationship between the depth camera coordinate system and the image coordinate system; determining the cleaning action pointing point corresponding to the user's limb movement based on the depth camera point cloud located at the user's limb joints; and generating cleaning control instructions for the sweeping robot based on the cleaning action pointing point.

[0141] In some embodiments, the limb joints include large joints and wrist joints, and the generating unit 903 determines the sweeping action pointing point corresponding to the user's limb movement according to the depth camera point cloud on the user's limb joints, including: determining the moving state of the wrist joint on the ground according to the depth camera point cloud on the user's wrist joint; determining the limb part corresponding to the user's limb movement according to the moving state of the wrist joint on the ground; connecting the depth camera point cloud coordinates corresponding to the large joints and wrist joints corresponding to the limb part and extending them to intersect with the ground, and determining the intersection point as the sweeping action pointing point corresponding to the user's limb movement.

[0142] In some embodiments, the movement state of the wrist joint point on the ground includes the first wrist or the second wrist of the user moving on the ground, or the first wrist and the second wrist of the user both moving on the ground; wherein, the generation unit 903 determines the limb part corresponding to the user's limb movement according to the movement state of the wrist joint point on the ground, including: if the first wrist or the second wrist of the user moves, then determine that the limb part to which the first wrist or the second wrist belongs is the limb part corresponding to the user's limb movement; if both the first wrist and the second wrist move, then determine the distance between the wrist joint points of the first wrist and the second wrist and the user's body center of mass, and select the limb part to which the wrist corresponding to the maximum distance belongs as the limb part corresponding to the user's limb movement.

[0143] In some embodiments, the limb joints include wrist joints, and the generation unit 903 generates cleaning control instructions for the sweeping robot according to the laser radar point cloud, including: determining the laser radar point cloud located on the wrist joint of the user according to the wrist joint of the user in the RGB image, and the mapping relationship between the laser radar coordinate system and the image coordinate system; determining the cleaning action pointing point corresponding to the user's limb movement according to the laser radar point cloud located on the wrist joint of the user; and generating cleaning control instructions for the sweeping robot according to the cleaning action pointing point.

[0144] In some embodiments, the generation unit 903 determines the sweeping action pointing point of the user's limb movement based on the laser radar point cloud located on the wrist joint of the user, including: determining the moving state of the wrist joint on the ground based on the laser radar point cloud on the wrist joint of the user; determining the limb part corresponding to the user's limb movement based on the moving state of the wrist joint on the ground; determining the sweeping action pointing point corresponding to the limb movement based on the laser radar point cloud of the wrist on the limb part corresponding to the user's limb movement.

[0145] The control device of the sweeping robot provided in this embodiment belongs to the same application concept as the control method of the sweeping robot provided in the above embodiments of this application, and can execute the control method of the sweeping robot provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the control method of the sweeping robot. For the technical details not fully described in this embodiment, please refer to the specific processing content of the control method of the sweeping robot provided in the above embodiments of this application, and will not be repeated here.

[0146] The functions implemented by the above acquisition unit 901, determination unit 902 and generation unit 903 may be implemented by the same or different processors respectively, and the embodiment of the present application is not limited thereto.

[0147] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0148] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0149] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0150] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0151] Exemplary Electronic Devices

[0152] The present application embodiment provides a control device for a sweeping robot. Fig.10 As shown, the control device includes:

[0153] Memory 200 and processor 210;

[0154] The memory 200 is connected to the processor 210 and is used to store programs;

[0155] The processor 210 is used to implement the control method of the sweeping robot disclosed in any of the above embodiments by running the program stored in the memory 200.

[0156] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0157] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0158] A bus may include a pathway that transfers information between components of a computer system.

[0159] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0160] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0161] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0162] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0163] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0164] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0165] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of the control method of any scanning robot provided in the above embodiments of the present application.

[0166] An embodiment of the present application also proposes a sweeping robot, which includes the above-mentioned exemplary control device and a sweeping robot body; wherein an RGB camera, a depth camera and a laser radar are installed on the sweeping robot body, and the RGB camera, the depth camera and the laser radar are respectively connected to the control device, and the RGB camera, the depth camera and the laser radar are respectively used to collect RGB images including the user, the depth camera point cloud and the laser radar point cloud, and send them to the control device to execute the control method of the sweeping robot introduced in any of the above-mentioned embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the control method of the sweeping robot mentioned above.

[0167] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored in the memory through the data interface to execute the control method of the sweeping robot introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the control method of the sweeping robot mentioned above.

[0168] Exemplary computer program products and storage media

[0169] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the control method of the sweeping robot according to various embodiments of the present application described in any of the above embodiments of this specification.

[0170] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0171] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the control method of the sweeping robot according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0172] Step S101, in response to a cleaning instruction issued by a user through body movements, obtaining an RGB image, a depth camera point cloud, and a lidar point cloud of the user.

[0173] Step S102: Determine the interval distance between the user and the cleaning robot according to the RGB image and the depth camera point cloud, and compare the interval distance with a preset distance.

[0174] Step S103: According to the comparison result between the interval distance and the preset distance, select the depth camera point cloud or the lidar point cloud to generate a cleaning control instruction for the sweeping robot.

[0175] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0176] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0178] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0179] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0180] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0182] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0183] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0184] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0185] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method for a sweeping robot, characterized in that: include: In response to a cleaning instruction issued by a user through a body movement, an RGB image, a depth camera point cloud, and a lidar point cloud of the user are acquired; Determine the interval distance between the user and the cleaning robot according to the RGB image and the depth camera point cloud, and compare the interval distance with the preset distance; According to the comparison result, a cleaning control instruction for the cleaning robot is generated based on the depth camera point cloud or the lidar point cloud.

2. The method according to claim 1, characterized in that The determining of the interval distance between the user and the cleaning robot according to the RGB image and the depth camera point cloud, and the comparison result with the preset distance, includes: Performing semantic segmentation according to the RGB image to obtain a human body mask image of the user; Determine a depth camera point cloud located on the human body mask image according to the human body mask image of the user and a mapping relationship between a depth camera coordinate system and an image coordinate system; A comparison result between the interval distance and a preset distance is determined according to the number of depth camera point clouds located on the human body mask image.

3. The method according to claim 2, characterized in that The step of determining a comparison result between the interval distance and a preset distance according to the number of depth camera point clouds located on the human body mask image comprises: If the number of depth camera point clouds located on the human body mask image is less than or equal to a preset number, determining that the interval distance is greater than a preset distance; If the number of depth camera point clouds located on the human body mask image is greater than a preset number, it is determined that the interval distance is less than or equal to the preset distance.

4. The method according to claim 1, characterized in that: The determining of the interval distance between the user and the cleaning robot according to the RGB image and the depth camera point cloud, and the comparison result with the preset distance, includes: Determine the body center of mass coordinates of the user according to the RGB image, the depth camera point cloud, and a mapping relationship between the depth camera coordinate system and the image coordinate system, wherein the body center of mass coordinates include a z-axis value; The z-axis value in the human body center of mass coordinates is compared with the preset distance to obtain a comparison result between the interval distance and the preset distance.

5. The method according to claim 4, characterized in that Determining the coordinates of the user's body center of mass according to the depth camera point cloud and the mapping relationship between the depth camera coordinate system and the image coordinate system includes: Performing semantic segmentation according to the RGB image to obtain a human body mask image of the user; Determine a depth camera point cloud located on the human body mask according to the human body mask image of the user and a mapping relationship between a depth camera coordinate system and an image coordinate system; The user's body center of mass coordinates are determined based on the depth camera point cloud located on the body mask image.

6. The method according to any one of claims 1 to 5, characterized in that The step of selecting, according to the comparison result between the interval distance and the preset distance, the depth camera point cloud or the laser radar point cloud to generate a cleaning control instruction for the cleaning robot comprises: If the interval distance is less than or equal to the preset distance, generating a cleaning control instruction for the cleaning robot according to the depth camera point cloud; If the interval distance is greater than the preset distance, a cleaning control instruction for the cleaning robot is generated according to the laser radar point cloud.

7. The method according to claim 6, characterized in that The step of generating a cleaning control instruction for the cleaning robot according to the depth camera point cloud comprises: Determine a depth camera point cloud located at a limb joint point of the user according to the limb joint point of the user in the RGB image and a mapping relationship between a depth camera coordinate system and an image coordinate system; Determining a cleaning action pointing point corresponding to the user's limb action according to a depth camera point cloud located at a limb joint point of the user; A cleaning control instruction for the cleaning robot is generated according to the cleaning action pointing point.

8. The method according to claim 7, characterized in that The limb joints include large joints and wrist joints, and determining the sweeping action pointing point corresponding to the limb action of the user according to the depth camera point cloud on the limb joints of the user includes: Determining a movement state of the wrist joint point on the ground according to a depth camera point cloud on the wrist joint point of the user; Determining the limb part corresponding to the limb movement of the user according to the movement state of the wrist joint point on the ground; The depth camera point cloud coordinates corresponding to the large joint points and wrist joint points corresponding to the limb parts are connected and extended to intersect with the ground, and the intersection point is determined as the cleaning action pointing point corresponding to the user's limb action.

9. The method according to claim 8, characterized in that The movement state of the wrist joint point on the ground includes the first wrist or the second wrist of the user moving on the ground, or the first wrist and the second wrist of the user moving on the ground; Wherein, determining the limb part corresponding to the limb movement of the user according to the movement state of the wrist joint point on the ground includes: If the first wrist or the second wrist of the user moves, determining that the limb part to which the first wrist or the second wrist belongs is the limb part corresponding to the limb movement of the user; If both the first wrist and the second wrist move, the distance between the wrist joint points of the first wrist and the second wrist and the body mass center of the user is determined, and the limb part to which the wrist corresponding to the maximum distance belongs is selected as the limb part corresponding to the limb movement of the user.

10. The method according to claim 6, characterized in that The limb joints include wrist joints, and the generating of cleaning control instructions for the cleaning robot according to the laser radar point cloud includes: Determine a laser radar point cloud located on the wrist joint point of the user according to the wrist joint point of the user in the RGB image and a mapping relationship between a laser radar coordinate system and an image coordinate system; Determining a cleaning action pointing point corresponding to the user's body movement according to a laser radar point cloud located at the user's wrist joint point; A cleaning control instruction for the cleaning robot is generated according to the cleaning action pointing point.

11. The method according to claim 10, characterized in that The step of determining the sweeping motion pointing point of the user's limb motion according to the laser radar point cloud located at the wrist joint of the user includes: Determining a movement state of the wrist joint point on the ground according to a laser radar point cloud on the wrist joint point of the user; Determining the limb part corresponding to the limb movement of the user according to the movement state of the wrist joint point on the ground; According to the laser radar point cloud of the wrist on the limb part corresponding to the limb movement of the user, the cleaning action pointing point corresponding to the limb movement is determined.

12. A control device for a sweeping robot, characterized in that: include: An acquisition unit, configured to acquire an RGB image, a depth camera point cloud, and a laser radar point cloud including the user in response to a cleaning instruction issued by the user through a body movement; A determination unit, configured to determine a comparison result between the interval distance between the user and the cleaning robot and a preset distance according to the RGB image and the depth camera point cloud; A generating unit is used to generate a cleaning control instruction for the cleaning robot based on the depth camera point cloud or the lidar point cloud according to the comparison result between the interval distance and the preset distance.

13. A control device for a sweeping robot, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 11 by running the program in the memory.

14. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 11.

15. A sweeping robot, characterized in that: include: The control device of the sweeping robot as claimed in claim 13, and the sweeping robot body.

Images