Cleaning robot control method, cleaning robot, cleaning system, medium and product

Abstract

The invention discloses a control method of a cleaning robot, the cleaning robot, a cleaning system, a computer readable storage medium and a computer program product. The method comprises the following steps: S1, determining a first target object; s2, controlling the cleaning robot to move to a first preset distance of the first target object; s3, identifying a current placement pose of the first target object; s4, determining a first pose of the mechanical arm for clamping the first target object; s5, the mechanical arm is controlled to clamp the first target object in the first pose, and the cleaning robot is controlled to move to the target area; s6, determining a target placement pose of the first target object on the placement object; s7, according to the first pose and the target placement pose, a second pose of the mechanical arm for placing the first target object is determined; and S8, the mechanical arm is controlled to place the first target object in the target placement pose at the second pose. Therefore, standardized ordered placement of the first target object can be realized, and the user experience is improved to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of cleaning robot technology, and in particular to a control method for a cleaning robot, a cleaning robot, a cleaning system, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In related technologies, cleaning robots, as intelligent home appliances capable of autonomous navigation and cleaning, can replace users in performing daily household chores such as sweeping and mopping. However, cleaning robots that only have floor cleaning functions have relatively limited application scenarios. Summary of the Invention

[0003] This application provides a control method for a cleaning robot, a cleaning robot, a cleaning system, a computer-readable storage medium, and a computer program product.

[0004] This application provides a control method for a cleaning robot, the cleaning robot including a robotic arm capable of performing gripping actions, the method comprising: S1: Identify the first target object; S2: Control the cleaning robot to move to within a first preset distance of the first target object; S3: Identify the current placement posture of the first target object; S4: Determine the first pose of the robotic arm gripping the first target object; S5: Control the robotic arm to grasp the first target object in the first position, and control the cleaning robot to move to the target area; S6: Determine the target placement pose of the first target object on the placement object; S7: Determine the second pose for the robotic arm to place the first target object based on the first pose and the target placement pose; S8: Control the robotic arm to place the first target object in the target placement pose in the second position.

[0005] Thus, by determining the second pose for the robotic arm to place the first target object based on the first pose and the target placement pose, and controlling the robotic arm to place the first target object in the target placement pose using the second pose, precise gripping and placement of the first target object can be achieved, improving the user experience. Compared to cleaning robots that only have floor cleaning functions or simple pick-up and place functions, the embodiment of this application can achieve standardized and orderly placement of the first target object while having floor cleaning functions, broadening the application scenarios of cleaning robots and improving the user experience to a certain extent.

[0006] In some implementations, the sub-step of determining the first target object includes: Acquire environmental images; Detect the second target object in the environmental image that is not placed on the placement object; The detected second target object is identified as the first target object.

[0007] In this way, an environmental image is acquired; a second target object that is not placed on the designated object is detected in the environmental image; and the detected second target object is identified as the first target object. Thus, by detecting the acquired environmental image, the second target object can be identified, and the first target object can be identified from the second target object, accurately distinguishing shoes from other objects and properly placed shoes. This provides a target for subsequent sorting tasks, avoiding problems such as accidental picking or omissions, and improving the accuracy of the cleaning robot's sorting tasks.

[0008] In some implementations, the sub-step of identifying the detected second target object as the first target object includes: In the case of detecting at least one second target object, the target object closest to the cleaning robot among the at least one second target object is identified as the first target object.

[0009] Thus, when at least one second target object is detected, the target object closest to the cleaning robot among the at least one second target object is determined as the first target object. By determining the target object closest to the cleaning robot among the at least one second target object as the first target object, a specific target can be provided for subsequent control of the cleaning robot's movement and movement, and the robot can avoid moving back and forth between multiple distant targets, thereby improving the efficiency of the cleaning robot in performing its cleaning tasks.

[0010] In some implementations, the sub-step of identifying the current placement pose of the first target object includes: Obtain the first target image of the first target object; Based on the first target image, identify the current placement posture of the first target object.

[0011] Thus, a first target image of the first target object is acquired; based on the first target image, the current placement pose of the first target object is identified. In this way, by acquiring the first target image, the current placement pose of the first target object can be accurately calculated, providing accurate data support for subsequently determining the first pose for gripping the first target object, ensuring the reliability of the cleaning robot's gripping and placement.

[0012] In some implementations, the sub-step of identifying the current placement pose of the first target object based on the first target image includes: The first target image is segmented to obtain segmented image blocks containing the first target object. The segmented image block is input into a pre-trained placement pose prediction model to obtain the current placement pose output by the placement pose prediction model.

[0013] Thus, image segmentation processing is performed on the first target image to obtain segmented image blocks containing the first target object. These segmented image blocks are then input into a pre-trained placement pose prediction model to obtain the current placement pose output by the model. In this way, by performing image segmentation processing on the first target image, pixels related to the first target object can be extracted, resulting in segmented image blocks containing the first target object. This removes background interference, providing clean data for the subsequent placement pose prediction model. Inputting these segmented image blocks into the pre-trained model outputs the current placement pose of the first target object, providing data support for determining the first pose of the robotic arm gripping the first target object.

[0014] In some implementations, the sub-step of determining the first pose of the robotic arm gripping the first target object includes: Obtain the first point cloud data of the first target object; Based on the segmented image blocks and the first point cloud data, the first pose of the robotic arm gripping the first target object is determined.

[0015] Thus, the first point cloud data of the first target object is acquired; based on the segmented image blocks and the first point cloud data, the first pose of the robotic arm in grasping the first target object is determined. In this way, by fusing the planar features of the segmented image blocks and the stereo features of the first point cloud data, the first pose suitable for the robotic arm in grasping the first target object can be determined, thereby improving the grasping success rate and adaptability.

[0016] In some implementations, the sub-step of determining the first pose of the robotic arm gripping the first target object based on the segmented image patch and the first point cloud data includes: The first point cloud data is segmented based on the segmented image blocks to determine the second point cloud data; The current placement pose, the segmented image block, and the second point cloud data are input into a pre-trained gripping pose prediction model to obtain the first pose output by the gripping pose prediction model.

[0017] Thus, the first point cloud data is segmented based on the segmented image blocks to determine the second point cloud data. The current placement pose, the segmented image blocks, and the second point cloud data are then input into a pre-trained gripping pose prediction model to obtain the first pose output by the model. In this way, the gripping pose prediction model, based on the planar features of the segmented image blocks and the three-dimensional features of the second point cloud data, combined with the current placement pose of the first target object (i.e., its three-dimensional structure and placement state), can determine the appropriate gripping position and posture for the robotic arm to grasp the first target object—the first pose—thereby improving the gripping success rate and adaptability.

[0018] In some implementations, the sub-step of determining the target placement pose of the first target object on the placement object includes: Based on the segmented image blocks, a third target object is detected on the placement object, wherein the first target object and the third target object are both one of a set of multiple objects; The target placement posture is determined based on the detection results of the third target object.

[0019] Thus, based on the segmented image blocks, a third target object is detected on the placement object. Both the first and third target objects are part of a set of multiple objects. Based on the detection result of the third target object, the target placement pose is determined. In this way, based on the segmented image blocks, the cleaning robot can detect whether there is a third target object belonging to the same set as the first target object in the placement object, and then determine the target placement pose based on the detection result. This ensures that the target placement pose of the first target object meets the matching placement requirements when a third target object is present, and adapts to the spatial layout of the placement object when a third target object is absent.

[0020] In some implementations, the sub-step of determining the target placement pose based on the detection result of the third target object includes: When the third target object is detected, the target placement posture is determined according to the preset placement direction and the first placement position, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. If the third target object is not detected, the target placement posture is determined according to the preset placement direction and the second placement position, wherein the distance between the second placement position and the placement position of the target object already placed on the placement object is a preset interval distance.

[0021] Thus, when a third target object is detected, the target placement posture is determined based on the preset placement direction and the first placement position, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. When no third target object is detected, the target placement posture is determined based on the preset placement direction and the second placement position, wherein the distance between the second placement position and the placement positions of target objects already placed on the object being placed is a preset interval distance. This allows for the standardized and orderly placement of the first target object, ensuring that the target placement posture of the first target object is consistent with the placement direction of other target objects, and enabling sets of objects to be placed close together within a preset distance threshold, while non-set objects are placed at preset distance threshold intervals, improving the orderliness and aesthetics of the arrangement.

[0022] In some implementations, the sub-step of detecting a third target object on the placement object based on the segmented image patch includes: The segmented image block corresponding to the first target object is subjected to feature information extraction processing to determine the first feature information corresponding to the first target object; Determine the degree of similarity between the first feature information and the second feature information of the fourth target object already placed on the placement object; If the similarity is greater than a preset threshold, the fourth target object is identified as the third target object.

[0023] In this way, feature information extraction processing is performed on the segmented image block corresponding to the first target object to determine the first feature information corresponding to the first target object; the similarity between the first feature information and the second feature information of the fourth target object already placed on the object is determined; if the similarity is greater than a preset threshold, the fourth target object is determined as the third target object. Thus, by performing feature information extraction processing on the segmented image block corresponding to the first target object, two-dimensional image information can be transformed into a high-dimensional feature vector to determine the first feature information corresponding to the first target object. Then, by comparing the similarity between the first and second feature information, the fourth target object with a similarity greater than a preset threshold is determined as the third target object paired with the first target object, achieving accurate identification of the matching object. This provides a data foundation for subsequently determining the target placement pose of the first target object and supports the standardized and orderly placement of the first target object.

[0024] In some embodiments, the method further includes: Identify the placement status of the target objects already placed on the placement object; When a fifth target object is identified in the first placement state, the robotic arm is controlled to place the fifth target object a second time, so that the placement state of the fifth target object changes from the first placement state to the second placement state.

[0025] In this way, the system identifies the placement status of the target objects already placed on the placement platform. When a fifth target object is identified in its first placement state, the robotic arm is controlled to perform a second placement of the fifth target object, switching its placement status from the first to the second. By identifying the placement status and correcting the fifth target object's placement from the first to the second, standardized and orderly placement of target objects on the placement platform can be achieved, improving the organization efficiency and user experience to some extent.

[0026] In some embodiments, the method further includes: After performing a preset number of consecutive secondary placements on the fifth target object, if the placement state of the fifth target object is the first placement state, the second target image of the placement object is obtained; Upon receiving an image upload permission instruction, the second target image is sent to the server, wherein the server generates a target instruction based on the second target image and sends the target instruction to the cleaning robot; According to the received target instruction, the cleaning robot is controlled to move the fifth target object, so that the placement state of the fifth target object changes from the first placement state to the second placement state; and / or, The cleaning robot is updated according to the received target instruction.

[0027] Thus, after performing a preset number of consecutive repositioning operations on the fifth target object, if the fifth target object is in its first placement state, a second target image of the object is acquired. Upon receiving an image upload permission command, the second target image is sent to the server. The server generates target instructions based on the second target image and sends these instructions to the cleaning robot. Based on the received target instructions, the cleaning robot moves the fifth target object to switch its placement state from the first to the second. The cleaning robot is then updated based on the received target instructions. This allows the server to analyze the information in the second target image to determine why the placement of the fifth target object cannot be corrected, and to generate and send target instructions to the cleaning robot. This enables the cleaning robot to move the fifth target object based on the received target instructions, switching its placement state from the first to the second. Furthermore, the cleaning robot can update itself based on the received target instructions. This approach, while ensuring user privacy and security, solves the problem of failed repositioning and achieves standardized and orderly placement of target objects, improving organization and user experience to some extent.

[0028] In some implementations, the sub-step of controlling the cleaning robot to move to the target area includes: Based on the current placement posture and the target placement posture, the movement path of the cleaning robot to the target area is determined; Control the cleaning robot to move along the movement path to the target area.

[0029] Thus, based on the current and target placement postures, the movement path of the cleaning robot to the target area is determined; the cleaning robot is then controlled to move along the movement path to the target area. In this way, based on the current and target placement postures, the straight-line distance and approximate direction of the cleaning robot's movement to the target area can be calculated. Combined with scene information, such as information about fixed obstacles, an optimal travel route, i.e., the movement path, is generated to control the cleaning robot to move along the movement path to the target area, thereby achieving object sorting.

[0030] This application provides a cleaning robot, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the above-described method.

[0031] This application provides a cleaning system, including a base station and a cleaning robot, to implement the steps of the above method.

[0032] This application provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the steps of the above-described method.

[0033] The cleaning robot, cleaning system, and computer-readable storage medium provided in this application's embodiments determine a first target object; control the cleaning robot to move within a first preset distance of the first target object; identify the current placement posture of the first target object; determine a first posture for the robotic arm to grasp the first target object; control the robotic arm to grasp the first target object in the first posture, and control the cleaning robot to move to the target area; determine a target placement posture for the first target object on the placement object; determine a second posture for the robotic arm to place the first target object based on the first posture and the target placement posture; and control the robotic arm to place the first target object in the target placement posture using the second posture. Thus, by determining the second posture for the robotic arm to place the first target object based on the first posture and the target placement posture, and controlling the robotic arm to place the first target object in the target placement posture using the second posture, precise grasping and placement of the first target object can be achieved, improving the user experience. Compared to cleaning robots that only have floor cleaning functions or simple pick-up and place functions, this application's embodiments can achieve standardized and orderly placement of the first target object while possessing floor cleaning functions, broadening the application scenarios of the cleaning robot and improving the user experience to a certain extent.

[0034] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0035] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein: Figure 1 This is one of the flowcharts illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 2 This is a second schematic flowchart of the control method for a cleaning robot according to certain embodiments of this application; Figure 3 This is a third flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 4 This is the fourth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 5 This is the fifth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 6This is a sixth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 7 This is the seventh flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 8 This is the eighth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 9 This is the ninth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 10 This is the tenth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 11 This is eleventh of the flowcharts illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 12 This is the twelfth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application; Figure 13 This is the thirteenth flowchart illustrating the control method of a cleaning robot according to certain embodiments of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0037] Cleaning robots, as intelligent home appliances capable of autonomous navigation and cleaning, can replace users in performing daily chores such as sweeping and mopping, thereby reducing users' household burden.

[0038] However, cleaning robots that only have floor cleaning functions have limited application scenarios and cannot cover the more diverse tidying needs in a home environment. In daily home scenarios, shoes are often scattered in non-designated areas such as the entryway and living room, which not only affects the tidiness of the home but may also obstruct the movement path of the cleaning robot, increasing the trouble for users to clean up afterward.

[0039] Most cleaning robots in related technologies focus on single picking and storage functions, failing to achieve flexible, efficient, and orderly placement. There is an urgent need for a cleaning robot with precise picking and storage capabilities to achieve flexible, efficient, and orderly placement.

[0040] Based on the above issues, please refer to Figure 1This application provides a control method for a cleaning robot, the cleaning robot including a robotic arm capable of performing gripping actions, the method comprising: S1: Identify the first target object; S2: Control the cleaning robot to move to within a first preset distance of the first target object; S3: Identify the current placement posture of the first target object; S4: Determine the first pose of the robotic arm when gripping the first target object; S5: Control the robotic arm to grasp the first target object in the first pose, and control the cleaning robot to move to the target area; S6: Determine the target placement pose of the first target object on the placement object; S7: Based on the first pose and the target placement pose, determine the second pose for the robotic arm to place the first target object; S8: Control the robotic arm to place the first target object in the target placement pose in the second position.

[0041] This application provides a control device for a cleaning robot. The control method for the cleaning robot according to this application can be implemented by the control device for the cleaning robot according to this application. Specifically, the device for the cleaning robot includes a determining module, a control module, and an identifying module. The determining module is used to determine a first target object. The control module is used to control the cleaning robot to move within a first preset distance of the first target object. The identifying module is used to identify the current placement posture of the first target object. The determining module is also used to determine a first posture for the robotic arm to grasp the first target object. The control module is also used to control the robotic arm to grasp the first target object in the first posture and control the cleaning robot to move to the target area. The determining module is also used to determine a target placement posture of the first target object on the placement object. The determining module is also used to determine a second posture for the robotic arm to place the first target object in the target placement posture based on the first posture and the target placement posture. The control module is also used to control the robotic arm to place the first target object in the target placement posture in the second posture.

[0042] This application also provides a cleaning robot, which includes a memory and a processor. The control method of the cleaning robot according to this application can be implemented by the cleaning robot. Specifically, the memory stores a computer program, and the processor is used to determine a first target object. The processor is also used to control the cleaning robot to move within a first preset distance of the first target object. The processor is also used to identify the current placement posture of the first target object. The processor is also used to determine a first posture for the robotic arm to grasp the first target object. The processor is also used to control the robotic arm to grasp the first target object in the first posture and control the cleaning robot to move to the target area. The processor is also used to determine a target placement posture of the first target object on the placement object. The processor is also used to determine a second posture for the robotic arm to place the first target object based on the first posture and the target placement posture. The processor is also used to control the robotic arm to place the first target object in the target placement posture in the second posture.

[0043] Specifically, the first target object refers to the object that the cleaning robot arm needs to clean, such as the shoes located outside the shoe cabinet and closest to the cleaning robot in a home setting.

[0044] The first preset distance is the sensing distance between the cleaning robot and the first target object, used to ensure that the cleaning robot can identify the current placement posture of the first target object within the first preset distance, such as 30 to 50 centimeters.

[0045] The current placement pose refers to the six-dimensional pose of the first target object, including three-dimensional position parameters and three-dimensional attitude parameters, such as the X, Y, and Z coordinates relative to the cleaning robot reference frame and the rotation angles of the X, Y, and Z axes represented by Euler angles or quaternions, which are used to characterize the real-time placement state of the first target object.

[0046] The first pose is the specific parameter for the robotic arm to grasp the first target object, including the grasping position and grasping posture, which is used to ensure that the grasping action is accurate and reliable. For example, when the robotic arm of the cleaning robot is located at a 45-degree angle to the left of the shoe heel, 3 cm horizontally away from the shoe, 20 cm high, and rotated at a 30-degree angle, the robotic arm can accurately grasp the shoe.

[0047] The target area is the final destination where the robot moves the first target object, such as the area near the shoe cabinet where the shoes should be placed.

[0048] The placement object refers to the carrier that the first target object needs to be placed in a standardized manner, such as a shoe cabinet or shoe rack. The spatial structure of the placement object has been pre-stored in the cleaning robot.

[0049] The target placement posture refers to the standard placement state of the first target object on the placement object, including the placement position and placement direction. For example, paired shoes are placed close together, unpaired shoes are placed at intervals, and the toes of the shoes face the same direction.

[0050] The second pose is the specific parameter for the robotic arm to place the first target object. It is the transition parameter for the robotic arm to switch the shoe from the gripping state to the target placement state. It is derived from the first pose and the target placement pose and is used to ensure that the robotic arm places the first target object accurately and smoothly.

[0051] The cleaning robot can first collect images of the current environment through image acquisition devices such as cameras, and then identify the first target object that the cleaning robot arm needs to clean from the environmental images; Then, the cleaning robot is controlled to move to a first preset distance from the first target object so that it can clearly perceive the first target object and identify the current position of the first target object within the first preset distance, laying the foundation for determining the first position of the robotic arm to pick up the first target object. Next, based on the current position of the first target object, the gripping position and gripping posture of the robotic arm to grip the first target object can be determined, i.e., the first posture, to achieve precise gripping planning, so as to control the robotic arm to grip the first target object in the first posture and control the cleaning robot to move to the target area. Afterwards, the cleaning robot can identify the spatial distribution of objects in the target area and the target objects already placed on the objects. Based on preset placement rules, such as set objects need to be placed close together, non-set objects are placed at intervals to reserve placement space for set objects, the placement direction of objects, and the placement spacing, the robot can determine the target placement posture of the first target object on the objects. Then, based on the first pose of the first target object and the target placement pose, the placement position and posture of the robotic arm for placing the first target object can be determined, i.e., the second pose, so as to achieve precise placement planning. Finally, the robotic arm is controlled to place the first target object onto the placement object in a second pose, achieving precise gripping and placement of the first target object and improving the user experience. Compared to cleaning robots that only have floor cleaning functions or simple pick-up and placement functions, the embodiment of this application can achieve standardized and orderly placement of the first target object while having floor cleaning functions, thus broadening the application scenarios of cleaning robots and improving the user experience to a certain extent.

[0052] The following complete example illustrates the flow of the control method for the cleaning robot according to the embodiments of this application: First, in a home setting, the cleaning robot can use its binoculars to capture images of the surrounding environment and identify the shoes that are not placed in the shoe cabinet and are closest to the cleaning robot as the object to be cleaned, i.e., the first target object. Then, control the cleaning robot to move to a position within 40 centimeters of the shoe, which is the first preset distance; Next, a close-up image of the shoe is captured again using the binocular camera to identify the current placement posture of the shoe based on the close-up image. Based on the current placement posture of the shoe, the gripping posture of the cleaning robot to pick up the shoe is determined, i.e., the first posture. Then, control the robotic arm of the cleaning robot to grip the shoe in the first position, and control the cleaning robot to move to 10 centimeters in front of the shoe cabinet, which is the target area; Next, determine whether there is a matching shoe in the shoe cabinet. If there is a matching shoe, it can be determined that the shoe needs to be placed next to the matching shoe, that is, the target placement posture is determined. Then, based on the previous gripping posture and the target placement posture, the final placement posture of the shoe is determined, that is, the second posture. If there is no shoe that matches the shoe, the shoe needs to be placed alternately with other shoes to determine the target placement position. Then, based on the previous gripping position and the target placement position, the final placement position of the shoe, i.e. the second position, needs to be determined. Finally, based on the second pose, the first target object is placed in the target placement pose, ensuring that the target placement pose of the first target object is consistent with the placement direction of other target objects, and realizing the standardized and orderly placement of the first target object by placing set objects close together and non-set objects at intervals, thereby improving the user experience.

[0053] In summary, by identifying the first target object, controlling the cleaning robot to move within a first preset distance of the first target object, recognizing the current placement posture of the first target object, determining the first posture for the robotic arm to grasp the first target object, controlling the robotic arm to grasp the first target object in the first posture, controlling the cleaning robot to move to the target area, determining the target placement posture of the first target object on the placement object, determining the second posture for the robotic arm to place the first target object based on the first posture and the target placement posture, and controlling the robotic arm to place the first target object in the target placement posture in the second posture, the precise grasping and placement of the first target object can be achieved, improving the user experience. Compared to cleaning robots that only have floor cleaning functions or simple pick-up and place functions, the embodiment of this application can achieve standardized and orderly placement of the first target object while having floor cleaning functions, broadening the application scenarios of the cleaning robot and improving the user experience to a certain extent.

[0054] Please seeFigure 2 In some implementations, the sub-steps of step S1 (determining the first target object) include: S11: Acquire environmental images; S12: Detect a second target object in the environment image that is not placed on the placement object; S13: The detected second target object is identified as the first target object.

[0055] In some implementations, the determining module is further configured to acquire an environmental image. The determining module is further configured to detect a second target object in the environmental image that is not placed on the placement object. The determining module is further configured to determine the detected second target object as the first target object.

[0056] In some implementations, the processor is further configured to acquire an environmental image. The processor is also configured to detect a second target object in the environmental image that is not placed on the placement object. The processor is further configured to identify the detected second target object as the first target object.

[0057] Specifically, environmental images are scene images acquired by the cleaning robot through its onboard image acquisition device. These images are used to detect and determine the first target object that the robotic arm needs to grasp and place. For example, a panoramic image of a home scene is captured in real time by the cleaning robot's panoramic camera.

[0058] The second target object is an object identified in the environmental image that is not placed on the placement object and is used to determine the first target object from the second target object. For example, shoes that are not in the shoe cabinet in a home scene.

[0059] By acquiring environmental images through image acquisition devices such as cameras on the cleaning robot, second target objects that are not placed on the designated objects can be detected from the environmental images based on target detection. This avoids irrelevant objects interfering with the sorting process, accurately distinguishing shoes from other objects and shoes that have been properly placed. Thus, one of the second target objects is identified as the first target object, providing a target for subsequent sorting tasks and avoiding problems such as mis-picking or omissions, thereby improving the accuracy of the cleaning robot's sorting tasks.

[0060] In this way, an environmental image is acquired; a second target object that is not placed on the designated object is detected in the environmental image; and the detected second target object is identified as the first target object. Thus, by detecting the acquired environmental image, the second target object can be identified, and the first target object can be identified from the second target object, accurately distinguishing shoes from other objects and properly placed shoes. This provides a target for subsequent sorting tasks, avoiding problems such as accidental picking or omissions, and improving the accuracy of the cleaning robot's sorting tasks.

[0061] Please see Figure 3In some implementations, step S13 (the sub-step of determining the detected second target object as the first target object) includes: S131: In the case of detecting at least one second target object, the target object that is closest to the cleaning robot among the at least one second target object is identified as the first target object.

[0062] In some implementations, the determining module is further configured to, in the case of detecting at least one second target object, identify the target object closest to the cleaning robot among the at least one second target object as the first target object.

[0063] In some implementations, the processor is further configured to, in the case of detecting at least one second target object, identify the target object closest to the cleaning robot among the at least one second target object as the first target object.

[0064] Specifically, when at least one second target object is detected, the cleaning robot can calculate the distance to each second target object. By sorting all distance values, the target object closest to the cleaning robot among the at least one second target object can be identified as the first target object. For example, in a home setting, if multiple shoes are detected outside the shoe cabinet, the shoe closest to the cleaning robot can be identified as the execution target, i.e., the first target object. This provides a specific target for subsequent control of the cleaning robot's movement and avoids the robot moving back and forth between multiple distant targets, thereby improving the efficiency of the cleaning robot in performing tidying tasks.

[0065] Furthermore, if only one second target object is detected, that single second target object can be directly identified as the first target object.

[0066] Thus, when at least one second target object is detected, the target object closest to the cleaning robot among the at least one second target object is determined as the first target object. By determining the target object closest to the cleaning robot among the at least one second target object as the first target object, a specific target can be provided for subsequent control of the cleaning robot's movement and movement, and the robot can avoid moving back and forth between multiple distant targets, thereby improving the efficiency of the cleaning robot in performing its cleaning tasks.

[0067] Please see Figure 4 In some implementations, the sub-step of step S3 (identifying the current placement pose of the first target object) includes: S31: Obtain the first target image of the first target object; S32: Based on the first target image, identify the current placement posture of the first target object.

[0068] In some implementations, the recognition module is further configured to acquire a first target image of the first target object. The recognition module is also configured to identify the current placement pose of the first target object based on the first target image.

[0069] In some implementations, the processor is further configured to acquire a first target image of the first target object. The processor is also configured to identify the current placement pose of the first target object based on the first target image.

[0070] Specifically, the first target image refers to a close-up image focused on the first target object, which contains detailed feature information of the first target object and can provide a basic data source for identifying the current placement posture of the first target object.

[0071] When the cleaning robot moves within a first preset distance of the first target object, it can acquire a first target image focused on the first target object through an image acquisition device such as a camera to eliminate interference from other objects in the environment. The first target image accurately acquires the complete shape, texture, edge contour, and placement angle of the first target object, and then accurately calculates the current placement posture of the first target object based on the multi-dimensional features of the first target image. This provides accurate data support for determining the first posture of the first target object to be gripped, ensuring the reliability of the cleaning robot's gripping and placement.

[0072] Thus, a first target image of the first target object is acquired; based on the first target image, the current placement pose of the first target object is identified. In this way, by acquiring the first target image, the current placement pose of the first target object can be accurately calculated, providing accurate data support for subsequently determining the first pose for gripping the first target object, ensuring the reliability of the cleaning robot's gripping and placement.

[0073] Please see Figure 5 In some implementations, the sub-step of step S32 (identifying the current placement pose of the first target object based on the first target image) includes: S321: Perform image segmentation processing on the first target image to obtain segmented image blocks containing the first target object in the first target image; S322: Input the segmented image patch into the pre-trained placement pose prediction model to obtain the current placement pose output by the placement pose prediction model.

[0074] In some implementations, the recognition module performs image segmentation processing on the first target image to obtain segmented image blocks containing the first target object. The recognition module is also used to input the segmented image blocks into a pre-trained placement pose prediction model to obtain the current placement pose output by the placement pose prediction model.

[0075] In some implementations, the processor performs image segmentation on the first target image to obtain segmented image blocks containing the first target object. The processor is also configured to input the segmented image blocks into a pre-trained placement pose prediction model to obtain the current placement pose output by the placement pose prediction model.

[0076] Specifically, the segmented image block is the pixel region containing only the first target object obtained after image segmentation processing. Compared with the first target image, it can further remove background interference and fully preserve the shape outline, texture details, local structure and other features of the first target object, providing accurate data support for predicting the current placement pose.

[0077] The pre-trained placement pose prediction model is a deep learning model that identifies the position and orientation of an object in three-dimensional space. It can take a segmented image patch of the first target image as input, and output the 6-dimensional pose of the first target object, i.e., the current placement pose, through network layers such as feature extraction, feature matching, and pose calculation.

[0078] By performing image segmentation on the first target image, pixels related to the first target object can be extracted from the first target image to obtain segmented image blocks containing the first target object. This process removes background interference and provides clean data for the subsequent placement pose prediction model. The segmented image blocks are then input into the pre-trained placement pose prediction model, which outputs the current placement pose of the first target object, providing accurate data support for determining the first pose of the robotic arm gripping the first target object.

[0079] In one example, the acquired first target image can be preprocessed with noise reduction and color normalization to unify the image's brightness and contrast, reducing the impact of ambient light on feature extraction. Then, based on a 2D detection and segmentation network, semantic annotation is performed on the preprocessed image, dividing pixels into two categories: shoes and background, to obtain segmented image blocks containing only shoes. Next, the segmented image blocks are scaled, normalized, and adapted to meet the input format requirements of the placement pose prediction model. The adapted segmented image blocks are then input into the placement pose prediction model, which outputs and stores the three-dimensional position coordinates and three-dimensional rotation parameters of the first target object, i.e., the current placement pose. This provides accurate data support for subsequently determining the first pose of the robotic arm gripping the first target object and the target placement pose.

[0080] Thus, image segmentation is performed on the first target image to obtain segmented image blocks containing the first target object. These segmented image blocks are then input into a pre-trained placement pose prediction model to obtain the current placement pose output by the model. In this way, by performing image segmentation on the first target image, pixels related to the first target object can be extracted, resulting in segmented image blocks containing the first target object. This removes background interference, providing clean data for the subsequent placement pose prediction model. Inputting these segmented image blocks into the pre-trained model outputs the current placement pose of the first target object, providing accurate data support for determining the first pose of the robotic arm gripping the first target object.

[0081] Please see Figure 6 In some implementations, the sub-step of step S4 (determining the first pose of the robotic arm gripping the first target object) includes: S41: Obtain the first point cloud data of the first target object; S42: Determine the first pose of the robotic arm gripping the first target object based on the segmented image block and the first point cloud data.

[0082] In some implementations, the determining module is further configured to acquire first point cloud data of the first target object. The determining module is also configured to determine the first pose of the robotic arm gripping the first target object based on the segmented image blocks and the first point cloud data.

[0083] In some implementations, the processor is further configured to acquire first point cloud data of the first target object. The processor is also configured to determine the first pose of the robotic arm gripping the first target object based on the segmented image blocks and the first point cloud data.

[0084] Specifically, the first point cloud data is a set of three-dimensional point clouds containing the first target object and its surrounding environment, collected by the cleaning robot through point cloud acquisition devices such as binocular cameras, and is used to characterize the three-dimensional structure of the first target object and its environment.

[0085] Understandably, the segmented image patch contains two-dimensional planar features of the first target object, such as the shoe's outline, texture details, and local structure, while the first point cloud data contains three-dimensional features of the first target object that the segmented image patch cannot present, such as the shoe's thickness and heel height. This data can provide support for depth adaptation in obtaining the first pose of the robotic arm gripping the first target object.

[0086] Based on the planar features of the acquired segmented image blocks and the stereo features of the first point cloud data, combined with the stereo structure and placement of the first target object, the gripping position and gripping posture of the robotic arm for gripping the first target object can be determined, i.e., the first pose, to improve the gripping success rate and adaptability. For example, for shoes with protruding heels, the three-dimensional coordinates of the heel are determined based on the point cloud data, and the position of the robotic arm gripping from the side and rear is planned to avoid compressing and deforming the shoe body during gripping.

[0087] Thus, the first point cloud data of the first target object is obtained; based on the segmented image blocks and the first point cloud data, the first pose of the robotic arm in grasping the first target object is determined. In this way, by fusing the planar features obtained from the segmented image blocks and the stereo features of the first point cloud data, the first pose suitable for the robotic arm in grasping the first target object can be determined, thereby improving the grasping success rate and adaptability.

[0088] Please see Figure 7 In some implementations, the sub-step S42 (determining the first pose of the robotic arm gripping the first target object based on the segmented image patch and the first point cloud data) includes: S421: Segment the first point cloud data according to the segmented image blocks to determine the second point cloud data; S422: Input the current placement pose, segmented image block and second point cloud data into the pre-trained gripping pose prediction model to obtain the first pose output by the gripping pose prediction model.

[0089] In some implementations, the determining module is further configured to segment the first point cloud data based on the segmented image blocks to determine the second point cloud data. The determining module is also configured to input the current placement pose, the segmented image blocks, and the second point cloud data into a pre-trained gripping pose prediction model to obtain the first pose output by the gripping pose prediction model.

[0090] In some implementations, the processor is further configured to segment the first point cloud data based on the segmented image blocks to determine the second point cloud data. The processor is also configured to input the current placement pose, the segmented image blocks, and the second point cloud data into a pre-trained gripping pose prediction model to obtain the first pose output by the gripping pose prediction model.

[0091] Specifically, the second point cloud data is a set of three-dimensional point clouds that belongs only to the first target object, obtained by filtering the first point cloud data based on segmented image blocks. The point clouds corresponding to the background environment are removed, and it is the core data representing the three-dimensional structure of the target object.

[0092] The pre-trained gripping pose prediction model is a deep learning model used for robot grasping tasks, such as the Grasp Net model. It can take multi-source data, including the current placement pose, segmented image blocks, and second point cloud data, as input. Through feature fusion and stereo structure analysis, it outputs the gripping pose of the robotic arm of the cleaning robot for gripping the first target object, i.e., the first pose.

[0093] Understandably, the second point cloud data is a set of three-dimensional point clouds of the first target object that has been further removed from the background interference and is completely preserved, which can provide accurate data support for predicting the current placement pose.

[0094] By inputting the current placement pose, segmented image blocks, and second point cloud data into a pre-trained gripping pose prediction model, the gripping pose prediction model can determine the optimal gripping position and gripping posture (i.e., the first posture) for the robotic arm to grip the first target object, based on the planar features of the segmented image blocks and the three-dimensional features of the second point cloud data, combined with the current placement pose of the first target object, i.e., its three-dimensional structure and placement state, thereby improving the gripping success rate and adaptability.

[0095] Thus, the first point cloud data is segmented based on the segmented image blocks to determine the second point cloud data. The current placement pose, the segmented image blocks, and the second point cloud data are then input into a pre-trained gripping pose prediction model to obtain the first pose output by the model. This allows the gripping pose prediction model to determine the appropriate gripping position and posture (the first pose) for the robotic arm to grasp the first target object, based on the planar features of the segmented image blocks and the three-dimensional features of the second point cloud data, combined with the current placement pose (three-dimensional structure and placement state) of the first target object. This improves the gripping success rate and adaptability.

[0096] Please see Figure 8 In some implementations, the sub-step of step S6 (determining the target placement pose of the first target object on the placement object) includes: S61: Based on the segmented image blocks, detect the third target object on the placed object, wherein the first target object and the third target object are both one of a set of multiple objects; S62: Determine the target placement pose based on the detection results of the third target object.

[0097] In some implementations, the determining module is further configured to detect a third target object on the placement object based on the segmented image blocks, wherein both the first target object and the third target object are one of a set of multiple objects. The determining module is further configured to determine the target placement pose based on the detection result of the third target object.

[0098] In some implementations, the processor is further configured to detect a third target object on the placement object based on segmented image blocks, wherein both the first and third target objects are one of a set of multiple objects. The processor is also configured to determine the target placement pose based on the detection result of the third target object.

[0099] Specifically, the third target object refers to an object that belongs to the same set as the first target object. For example, if the first target object is a left shoe, the third target object is the corresponding right shoe.

[0100] Based on segmented image blocks, the cleaning robot can detect whether there is a third target object in the same set as the first target object, according to the feature information of the first target object extracted from the segmented image blocks. Then, based on the detection results, it determines the target placement pose. If the third target object exists, the target placement pose is determined by combining the placement pose of the third target object, such as placing the first target object next to the third target object to achieve set placement. If the third target object does not exist, the target placement pose is determined by combining the spatial layout of the placement objects, such as placing the first target object at intervals. This ensures that the target placement pose of the first target object can meet the matching placement requirements when the third target object exists, and can also adapt to the spatial layout of the placement objects when the third target object does not exist.

[0101] In one example, the feature information of the shoe can be determined based on the segmented image block of the shoe gripped by the robotic arm, i.e., the first target object. The target placement pose of the robotic arm for placing the shoe can be determined by detecting whether there is a similar shoe, i.e., a third target object, on the shoe cabinet, i.e. the placement object.

[0102] Thus, based on the segmented image blocks, a third target object is detected on the placement object. Both the first and third target objects are part of a set of multiple objects. Based on the detection result of the third target object, the target placement pose is determined. In this way, based on the segmented image blocks, the cleaning robot can detect whether there is a third target object belonging to the same set as the first target object in the placement object, and then determine the target placement pose based on the detection result. This ensures that the target placement pose of the first target object meets the matching placement requirements when a third target object is present, and adapts to the spatial layout of the placement object when a third target object is absent.

[0103] Please see Figure 9 In the implementation method, the sub-step of step S62 (determining the target placement pose based on the detection result of the third target object) includes: S621: When a third target object is detected, the target placement pose is determined according to the preset placement direction and the first placement position, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. S622: If no third target object is detected, determine the target placement posture according to the preset placement direction and the second placement position, wherein the distance between the second placement position and the placement position of the target object already placed on the placement object is a preset interval distance.

[0104] In some embodiments, the determining module is further configured to determine the target placement pose based on a preset placement direction and a first placement position when a third target object is detected, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. The determining module is further configured to determine the target placement pose based on a preset placement direction and a second placement position when no third target object is detected, wherein the distance between the second placement position and the placement position of an already placed target object on the placement object is a preset interval distance.

[0105] In some embodiments, the processor is further configured to, upon detecting a third target object, determine a target placement pose based on a preset placement direction and a first placement position, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. The processor is further configured to, upon not detecting a third target object, determine a target placement pose based on a preset placement direction and a second placement position, wherein the distance between the second placement position and the placement position of an already placed target object on the placement object is a preset interval distance.

[0106] Specifically, the preset placement direction is a uniform placement direction set in advance in the cleaning system of the cleaning robot. For example, it is parallel to the edge of the object being placed, such as a shoe cabinet, with the toes of the shoes facing the door, to ensure that all objects within the object are placed in the same orientation, thus improving aesthetics and ease of use.

[0107] The first placement position refers to the spatial position where the first target object should be placed on the placement object when a third target object is detected, and the distance between this position and the placement position of the third target object is less than a preset distance threshold. For example, if there are matching shoes in the shoe cabinet, the shoes that are picked up are placed next to the matching shoes.

[0108] The preset distance threshold is a pre-defined maximum distance value for placing matching objects close together, such as 3 cm or 5 cm, to ensure that the set of objects can be placed close together.

[0109] The second placement position refers to the spatial position on the placement object where the first target object should be placed if no third target object is detected. The distance between this position and the placement position of the target object already placed on the placement object is a preset interval distance. For example, if there are no matching shoes in the shoe cabinet, the shoes that are picked up are placed at a preset interval distance from other shoes.

[0110] The preset interval distance is a standard distance for placing non-matching objects in advance, ensuring that the objects are placed neatly and regularly, making it easy to retrieve and add more objects later.

[0111] If a third target object is detected, it can be assumed that there is an object on the object that is in a set with the first target object, i.e., the third target object. The first target object must be placed in a set with the third target object. Therefore, based on the preset placement direction and combined with the first placement position related to the placement position of the third target object, the target placement posture of the first target object is determined. If no third target object is detected, it can be assumed that there are no objects that match the first target object on the placement object. There may be objects that do not match the first target object, i.e., target objects have been placed. The first target object needs to be placed at intervals with these non-matching objects. Therefore, based on the preset placement direction and combined with the second placement position related to the placement position of the already placed target objects, the target placement posture of the first target object can be determined. This enables the standardized and orderly placement of the first target object, ensuring that the target placement posture of the first target object is consistent with the placement direction of other target objects. It also enables objects that match to be placed close together within a preset distance threshold and non-matching objects to be placed at intervals of a preset distance threshold, improving the orderliness and aesthetics of the arrangement.

[0112] In one example, if a pair of shoes (a third target object) similar to the shoes gripped by the robotic arm (a first target object) is detected on the shoe cabinet (the placement object), the position of the shoes gripped by the robotic arm can be determined based on the position of the pair of shoes and the predefined toe orientation. The shoes should be placed in the shoe cabinet with their toes facing the door, and the shoes should be placed 3 cm (a preset distance threshold) to the right of the pair of shoes. This is the target placement pose. If no pair of shoes similar to the shoes gripped by the robotic arm is detected on the shoe cabinet, but other shoes (already placed target objects) are present, the position of the shoes gripped by the robotic arm can be determined based on the position of the other shoes and the predefined toe orientation. The shoes should be placed 10 cm (a preset interval distance) away from the other shoes. This is the target placement pose.

[0113] Thus, when a third target object is detected, the target placement posture is determined based on the preset placement direction and the first placement position, wherein the distance between the first placement position and the placement position of the third target object is less than a preset distance threshold. When no third target object is detected, the target placement posture is determined based on the preset placement direction and the second placement position, wherein the distance between the second placement position and the placement positions of target objects already placed on the object being placed is a preset interval distance. This allows for the standardized and orderly placement of the first target object, ensuring that the target placement posture of the first target object is consistent with the placement direction of other target objects, and enabling sets of objects to be placed close together within a preset distance threshold, while non-set objects are placed at preset distance threshold intervals, improving the orderliness and aesthetics of the arrangement.

[0114] Please see Figure 10 In some implementations, the sub-step of step S61 (detecting a third target object on the placement object based on the segmented image patch) includes: S611: Perform feature information extraction processing on the segmented image block corresponding to the first target object to determine the first feature information corresponding to the first target object; S612: Determine the similarity between the first feature information and the second feature information of the fourth target object already placed on the placement object; S613: If the similarity is greater than a preset threshold, the fourth target object is identified as the third target object.

[0115] In some embodiments, the determining module is further configured to perform feature information extraction processing on the segmented image block corresponding to the first target object, and determine the first feature information corresponding to the first target object. The determining module is further configured to determine the similarity between the first feature information and the second feature information of the fourth target object already placed on the placement object. The determining module is further configured to determine the fourth target object as the third target object if the similarity is greater than a preset threshold.

[0116] In some embodiments, the processor is further configured to perform feature information extraction processing on the segmented image block corresponding to the first target object to determine first feature information corresponding to the first target object. The processor is further configured to determine the similarity between the first feature information and the second feature information of a fourth target object already placed on the placement object. The processor is further configured to determine the fourth target object as a third target object if the similarity is greater than a preset threshold.

[0117] Specifically, the first feature information is a feature vector extracted from the segmented image block corresponding to the first target object, such as a 512-dimensional vector, which contains multi-dimensional features such as the shape, texture, size, and proportion of the first target object.

[0118] The fourth target object refers to the objects that have already been placed in the placement object, such as the shoes that have been neatly arranged in the shoe cabinet.

[0119] The second feature information is the feature vector extracted from the segmented image block corresponding to the fourth target object. Unlike the first feature information, the second feature information can be stored during the previous placement of the fourth target object, or it can be obtained by the cleaning robot through feature information extraction processing of the image of the fourth target object.

[0120] Similarity is the degree of matching between the first feature information and the second feature information. The higher the similarity, the more similar the features of the first target object and the fourth target object corresponding to the second feature information are, and the more likely they are to be a matching object.

[0121] The preset threshold is the critical value for determining the similarity between the first target object corresponding to the first feature information and the second target object corresponding to the second feature information as a matching object.

[0122] By extracting feature information from the segmented image blocks corresponding to the first target object, the two-dimensional image information can be transformed into a high-dimensional feature vector to determine the first feature information corresponding to the first target object. Then, by comparing the similarity between the first feature information and the second feature information, the fourth target object with a similarity greater than a preset threshold can be identified as the third target object that completes the set with the first target object. This achieves accurate identification of the matching objects, provides a data foundation for determining the target placement pose of the first target object, and supports the standardized and orderly placement of the first target object.

[0123] In one example, feature information can be extracted from the segmented image block corresponding to the grabbed shoe (the first target object) based on a Person Re-Identification (ReID) model, generating and storing a corresponding 512-dimensional vector, i.e., the first feature information. Based on this 512-dimensional vector of the grabbed shoe, by comparing it with the feature vectors of shoes already placed on the shoe rack (the second feature information) stored in advance, it can be determined whether there is a pair of shoes with the grabbed shoe, i.e., whether there is a third target object among the fourth target objects that matches the first target object.

[0124] In this way, feature information extraction processing is performed on the segmented image block corresponding to the first target object to determine the first feature information corresponding to the first target object; the similarity between the first feature information and the second feature information of the fourth target object already placed on the object is determined; if the similarity is greater than a preset threshold, the fourth target object is determined as the third target object. Thus, by performing feature information extraction processing on the segmented image block corresponding to the first target object, two-dimensional image information can be transformed into a high-dimensional feature vector to determine the first feature information corresponding to the first target object. Then, by comparing the similarity between the first and second feature information, the fourth target object with a similarity greater than a preset threshold is determined as the third target object paired with the first target object, achieving accurate identification of the matching object. This provides a data foundation for subsequently determining the target placement pose of the first target object and supports the standardized and orderly placement of the first target object.

[0125] Please see Figure 11 In some implementations, the method further includes: S9: Identify the placement status of the target object already placed on the placement object; S10: When a fifth target object is identified in the first placement state, the robotic arm is controlled to place the fifth target object a second time so that the placement state of the fifth target object changes from the first placement state to the second placement state.

[0126] In some implementations, the identification module is also used to identify the placement state of the target object already placed on the placement object. The control module is also used to control the robotic arm to perform a second placement of the fifth target object when a fifth target object is identified as having a first placement state, so that the placement state of the fifth target object switches from the first placement state to the second placement state.

[0127] In some implementations, the processor is further configured to identify the placement state of the target object already placed on the placement object. The processor is also configured to, when a fifth target object is identified as having a first placement state, control the robotic arm to perform a secondary placement of the fifth target object, thereby switching the placement state of the fifth target object from the first placement state to the second placement state.

[0128] Specifically, the placement state refers to the degree of orderliness of the placed target object, which can be divided into the first placement state and the second placement state.

[0129] The first placement state refers to an irregular state that does not conform to the preset placement standard, including but not limited to target objects that are in a set but placed outside the preset distance threshold, target objects that are not in a set but placed within the preset distance threshold, target objects that are not in a set but placed within the preset interval distance, and target objects that do not conform to the preset placement direction, such as pairs of shoes that are not placed close together, pairs of shoes that are placed close together, and shoes with a deviation in the direction of the toe. The second placement state is one that conforms to the preset placement standards, such as being placed in pairs next to each other, facing the same direction, and in a compliant position.

[0130] The fifth target object is the target object that is identified as being in the first placement state, which is the target object that needs to be placed a second time.

[0131] By detecting the feature information, similarity of feature information, placement posture, and shape of the target objects already placed on the object, the placement status of the target objects already placed on the object can be identified.

[0132] When a fifth target object is identified as being in its first placement state, it can be assumed that the fifth target object needs to be placed a second time. By identifying the current placement posture of the fifth target object, the gripping posture of the robotic arm is determined. Based on placement standard parameters such as preset placement direction, preset distance threshold, and preset interval distance, the placement posture of the fifth target object is determined, and then the fifth target object is placed a second time to switch its placement state from the first placement state to the second placement state. This achieves placement correction of the target object on the placement object, improving the sorting effect and user experience.

[0133] In addition, after the second placement, the placement status of the target object already placed on the placement object can be identified again. If it has switched to the second placement status, the placement correction ends; if the placement status of the fifth target object is still the first placement status, the second placement can be performed until the placement status of the fifth target object is the second placement status.

[0134] In one example, the cleaning robot can identify the placement status of shoes (target objects) in a shoe cabinet (placement object). If it identifies a shoe that is not placed correctly, i.e., a shoe in the first placement state (the fifth target object), it will reposition the shoe until the shoe is placed correctly. For example, it can re-pair and identify mismatched shoes and place them in paired or spaced positions, or it can correct shoes with the wrong toe direction so that the toe faces the door like other shoes.

[0135] In this way, the system identifies the placement status of the target objects already placed on the placement platform. When a fifth target object is identified in its first placement state, the robotic arm is controlled to perform a second placement of the fifth target object, switching its placement status from the first to the second. By identifying the placement status and correcting the fifth target object's placement from the first to the second, standardized and orderly placement of target objects on the placement platform can be achieved, improving the organization efficiency and user experience to some extent.

[0136] Please see Figure 12 In some implementations, the method further includes: S11: After the number of consecutive times of performing secondary placement on the fifth target object reaches a preset number, if the placement state of the fifth target object is the first placement state, obtain the second target image of the placement object; S12: Upon receiving an image upload permission instruction, a second target image is sent to the server, wherein the server generates a target instruction based on the second target image and sends the target instruction to the cleaning robot; S13: Based on the received target instruction, control the cleaning robot to move the fifth target object, so that the placement state of the fifth target object changes from the first placement state to the second placement state; and / or, S14: Update the cleaning robot according to the received target instructions.

[0137] In some implementations, the control module is further configured to, after performing a preset number of consecutive secondary placements on the fifth target object, acquire a second target image of the object if the placement state of the fifth target object is the first placement state. The control module is also configured to, upon receiving an image upload permission instruction, send the second target image to the server, wherein the server generates a target instruction based on the second target image and sends the target instruction to the cleaning robot. The control module is further configured to, according to the received target instruction, control the cleaning robot to move the fifth target object, thereby switching the placement state of the fifth target object from the first placement state to the second placement state. The control module is also configured to, according to the received target instruction, update the cleaning robot.

[0138] In some embodiments, the processor is further configured to, after performing a preset number of consecutive secondary placements on the fifth target object, acquire a second target image of the placed object if the placement state of the fifth target object is the first placement state. The processor is further configured to, upon receiving an image upload permission instruction, send the second target image to the server, wherein the server generates a target instruction based on the second target image and sends the target instruction to the cleaning robot. The processor is further configured to, according to the received target instruction, control the cleaning robot to move the fifth target object, so that the placement state of the fifth target object switches from the first placement state to the second placement state. The processor is further configured to, according to the received target instruction, update the cleaning robot.

[0139] Specifically, the preset number of times is the critical number of consecutive corrections set in advance by the cleaning robot. If the preset number of consecutive times of performing secondary placement on the fifth target object reaches the preset number, it can be considered that performing secondary placement cannot correct the placement posture of the fifth target object from the first placement state to the second placement state, and there is no need to continue performing secondary placement.

[0140] The second target image refers to the panoramic image collected by the cleaning robot, which includes the fifth target object and its surrounding environment, and is used by the server to analyze the cause of the failure.

[0141] The image upload permission instruction is an authorization instruction confirmed by the user, allowing the robot to upload a second target image. This can be done through confirmation via the cleaning robot's corresponding application or voice authorization, and is used to protect user privacy and security.

[0142] The target instructions include server-generated instructions for correcting the placement posture of the fifth target object from the first placement state to the second placement state, and may also include instructions for updating the parameters of the cleaning robot.

[0143] After performing a second placement on the fifth target object a preset number of times, if the fifth target object is in the first placement state, it can be considered that performing a second placement cannot correct the placement posture of the fifth target object from the first placement state to the second placement state, and there is no need to continue performing a second placement.

[0144] Upon receiving an image upload permission instruction, the acquired second target image can be sent to the server. The server then analyzes the information in the second target image to determine why the placement posture of the fifth target object cannot be corrected. It then generates and sends a target instruction to the cleaning robot. This allows the cleaning robot to move the fifth target object according to the received instruction, switching its placement from the first to the second state. Furthermore, the cleaning robot can update itself based on the received instruction. This approach, while ensuring user privacy and security, resolves the issue of failed secondary placement and achieves standardized and orderly placement of target objects, thereby improving organization efficiency and user experience to some extent.

[0145] Understandably, if the placement of the fifth target object cannot be corrected during the secondary placement, the cleaning robot can upload an image containing the fifth target object and the current scene to the server upon receiving an image upload permission command. The server can then use more complex AI algorithms and large models to verify the placement results, analyze the causes of failures, and optimize the model parameters in the cleaning robot. This allows the generation of targeted instructions. If the problem is at the operational level, such as improper gripping position, specific placement adjustment instructions are generated. If the problem is a model deviation issue in the cleaning robot, such as inaccurate recognition results from the placement pose prediction model and the gripping pose prediction model, then a model parameter update instruction for the cleaning robot is generated. This ensures that the cleaning robot can make targeted adjustments based on the received target instructions, such as adjusting the parameters of the placement pose prediction model and the gripping pose prediction model. This optimizes the cleaning robot, improves the recognition accuracy of the placement pose prediction model and the gripping pose prediction model, and thus solves the problem of secondary placement failure. Ultimately, this achieves standardized and orderly placement of the target objects on the placement object, improving the organization effect and user experience to a certain extent.

[0146] Thus, after performing a preset number of consecutive repositioning operations on the fifth target object, if the fifth target object is in its first placement state, a second target image of the object is acquired. Upon receiving an image upload permission command, the second target image is sent to the server. The server generates target instructions based on the second target image and sends these instructions to the cleaning robot. Based on the received target instructions, the cleaning robot moves the fifth target object to switch its placement state from the first to the second. The cleaning robot is then updated based on the received target instructions. This allows the server to analyze the information in the second target image to determine why the placement of the fifth target object cannot be corrected, and to generate and send target instructions to the cleaning robot. This enables the cleaning robot to move the fifth target object based on the received target instructions, switching its placement state from the first to the second. Furthermore, the cleaning robot can update itself based on the received target instructions. This approach, while ensuring user privacy and security, solves the problem of failed repositioning and achieves standardized and orderly placement of target objects, improving organization and user experience to some extent.

[0147] Please see Figure 13 In some implementations, step S5 (controlling the cleaning robot to move to the target area) includes the following sub-steps: S51: Determine the movement path of the cleaning robot to the target area based on the current placement posture and the target placement posture; S52: Control the cleaning robot to move along the movement path to the target area.

[0148] In some implementations, the determining module is further configured to determine a movement path for the cleaning robot to move to the target area based on the current placement pose and the target placement pose. The control module is further configured to control the cleaning robot to move along the movement path to the target area.

[0149] In some implementations, the processor is further configured to determine a movement path for the cleaning robot to move to the target area based on the current placement pose and the target placement pose. The processor is also configured to control the cleaning robot to move along the movement path to the target area.

[0150] Specifically, the movement path refers to the optimal route for the robot to travel from the position corresponding to the current pose of the first target object to the position corresponding to the target pose, which includes a series of continuous spatial coordinate points and turning commands.

[0151] Based on the current and target placement positions, the straight-line distance and approximate direction of the cleaning robot moving to the target area can be calculated. Combined with scene information, such as information about fixed obstacles, the optimal driving route, i.e., the movement path, is generated to control the cleaning robot to move along the movement path to the target area and achieve object sorting.

[0152] In one example, the cleaning robot can calculate the straight-line distance and approximate direction to the target area based on its current and target placement positions. It can also combine information about fixed obstacles in the home environment to plan multiple routes, select the optimal route as the movement path, and use sensors to perceive the environment, avoid dynamic obstacles, and change the movement path in real time. Finally, it can control the cleaning robot to move along the movement path to the target area.

[0153] Thus, based on the current and target placement postures, the movement path of the cleaning robot to the target area is determined; the cleaning robot is then controlled to move along the movement path to the target area. In this way, based on the current and target placement postures, the straight-line distance and approximate direction of the cleaning robot's movement to the target area can be calculated. Combined with scene information, such as information about fixed obstacles, an optimal travel route, i.e., the movement path, is generated to control the cleaning robot to move along the movement path to the target area, thereby achieving object sorting.

[0154] This application also provides a cleaning system, including a base station and a cleaning robot.

[0155] This application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the control method for the cleaning robot described above.

[0156] It is understood that a computer program includes computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0157] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0158] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of executable request code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0159] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A control method of a cleaning robot, characterized by, A method for a cleaning robot, the cleaning robot comprising a mechanical arm capable of performing a picking action, the method comprising: S1: determining a first target object; S2: controlling the cleaning robot to move to within a first preset distance of the first target object; S3: identifying a current placement pose of the first target object; S4: determining a first pose of the mechanical arm picking the first target object; S5: controlling the mechanical arm to pick the first target object at the first pose, and controlling the cleaning robot to move to a target area; S6: determining a target placement pose of the first target object on a placement object; S7: determining a second pose of the mechanical arm placing the first target object according to the first pose and the target placement pose; S8: controlling the mechanical arm to place the first target object at the target placement pose at the second pose.

2. The method of claim 1, wherein, The sub-step of determining the first target object comprises: obtaining an environment image; detecting a second target object in the environment image that is not placed on the placement object; determining the detected second target object as the first target object.

3. The method of claim 2, wherein, The sub-step of determining the detected second target object as the first target object comprises: in the case of detecting at least one second target object, determining a target object closest to the cleaning robot among the at least one second target object as the first target object.

4. The method of claim 1, wherein, The sub-step of identifying the current placement pose of the first target object comprises: obtaining a first target image of the first target object; identifying the current placement pose of the first target object according to the first target image.

5. The method of claim 4, wherein, The sub-step of identifying the current placement pose of the first target object according to the first target image comprises: performing image segmentation processing on the first target image to obtain a segmentation image block containing the first target object in the first target image; inputting the segmentation image block into a pre-trained placement pose prediction model to obtain the current placement pose output by the placement pose prediction model.

6. The method of claim 5, wherein, The sub-step of determining the first pose of the mechanical arm picking the first target object comprises: obtaining first point cloud data of the first target object; determining the first pose of the mechanical arm picking the first target object according to the segmentation image block and the first point cloud data.

7. The method of claim 6, wherein, The sub-step of determining the first pose of the mechanical arm picking the first target object according to the segmentation image block and the first point cloud data comprises: segmenting the first point cloud data according to the segmentation image block to determine second point cloud data; inputting the current placement pose, the segmentation image block, and the second point cloud data into a pre-trained picking pose prediction model to obtain the first pose output by the picking pose prediction model.

8. The method of claim 5, wherein, The sub-step of determining the target placement pose of the first target object on the placement object comprises: detecting a third target object on the placing object according to the segmented image block, wherein the first target object and the third target object are both one of a plurality of objects constituting a complete set; determining the target placing pose according to a detection result of the third target object.

9. The method of claim 8, wherein, The sub-step of determining the target placing pose according to the detection result of the third target object comprises: in a case where the third target object is detected, determining the target placing pose according to a preset placing direction and a first placing position, wherein a distance between the first placing position and a placing position of the third target object is less than a preset distance threshold; in a case where the third target object is not detected, determining the target placing pose according to a preset placing direction and a second placing position, wherein a distance between the second placing position and a placing position of a target object already placed on the placing object is a preset interval distance.

10. The method of claim 8, wherein, The sub-step of detecting the third target object on the placing object according to the segmented image block comprises: performing feature information extraction processing on the segmented image block corresponding to the first target object to determine first feature information corresponding to the first target object; determining a similarity degree between the first feature information and second feature information of a fourth target object already placed on the placing object; in a case where the similarity degree is greater than a preset threshold, determining the fourth target object as the third target object.

11. The method of claim 1, wherein, The method further comprises: recognizing a placing state of a target object already placed on the placing object; in a case where a fifth target object in a first placing state is recognized, controlling the mechanical arm to perform secondary placing on the fifth target object to switch the placing state of the fifth target object from the first placing state to a second placing state.

12. The method of claim 11, wherein, The method further comprises: after a preset number of times of performing secondary placing on the fifth target object, if the placing state of the fifth target object is the first placing state, acquiring a second target image of the placing object; in a case where an image upload permission instruction is acquired, sending the second target image to a server, wherein the server generates a target instruction according to the second target image and sends the target instruction to the cleaning robot; controlling the cleaning robot to move the fifth target object according to the received target instruction to switch the placing state of the fifth target object from the first placing state to the second placing state; and / or updating the cleaning robot according to the received target instruction.

13. The method of claim 1, wherein, The sub-step of controlling the cleaning robot to move to the target region comprises: determining a moving path of the cleaning robot moving to the target region according to the current placing pose and the target placing pose; controlling the cleaning robot to move to the target region along the moving path.

14. A cleaning robot, characterized in that, A device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-13.

15. A cleaning system characterized by, A device comprising a base station and the cleaning robot of claim 14.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program which, when executed by one or more processors, implements the method of any one of claims 1-13.

17. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the method of any one of claims 1-13.

Images