Article grabbing control device and method and self-moving cleaning equipment

Abstract

The invention provides an article grabbing control device and method and self-moving cleaning equipment, and relates to the technical field of cleaning equipment. And under the condition that the mechanical arm is in the warehouse-out state, the environment image, collected by the image collecting component, in front of the moving path of the self-moving cleaning equipment is received, and under the condition that the mechanical arm is in the warehouse-out state, the mechanical arm is higher than the machine body, and the full view of the articles can be reflected in the environment image. According to the environment image, a grabbing point on the article is recognized, coordinates of the grabbing point of the article in an image acquisition component coordinate system are determined, pose data of the tail end of the mechanical arm are obtained, and coordinates of the grabbing point of the article in a preset machine body coordinate system are determined according to the pose data and the coordinates of the grabbing point of the article in the image acquisition component coordinate system; according to the coordinate of the grabbing point under the coordinate system of the machine body, the mechanical arm is controlled to move to one side of the grabbing point, and when the clamping jaw is controlled to grab the article through the grabbing point, the article is not prone to falling off from the clamping jaw.

Description

Technical Field

[0001] The present application relates to the technical field of cleaning equipment, and in particular to an object grabbing control device, method and self-moving cleaning equipment. Background Art

[0002] With the improvement of living standards and the development of science and technology, consumers' demand for self-mobile cleaning equipment continues to increase; especially in the field of smart homes, self-mobile cleaning equipment has gradually become the mainstream choice for household cleaning.

[0003] At present, the self-moving cleaning device includes a body for cleaning and a mechanical arm with adjustable posture for grasping. When the body of the self-moving cleaning device is in the process of cleaning the ground, the image acquisition component set on the body can detect the road conditions in front of the moving path. If an object (such as a shoe, a plastic stool, or a paper ball) is detected in front of the moving path, the grasping point on the object that is easy to grasp is identified. Then, the gripper on the mechanical arm is controlled to move the object to one side of the moving path.

[0004] However, since the position of the fuselage is usually low, the image acquisition component usually cannot capture the full view of high objects (such as boots and plastic stools), resulting in inaccurate grasping points. In turn, the objects are prone to falling when being grasped, and the reliability is poor. Summary of the invention

[0005] The present application provides an article grabbing control device, method and self-moving cleaning equipment, which are used to solve the problem of inaccurate recognition of grabbing points on articles in the prior art, which further causes the articles to fall easily when being grabbed and has poor reliability.

[0006] In a first aspect, the present application provides an object grabbing control method, which is applied to a self-moving cleaning device, wherein the self-moving cleaning device comprises a body and a mechanical arm, wherein the starting end of the mechanical arm is connected to the body, and the end of the mechanical arm is provided with a clamping claw and an image acquisition component. The method provided by the present application comprises:

[0007] When the robot arm is in the out-of-warehouse state, receiving an environmental image in front of the moving path of the self-moving cleaning device collected by the image collection component;

[0008] According to the environment image, the grasping point on the object is identified, and the coordinates of the grasping point of the object in the coordinate system of the image acquisition component are determined;

[0009] Acquire the posture data of the end of the robot arm, and determine the coordinates of the grasping point of the object in the preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the image acquisition component coordinate system;

[0010] According to the coordinates of the grasping point in the fuselage coordinate system, the robot arm is controlled to move to one side of the grasping point, and the gripper is controlled to grasp the object through the grasping point.

[0011] In some embodiments, determining the coordinates of the grabbing point of the object in the body coordinate system according to the posture data and the coordinates of the grabbing point of the object in the image acquisition component coordinate system includes:

[0012] Determine a first transformation matrix from a fuselage coordinate system to an image acquisition component coordinate system according to the position and posture data;

[0013] The coordinates of the grabbing point of the object in the coordinate system of the fuselage are determined according to the coordinates of the grabbing point of the object in the coordinate system of the image acquisition component and the first conversion matrix.

[0014] In some embodiments, the mechanical arm includes a plurality of rotatable mechanical axes connected in sequence, the gripper and the image acquisition component are arranged at the end of the last mechanical axis, each mechanical axis corresponds to a mechanical axis coordinate system, each mechanical axis coordinate system takes the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis coordinate system, the posture data includes a first angle between the first projection of each mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, a first distance between the end of the first projection and the X-axis, a second angle between the first projection and the Z-axis, and a second distance between the end of the first projection and the Z-axis, and according to the posture data, determining a first conversion matrix from the fuselage coordinate system to the image acquisition component coordinate system, including:

[0015] For each mechanical axis, construct a sub-conversion matrix according to the first angle, the first distance, the second angle, and the second distance corresponding to the mechanical axis;

[0016] According to the formula T = A 1 ×A 2 ×...×A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system, where T is the first transformation matrix, A 1 is the sub-transformation matrix corresponding to the first mechanical axis, A 2 is the sub-transformation matrix corresponding to the second mechanical axis, A n It is the sub-transformation matrix corresponding to the nth mechanical axis.

[0017] In some implementations, for each mechanical axis, a sub-conversion matrix is ​​constructed according to a first angle, a first distance, a second angle, and a second distance corresponding to the mechanical axis, including:

[0018] According to the formula Construct the sub-transformation matrix, A iis the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

[0019] In some embodiments, obtaining the position and posture data of the end of the robotic arm includes:

[0020] Obtaining the rotation angle of the electric joint at the starting end of each rotatable mechanical axis;

[0021] According to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table.

[0022] In some embodiments, determining the coordinates of the grabbing point of the object in the image acquisition component coordinate system includes:

[0023] According to the position of the grasping point of the object in the environment image, the pixel coordinates of the grasping point in the environment image are determined;

[0024] According to the preset second conversion matrix, the pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system.

[0025] In some embodiments, the pixel coordinates of the grasping point in the environment image are transformed according to a preset second transformation matrix to obtain the coordinates of the grasping point in the image acquisition component coordinate system, including:

[0026] According to the formula The pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system, where x 1 is the coordinate of the grabbing point on the X axis of the image acquisition component coordinate system, y 1 is the coordinate of the grabbing point on the Y axis of the image acquisition component coordinate system, z 1 is the coordinate of the grabbing point on the Z axis of the image acquisition component coordinate system, is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, and f x is the horizontal focal length of the lens of the image acquisition component, f y is the focal length of the lens of the image acquisition component in the vertical direction, cx is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c yIt is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction.

[0027] In some embodiments, or, u 1 is the coordinate of the grab point on the X axis of the environment image, v 1 is the coordinate of the grab point on the Y axis of the environment image, w 1 is the depth coordinate of the grasping point in the environment image.

[0028] In some implementations, identifying a grasping point on an object based on an environment image includes:

[0029] When it is recognized that there are objects in the environment image, the grasping points of the objects in the environment image are identified based on a pre-trained grasping point recognition model, wherein the grasping point recognition model is trained based on multiple training samples input into the network to be trained, and each training sample includes a historical object image marked with a standard grasping point.

[0030] In some implementations, identifying a grasping point on an object based on an environment image includes:

[0031] Based on the preset machine vision algorithm, the object’s grabbing point is determined.

[0032] In some embodiments, the gripping point is the point where the center of mass of the object is projected onto the surface of the object in the gripping direction of the gripper.

[0033] In a second aspect, the present application further provides an article grabbing control device, which is configured on a self-moving cleaning device. The self-moving cleaning device includes a body and a mechanical arm. The starting end of the mechanical arm is connected to the body, and the end of the mechanical arm is provided with a clamp and an image acquisition component. The device provided by the present application includes:

[0034] A data receiving unit, used for receiving an image of the environment in front of the moving path of the self-moving cleaning device collected by the image collecting component when the robot arm is in the out-of-warehouse state;

[0035] A grasping point recognition unit, used to recognize the grasping points on the object according to the environment image, and determine the coordinates of the grasping points of the object in the coordinate system of the image acquisition component;

[0036] A coordinate conversion unit, used to obtain the posture data of the end of the robot arm, and determine the coordinates of the grasping point of the object in a preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the image acquisition component coordinate system;

[0037] The grabbing control unit is used to control the robot arm to move to one side of the grabbing point according to the coordinates of the grabbing point in the fuselage coordinate system, and control the gripper to grab the object through the grabbing point.

[0038] In some embodiments, the grasping point identification unit is specifically used to

[0039] Determine a first transformation matrix from a fuselage coordinate system to an image acquisition component coordinate system according to the position and posture data;

[0040] The coordinates of the grabbing point of the object in the coordinate system of the fuselage are determined according to the coordinates of the grabbing point of the object in the coordinate system of the image acquisition component and the first conversion matrix.

[0041] In some embodiments, the mechanical arm includes a plurality of rotatable mechanical axes connected in sequence, the gripper and the image acquisition component are arranged at the end of the last mechanical axis, each mechanical axis corresponds to a mechanical axis coordinate system, each mechanical axis coordinate system takes the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis coordinate system, and the posture data includes a first angle between a first projection of each mechanical axis on a vertical plane and the X-axis of the corresponding mechanical axis coordinate system, a first distance between the end of the first projection and the X-axis, a second angle between the first projection and the Z-axis, and a second distance between the end of the first projection and the Z-axis.

[0042] A coordinate conversion unit, specifically configured to construct a sub-conversion matrix for each mechanical axis according to a first angle, a first distance, a second angle, and a second distance corresponding to the mechanical axis;

[0043] According to the formula T = A 1 ×A 2 ×...×A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system, where T is the first transformation matrix, A 1 is the sub-transformation matrix corresponding to the first mechanical axis, A 2 is the sub-transformation matrix corresponding to the second mechanical axis, A n It is the sub-transformation matrix corresponding to the nth mechanical axis.

[0044] In some embodiments, the coordinate conversion unit is specifically used to

[0045] According to the formula Construct the sub-transformation matrix, A i is the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

[0046] In some embodiments, the coordinate conversion unit is specifically used to obtain the rotation angle of the electric joint at the starting end of each rotatable mechanical axis;

[0047] According to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table.

[0048] In some embodiments, the coordinate conversion unit is specifically used to determine the pixel coordinates of the grasping point in the environment image according to the position of the grasping point of the object in the environment image;

[0049] According to the preset second conversion matrix, the pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system.

[0050] In some embodiments, the coordinate conversion unit is specifically used to

[0051] According to the formula The pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system, where x 1 is the coordinate of the grabbing point on the X axis of the image acquisition component coordinate system, y 1 is the coordinate of the grabbing point on the Y axis of the image acquisition component coordinate system, z 1 is the coordinate of the grabbing point on the Z axis of the image acquisition component coordinate system, is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, c x is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c y It is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction.

[0052] In some embodiments, the grasping point recognition unit is specifically used to identify the grasping points of objects in the environment image based on a pre-trained grasping point recognition model when the existence of objects in the environment image is recognized, wherein the grasping point recognition model is trained based on multiple training samples input into the network to be trained, and each training sample includes a historical object image marked with a standard grasping point.

[0053] In some embodiments, the grasping point identification unit is specifically configured to determine the grasping point of the object based on a preset machine vision algorithm.

[0054] In the second aspect, the present application also provides a self-moving cleaning device, including a main controller, a body and a robotic arm, wherein the starting end of the robotic arm is connected to the body, and the end of the robotic arm is provided with a clamp and an image acquisition component, wherein the main controller is configured to execute to implement the object grabbing control method provided in the first aspect of the present application.

[0055] In a third aspect, the present application further provides a computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the self-moving cleaning device, the self-moving cleaning device can execute an object grabbing control method as provided in the first aspect.

[0056] In a fourth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, is used to implement an object grasping control method as provided in the first aspect.

[0057] The present application provides an article grabbing control device, method and self-moving cleaning equipment, which receives an environmental image in front of the moving path of the self-moving cleaning equipment collected by an image acquisition component when the mechanical arm is in the out-of-warehouse state. It can be understood that when the mechanical arm is in the out-of-warehouse state, the height of the mechanical arm is higher than the height of the fuselage, and even if the height of the object is high, the full view of the object can be reflected in the environmental image. According to the environmental image, the grabbing point on the object is identified, and the coordinates of the grabbing point of the object in the image acquisition component coordinate system are determined. Since the full view of the object can be reflected in the environmental image, the accuracy of the identified grabbing point on the object is high, and the accuracy of the coordinates of the determined grabbing point in the image acquisition component coordinate system is also high. The posture data of the end of the mechanical arm is obtained, and the coordinates of the grabbing point of the object in the preset fuselage coordinate system are determined according to the posture data and the coordinates of the grabbing point of the object in the image acquisition component coordinate system. In this way, the coordinates of the grabbing point in the image acquisition component coordinate system can be accurately converted to the coordinates in the fuselage coordinate system. Furthermore, according to the coordinates of the grasping point in the fuselage coordinate system, the robotic arm is controlled to move to one side of the grasping point, and the gripper is controlled to grasp the object through the grasping point, so that the object is not easy to fall off the gripper. In this way, the gripper can accurately and reliably place the object in a position away from the moving path to achieve obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1A schematic diagram of the structure of a self-moving cleaning device provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the structure of the body of the self-moving cleaning device provided in an embodiment of the present application;

[0061] Figure 3 A flowchart of an object grabbing control method provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of a robotic arm provided in an embodiment of the present application;

[0063] Figure 5 A block diagram of the functional modules of the object grabbing control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0065] Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may further design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0066] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element may be directly on the other layer / element or an intervening layer / element may exist therebetween. In addition, if a layer / element is "on" another layer / element in one orientation, the layer / element may be "below" the other layer / element when the orientation is reversed.

[0067] The following is a detailed description of the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Explanation of terms involved in this application:

[0069] Self-moving cleaning equipment: an electronic device that automatically cleans, which can be a sweeping robot, a mopping robot, a sweeping and mopping robot, etc., without limitation. Figure 1 As shown, the self-moving cleaning device 101 provided in the embodiment of the present application mainly includes two parts, specifically including a body 102 and a mechanical arm 103 arranged on the body 102 .

[0070] The mechanical arm 103 is arranged on the fuselage 102. The mechanical arm 103 is a mechanical axis module that can realize multi-degree-of-freedom posture adjustment. The end of the mechanical arm 103 is set at the clamp 104, and the clamp 104 can grab or push some small objects. For example, during the process of cleaning or mopping the floor by the self-mobile cleaning device 101, socks or small toys on the ground are detected, and the mechanical arm 103 can be used to grab the socks or small toys and move them to the corresponding storage area. The specific structure of the mechanical arm 103 is not limited in the embodiment of the present application.

[0071] The self-moving cleaning device 101 also includes a housing chamber 106 for accommodating the robot arm 103. When the robot arm 103 is not in use, the robot arm 103 can be stored in the housing chamber 106 to minimize the space occupied by the self-moving cleaning device 101 and the robot arm 103. At this time, the robot arm 103 is in a warehoused state. Figure 2 As shown in FIG. 1 , the shaded area on the surface of the self-mobile cleaning device 101 is the area corresponding to the accommodating cavity 106, and the robot arm 103 is accommodated in the accommodating cavity 106. When the robot arm 103 is needed, the robot arm 103 is extended from the accommodating cavity 106 to obtain the following Figure 1 In the state shown, the robot arm 103 is in the out-of-warehouse state. It can be seen that the height of the robot arm 103 in the out-of-warehouse state is higher than the height of the fuselage 102. Figure 1 and Figure 2 In addition to arranging the robotic arm 103 on the top of the self-moving cleaning device 101, the robotic arm 103 can also be arranged on the side of the self-moving cleaning device 101. The specific arrangement position can be selected according to actual conditions. The embodiment of the present application only takes the arrangement of the robotic arm 103 on the top of the self-moving cleaning device 101 as an example for subsequent explanation.

[0072] Of course, the self-mobile cleaning device 101 may not be provided with the accommodating cavity 106, and the mechanical arm 103 may be arranged on the outer surface of the self-mobile cleaning device 101. When the mechanical arm 103 is not in use, the mechanical arm 103 may be stored on the surface of the self-mobile cleaning device 101. The storage method of the mechanical arm 103 on the body 102 may be adjusted according to actual conditions. The embodiment of the present application only takes the method of providing the accommodating cavity 106 in the self-mobile cleaning device 101 to accommodate the mechanical arm 103 as an example for subsequent description.

[0073] The present application provides an object grabbing control method, which is applied to a main controller of a self-moving cleaning device 101. Figure 1 As shown, the self-moving cleaning device 101 also includes a body 102 for cleaning and embedded with a main controller and a mechanical arm 103 capable of adjusting the posture. The starting end of the mechanical arm 103 is connected to the body 102, and the end of the mechanical arm 103 is provided with a clamping claw 104 and an image acquisition component 105. Figure 3 As shown, the method provided in the embodiment of the present application includes:

[0074] S301 : When the robot arm 103 is in the out-of-warehouse state, an image of the environment in front of the moving path of the self-moving cleaning device 101 collected by the image collection component 105 is received.

[0075] The moving path may be a cleaning path of the self-mobile cleaning device 101 , or a destination path from one location to another, or a path back to a base station, etc. It is understandable that the moving path may vary depending on the task of the self-mobile cleaning device 101 .

[0076] It can be understood that when the robot arm 103 is in the warehouse exit state, the height of the robot arm 103 is higher than the height of the fuselage 102. Even if the height of the object is relatively high, the full appearance of the object can be reflected in the environmental image.

[0077] S302: Identify the grasping points on the object according to the environment image, and determine the coordinates of the grasping points of the object in the image acquisition component coordinate system.

[0078] The gripping point is the contact point between the clamp 104 and the object when the clamp 104 grips the object so that the object is not easily dropped from the clamp 104. For example, the gripping point may be, but is not limited to, the point where the center of mass of the object is projected onto the surface of the object in the gripping direction of the clamp 104.

[0079] Exemplarily, the ways of identifying the gripping points on the object may include but are not limited to the following two:

[0080] The first type: In some embodiments, when an object is identified in an environment image, the grasping point of the object in the environment image is identified based on a pre-trained grasping point recognition model, wherein the grasping point recognition model is trained based on multiple training samples input into a network to be trained, each training sample including a historical object image marked with a standard grasping point. For example, the network to be trained may be, but is not limited to, a convolutional neural network.

[0081] The second method is to determine the grabbing point of the object based on a preset machine vision algorithm. For example, the machine vision algorithm may be, but is not limited to, the Otsu method or the GraphCut algorithm (i.e., an image segmentation algorithm based on graph theory). For example, the machine vision algorithm may determine the grabbing point of the object by extracting the contour data of the object based on the machine vision algorithm, determining the center of mass of the object based on the contour data, and calculating the center of mass of the object based on the formula (C x , C y ) is the pixel coordinate of the object’s center of mass in the environment image, N is the number of pixels in the contour data, (x i ,y i ) is the pixel coordinate of the i-th pixel point in the contour data in the environment image.

[0082] Exemplarily, the method of determining the coordinates of the grabbing point of the object in the image acquisition component coordinate system may be, but is not limited to:

[0083] Step A: According to the position of the grasping point of the object in the environment image, the pixel coordinates of the grasping point in the environment image are determined.

[0084] For example, the pixel point in the upper left corner of the environment image can be used as the origin of the image coordinate system, the length direction of the environment image can be used as the X-axis of the image coordinate system, and the width direction of the environment image can be used as the Y-axis of the image coordinate system. In this way, the pixel coordinates of the grabbing point in the image coordinate system of the environment image can be determined according to the position of the grabbing point of the object in the image coordinate system of the environment image.

[0085] Step B: According to a preset second conversion matrix, the pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system.

[0086] For example, according to the formula The pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system. 1 is the coordinate of the grabbing point on the X axis of the image acquisition component coordinate system, y 1 is the coordinate of the grabbing point on the Y axis of the image acquisition component coordinate system, z 1 is the coordinate of the grabbing point on the Z axis of the image acquisition component coordinate system, is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, fx is the focal length of the lens of the image acquisition component 105 in the horizontal direction, and f y is the focal length of the lens of the image acquisition component 105 in the vertical direction, c x is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c yis the distance from the geometric center of the environment image to the X-axis direction of the image coordinate system of the environment image. In this way, the coordinates of the grabbing point in the image acquisition component coordinate system can be accurately obtained.

[0087] It should be noted that or, u 1 is the coordinate of the grab point on the X axis of the environment image, v 1 is the coordinate of the grab point on the Y axis of the environment image, w 1 is the depth coordinate of the grasping point in the environment image.

[0088] S303: Acquire the posture data of the end of the robot arm 103, and determine the coordinates of the grasping point of the object in the preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the image acquisition component coordinate system.

[0089] For example, Figure 4 As shown, the mechanical arm 103 includes a plurality of rotatable mechanical axes connected in sequence, and the clamping claw 104 and the image acquisition component 105 are arranged at the end of the last mechanical axis. The mechanical arm 103 includes: a base 41, a rotating base 42, a support shaft 43, a connecting shaft 44, and a working shaft 45. The base 41 is connected to the accommodating cavity 106 of the fuselage 102, that is, the entire mechanical arm 103 is installed in the accommodating cavity 106 through the base and the mounting structure arranged in the accommodating cavity 106 to be fixed on the fuselage 102. Specifically, the mounting structure can be a mounting seat, a mounting hole, a slot or other structure, that is, the base can be fixed in the accommodating cavity 106 by screws, slot hooks or other structures that meet the requirements.

[0090] Further, the rotating seat 42 is connected to the base 41 through the first mechanical joint A1 so that the rotating seat 42 can rotate relative to the base 41, and the support shaft 43 is connected to the rotating seat 42 through the second mechanical joint A2 so that the support shaft 43 can be folded or unfolded relative to the rotating seat 42, such as the support shaft 43 can be lifted or lowered relative to the rotating seat 42. The first end of the connecting shaft 44 is connected to the supporting shaft 43 through a third mechanical joint A3 so that the connecting shaft 44 can be folded or unfolded relative to the supporting shaft, such as the connecting shaft 44 can be lifted or lowered relative to the supporting shaft 43, and the second end of the connecting shaft 44 is connected to the working shaft 45 through another third mechanical joint A3 so that the working shaft 45 can be folded or unfolded relative to the connecting shaft 44, that is, the working shaft 45 can be lifted or lowered relative to the connecting shaft 44, and the connecting shaft 44 is connected to the clamp 104 through a fourth mechanical joint A4 so that the clamp 104 can rotate relative to the working shaft 45.

[0091] It can be understood that the above-mentioned support shaft 43, connecting shaft 44, and working shaft 45 are the above-mentioned multiple rotatable mechanical shafts connected in sequence. The first mechanical joint A1, the second mechanical joint A2, the third mechanical joint A3, and the fourth mechanical joint A4 can be used to drive the motor to drive the corresponding mechanical shaft to rotate. Among them, the first mechanical joint A1 can be understood as a waist rotation joint, the second mechanical joint A2 can be understood as a waist lifting joint, the two third mechanical joints A3 can be shoulder joints and elbow joints that can be lifted and lowered, respectively, and the fourth mechanical joint A4 is a wrist joint that can rotate.

[0092] It can be understood that the robotic arm 103 provided in the embodiment of the present application can drive multiple mechanical joints (such as the first mechanical joint A1, the second mechanical joint A2, the third mechanical joint A3, and the fourth mechanical joint A4) by driving multiple driving motors to drive the end of the last mechanical axis (i.e., the location of the gripper 104 and the image acquisition component 105) to achieve multiple degrees of freedom rotation, and any posture adjustment of the end of the robotic arm 103 can be achieved.

[0093] Further, each mechanical axis corresponds to a mechanical axis coordinate system, and each mechanical axis coordinate system is a coordinate system with the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis. The above-mentioned posture data may include: the first angle between the first projection of each mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis. In this way, the rotation angle of the electric joint at the starting end of each rotatable mechanical axis can be obtained; according to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table, and the posture data of the end of the mechanical arm 103 is obtained.

[0094] Exemplarily, the rotation parameters of each drive motor driving the corresponding mechanical joint can be received, and according to the rotation parameters, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset mapping relationship table. For example, the rotation parameters of the control drive motor driving the third mechanical joint A3 are found from the preset mapping relationship table to find the first angle between the first projection of the working axis 45 on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis.

[0095] Specifically, S303 can be implemented as follows:

[0096] Step 1: According to the posture data, determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system.

[0097] Exemplarily, for each mechanical axis, a sub-conversion matrix is ​​constructed according to the first angle, the first distance, the second angle, and the second distance corresponding to the mechanical axis. Construct the sub-transformation matrix. Among them, A i is the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

[0098] Furthermore, according to the formula T = A 1 ×A 2 ×...×A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system. Where T is the first transformation matrix, A 1 is the sub-transformation matrix corresponding to the first mechanical axis, A 2 is the sub-transformation matrix corresponding to the second mechanical axis, A n is the sub-conversion matrix corresponding to the nth mechanical axis. In this way, the first conversion matrix from the fuselage coordinate system to the image acquisition component coordinate system can be accurately obtained.

[0099] Step 2: According to the coordinates of the grabbing point of the object in the coordinate system of the image acquisition component and the first conversion matrix, the coordinates of the grabbing point of the object in the coordinate system of the fuselage are determined.

[0100] For example, according to the formula (x 2 ,y 2 , z 2 )=T(x 1,y 1 , z 1 ), where (x 1 ,y 1 , z 1 ) is the coordinate of the grasping point in the image acquisition component coordinate system, (x 2 ,y 2 , z 2 ) is the coordinate of the grasping point in the fuselage coordinate system, and T is the first transformation matrix.

[0101] S304: According to the coordinates of the grasping point in the fuselage coordinate system, the robot arm 103 is controlled to move to one side of the grasping point, and the gripper 104 is controlled to grasp the object through the grasping point.

[0102] Exemplarily, the coordinates of the geometric center of the gripper 104 disposed at the end of the robot arm 103 in the fuselage coordinate system can be determined based on the posture data of the end of the above-mentioned robot arm 103, and then, the moving direction and moving distance of the gripper 104 at the end of the robot arm 103 can be determined based on the coordinates of the geometric center of the gripper 104 in the fuselage coordinate system and the coordinates of the grasping point of the object in the fuselage coordinate system, and then, the robot arm 103 is controlled to drive the gripper 104 to move to one side of the grasping point based on the determined moving direction and moving distance, and the gripper 104 is controlled to grasp the object through the grasping point and place the object at a position away from the moving path.

[0103] In summary, the embodiment of the present application provides an article grabbing control method, when the mechanical arm 103 is in the out-of-warehouse state, the environmental image in front of the moving path of the self-mobile cleaning device 101 collected by the image acquisition component 105 is received. It can be understood that when the mechanical arm 103 is in the out-of-warehouse state, the height of the mechanical arm 103 is higher than the height of the fuselage 102, even if the height of the object is high, the full view of the object can be reflected in the environmental image. According to the environmental image, the grabbing point on the object is identified, and the coordinates of the grabbing point of the object in the image acquisition component coordinate system are determined. Since the full view of the object can be reflected in the environmental image, the accuracy of the identified grabbing point on the object is high, and the accuracy of the coordinates of the determined grabbing point in the image acquisition component coordinate system is also high. The posture data of the end of the mechanical arm 103 is obtained, and the coordinates of the grabbing point of the object in the preset fuselage coordinate system are determined according to the posture data and the coordinates of the grabbing point of the object in the image acquisition component coordinate system. In this way, the coordinates of the grabbing point in the image acquisition component coordinate system can be accurately converted to the coordinates in the fuselage coordinate system. Furthermore, according to the coordinates of the grasping point in the fuselage coordinate system, the robot arm 103 is controlled to move to one side of the grasping point, and the gripper 104 is controlled to grasp the object through the grasping point. The object is not easy to fall off the gripper 104. In this way, the gripper 104 can accurately and reliably place the object at a position away from the moving path to achieve obstacle avoidance.

[0104] In addition, the embodiment of the present application further provides an article grabbing control device, which is configured in the main controller of the self-moving cleaning device 101. It should be noted that the basic principle and technical effect of the article grabbing control device provided in the embodiment of the present application are the same as those in the above-mentioned embodiment. For the sake of brief description, for parts not mentioned in the embodiment of the present application, reference can be made to the corresponding content in the above-mentioned embodiment. In some embodiments, the self-moving cleaning device 101 also includes a body 102 for cleaning and embedded with a main controller, a robotic arm 103 capable of adjusting posture, and a clamp 104. The starting end of the robotic arm 103 is connected to the body 102, and the end of the robotic arm 103 is provided with a clamp 104 and an image acquisition component 105. Figure 5 As shown, the device provided by the embodiment of the present application includes a data receiving unit, a grasping point identification unit, a coordinate conversion unit, and a grasping control unit, wherein:

[0105] A data receiving unit, used to receive an environmental image in front of the moving path of the self-moving cleaning device 101 collected by the image collection component 105 when the robot arm 103 is in the out-of-warehouse state;

[0106] A grasping point recognition unit, used to recognize the grasping points on the object according to the environment image, and determine the coordinates of the grasping points of the object in the coordinate system of the image acquisition component;

[0107] A coordinate conversion unit, used to obtain the posture data of the end of the robot arm 103, and determine the coordinates of the grasping point of the object in a preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the image acquisition component coordinate system;

[0108] The grabbing control unit is used to control the robot arm 103 to move to one side of the grabbing point according to the coordinates of the grabbing point in the fuselage coordinate system, and control the gripper 104 to grab the object through the grabbing point.

[0109] In some embodiments, the grasping point identification unit is specifically used to

[0110] Determine a first transformation matrix from a fuselage coordinate system to an image acquisition component coordinate system according to the position and posture data;

[0111] The coordinates of the grabbing point of the object in the coordinate system of the fuselage are determined according to the coordinates of the grabbing point of the object in the coordinate system of the image acquisition component and the first conversion matrix.

[0112] In some embodiments, the mechanical arm 103 includes a plurality of rotatable mechanical axes connected in sequence, the gripper 104 and the image acquisition component 105 are arranged at the end of the last mechanical axis, each mechanical axis corresponds to a mechanical axis coordinate system, each mechanical axis coordinate system takes the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis coordinate system, and the posture data includes a first angle between the first projection of each mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, a first distance between the end of the first projection and the X-axis, a second angle between the first projection and the Z-axis, and a second distance between the end of the first projection and the Z-axis.

[0113] A coordinate conversion unit, specifically configured to construct a sub-conversion matrix for each mechanical axis according to a first angle, a first distance, a second angle, and a second distance corresponding to the mechanical axis;

[0114] According to the formula T = A 1 ×A 2 ×...×A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system, where T is the first transformation matrix, A 1 is the sub-transformation matrix corresponding to the first mechanical axis, A 2 is the sub-transformation matrix corresponding to the second mechanical axis, A n It is the sub-transformation matrix corresponding to the nth mechanical axis.

[0115] In some embodiments, the coordinate conversion unit is specifically used to

[0116] According to the formula Construct the sub-transformation matrix, A i is the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

[0117] In some embodiments, the coordinate conversion unit is specifically used to obtain the rotation angle of the electric joint at the starting end of each rotatable mechanical axis;

[0118] According to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table.

[0119] In some embodiments, the coordinate conversion unit is specifically used to determine the pixel coordinates of the grasping point in the environment image according to the position of the grasping point of the object in the environment image;

[0120] According to the preset second conversion matrix, the pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system.

[0121] In some embodiments, the coordinate conversion unit is specifically used to

[0122] According to the formula The pixel coordinates of the grasping point in the environment image are converted to obtain the coordinates of the grasping point in the image acquisition component coordinate system, where x 1 is the coordinate of the grabbing point on the X axis of the image acquisition component coordinate system, y 1 is the coordinate of the grabbing point on the Y axis of the image acquisition component coordinate system, z 1 is the coordinate of the grabbing point on the Z axis of the image acquisition component coordinate system, is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, c x is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c y It is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction.

[0123] In some embodiments, the grasping point recognition unit is specifically used to recognize the grasping points of the objects in the environment image based on a pre-trained grasping point recognition model when recognizing the existence of the objects in the environment image. The grasping point recognition model is trained based on multiple training samples input into the network to be trained, and each training sample includes a historical object image marked with a standard grasping point.

[0124] In some embodiments, the grasping point identification unit is specifically configured to determine the grasping point of the object based on a preset machine vision algorithm.

[0125] In some embodiments, the gripping point is the point where the center of mass of the object is projected onto the surface of the object in the gripping direction of the gripper 104 .

[0126] In some embodiments, or, u 1 is the coordinate of the grab point on the X axis of the environment image, v 1 is the coordinate of the grab point on the Y axis of the environment image, w 1 is the depth coordinate of the grasping point in the environment image.

[0127] In addition, the embodiment of the present application also provides a self-moving cleaning device, which can be a sweeping robot, a mopping robot, a sweeping and mopping robot, etc., which is not limited here. The self-moving cleaning device includes a main controller, a fuselage for cleaning and embedded with the main controller, a mechanical arm capable of adjusting the posture, and a clamp. The starting end of the mechanical arm is connected to the fuselage, and the end of the mechanical arm is provided with a clamp and an image acquisition component. Optionally, the self-moving cleaning device also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc.

[0128] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0129] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0130] The main controller reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an object grabbing control device at the logical level. The main controller executes the program stored in the memory and is specifically used to execute the object grabbing control method provided in the above embodiment.

[0131] The above application Figure 3The method performed by the article grabbing control device disclosed in the illustrated embodiment can be applied to the main controller or implemented by the main controller. The main controller may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the main controller or the instructions in the form of software. The above main controller can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0132] Of course, in addition to software implementation methods, the self-moving cleaning device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0133] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a self-moving cleaning device including a plurality of application programs, enable the self-moving cleaning device to execute Figure 3 The object grabbing control method shown.

[0134] In addition, computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM) or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0135] In addition, the embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the present application. Figure 3 The object grabbing control method shown.

[0136] In the above description, the technical details such as the patterning of each layer are not described in detail. However, those skilled in the art should understand that various technical means can be used to form layers, regions, etc. of desired shapes. In addition, in order to form the same structure, those skilled in the art can also design methods that are not completely the same as the methods described above. In addition, although the various embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage.

[0137] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0138] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for controlling object grabbing, characterized in that: Applied to a self-moving cleaning device, the self-moving cleaning device comprises a body and a mechanical arm, the starting end of the mechanical arm is connected to the body, and the end of the mechanical arm is provided with a clamp and an image acquisition component, the method comprises: When the robot arm is in an out-of-warehouse state, receiving an environmental image in front of a moving path of the self-moving cleaning device acquired by the image acquisition component; According to the environment image, identifying the grasping point on the object in the environment image, and determining the coordinates of the grasping point of the object in the image acquisition component coordinate system; Acquire the posture data of the end of the robot arm, and determine the coordinates of the grasping point of the object in a preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the image acquisition component coordinate system; According to the coordinates of the grasping point in the fuselage coordinate system, the robot arm is controlled to move to one side of the grasping point, and the gripper is controlled to grasp the object through the grasping point.

2. The method according to claim 1, characterized in that Determining the coordinates of the grabbing point of the object in the body coordinate system according to the posture data and the coordinates of the grabbing point of the object in the image acquisition component coordinate system includes: Determine a first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system according to the posture data; The coordinates of the grabbing point of the object in the body coordinate system are determined according to the coordinates of the grabbing point of the object in the image acquisition component coordinate system and the first conversion matrix.

3. The method according to claim 2, characterized in that The mechanical arm includes a plurality of rotatable mechanical axes connected in sequence, the clamp and the image acquisition component are arranged at the end of the last mechanical axis, each of the mechanical axes corresponds to a mechanical axis coordinate system, each of the mechanical axis coordinate systems takes the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis coordinate system, the posture data includes a first angle between the first projection of each mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, a first distance between the end of the first projection and the X-axis, a second angle between the first projection and the Z-axis, and a second distance between the end of the first projection and the Z-axis, and the first conversion matrix from the fuselage coordinate system to the image acquisition component coordinate system is determined according to the posture data, including: For each of the mechanical axes, construct a sub-conversion matrix according to the first angle, the first distance, the second angle, and the second distance corresponding to the mechanical axis; According to the formula T = A1 × A2 × ... × A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system, wherein T is the first transformation matrix, A1 is the sub-transformation matrix corresponding to the first mechanical axis, A2 is the sub-transformation matrix corresponding to the second mechanical axis, and A n It is the sub-transformation matrix corresponding to the nth mechanical axis.

4. The method according to claim 3, characterized in that For each of the mechanical axes, constructing a sub-conversion matrix according to the first angle, the first distance, the second angle, and the second distance corresponding to the mechanical axis, includes: According to the formula Construct the sub-transformation matrix, A i is the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

5. The method according to claim 3, characterized in that: The step of obtaining the position and posture data of the end of the robotic arm comprises: Obtaining the rotation angle of the electric joint at the starting end of each of the rotatable mechanical axes; According to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table.

6. The method according to claim 1, characterized in that Determining the coordinates of the grabbing point of the object in the image acquisition component coordinate system includes: Determine the pixel coordinates of the grasping point in the environment image according to the position of the grasping point of the object in the environment image; According to a preset second conversion matrix, the pixel coordinates of the grabbing point in the environment image are converted to obtain the coordinates of the grabbing point in the image acquisition component coordinate system.

7. The method according to claim 6, characterized in that The step of converting the pixel coordinates of the grabbing point in the environment image according to a preset second conversion matrix to obtain the coordinates of the grabbing point in the image acquisition component coordinate system includes: According to the formula The pixel coordinates of the grab point in the environment image are converted to obtain the coordinates of the grab point in the image acquisition component coordinate system, wherein x1 is the coordinate of the grab point on the X axis of the image acquisition component coordinate system, y1 is the coordinate of the grab point on the Y axis of the image acquisition component coordinate system, and z1 is the coordinate of the grab point on the Z axis of the image acquisition component coordinate system. is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, and f x is the horizontal focal length of the lens of the image acquisition component, f y is the focal length of the lens of the image acquisition component in the vertical direction, c x is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c y is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction.

8. The method according to claim 7, characterized in that or, u1 is the coordinate of the grasping point on the X-axis of the environment image, v1 is the coordinate of the grasping point on the Y-axis of the environment image, and w1 is the depth coordinate of the grasping point in the environment image.

9. The method according to any one of claims 1 to 8, characterized in that: The step of identifying, according to the environment image, a grasping point on an object in the environment image comprises: In the case where an object is identified to exist in the environment image, the grasping points of the object in the environment image are identified based on a pre-trained grasping point recognition model, wherein the grasping point recognition model is trained based on multiple training samples input into the network to be trained, and each of the training samples includes a historical object image marked with a standard grasping point.

10. The method according to any one of claims 1 to 8, characterized in that: The step of identifying, according to the environment image, a grasping point on an object in the environment image comprises: Based on a preset machine vision algorithm, the grasping point of the object is determined.

11. The method according to any one of claims 1 to 8, characterized in that: The grasping point is the point where the center of mass of the object is projected onto the surface of the object in the clamping direction of the clamping claw.

12. An object grabbing control device, characterized in that: The self-moving cleaning device is configured to include a body and a mechanical arm, the starting end of the mechanical arm is connected to the body, and the end of the mechanical arm is provided with a clamp and an image acquisition component, and the device includes: A data receiving unit, configured to receive, when the robot arm is in an out-of-warehouse state, an environmental image in front of a moving path of the self-moving cleaning device acquired by the image acquisition component; A grasping point recognition unit, used to recognize grasping points on an object in the environment image according to the environment image, and determine the coordinates of the grasping points of the object in the image acquisition component coordinate system; A coordinate conversion unit, used to obtain the posture data of the end of the robot arm, and determine the coordinates of the grasping point of the object in a preset fuselage coordinate system according to the posture data and the coordinates of the grasping point of the object in the coordinate system of the image acquisition component; A grasping control unit is used to control the robot arm to move to one side of the grasping point according to the coordinates of the grasping point in the fuselage coordinate system, and control the gripper to grasp the object through the grasping point.

13. The device according to claim 12, characterized in that The grasping point identification unit is specifically used for Determine a first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system according to the posture data; The coordinates of the grabbing point of the object in the body coordinate system are determined according to the coordinates of the grabbing point of the object in the image acquisition component coordinate system and the first conversion matrix.

14. The device according to claim 13, characterized in that The mechanical arm includes a plurality of rotatable mechanical axes connected in sequence, the clamping claw and the image acquisition component are arranged at the end of the last mechanical axis, each of the mechanical axes corresponds to a mechanical axis coordinate system, each of the mechanical axis coordinate systems takes the starting end of the corresponding mechanical axis as the origin, the first direction on the horizontal plane as the X-axis, the second direction on the horizontal plane perpendicular to the X-axis as the Y-axis, and the third direction perpendicular to the horizontal plane as the Z-axis coordinate system, the posture data includes a first angle between the first projection of each mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, a first distance between the end of the first projection and the X-axis, a second angle between the first projection and the Z-axis, and a second distance between the end of the first projection and the Z-axis. The coordinate conversion unit is specifically configured to construct, for each of the mechanical axes, a sub-conversion matrix according to the first angle, the first distance, the second angle, and the second distance corresponding to the mechanical axis; According to the formula T = A1 × A2 × ... × A n , determine the first transformation matrix from the fuselage coordinate system to the image acquisition component coordinate system, wherein T is the first transformation matrix, A1 is the sub-transformation matrix corresponding to the first mechanical axis, A2 is the sub-transformation matrix corresponding to the second mechanical axis, and A n It is the sub-transformation matrix corresponding to the nth mechanical axis.

15. The device according to claim 14, characterized in that The coordinate conversion unit is specifically used for According to the formula Construct the sub-transformation matrix, A i is the sub-transformation matrix corresponding to the i-th mechanical axis, α i is the first angle, θ i is the second angle, a i is the first distance, d i is the second distance.

16. The device according to claim 14, characterized in that The coordinate conversion unit is specifically used to obtain the rotation angle of the electric joint at the starting end of each of the rotatable mechanical axes; According to the rotation angle of the electric joint at the starting end of each mechanical axis, the first angle between the first projection of the mechanical axis on the vertical plane and the X-axis of the corresponding mechanical axis coordinate system, the first distance between the end of the first projection and the X-axis, the second angle between the first projection and the Z-axis, and the second distance between the end of the first projection and the Z-axis are found from the preset relationship table.

17. The device according to claim 12, characterized in that The coordinate conversion unit is specifically used to determine the pixel coordinates of the grasping point in the environment image according to the position of the grasping point of the object in the environment image; According to a preset second conversion matrix, the pixel coordinates of the grabbing point in the environment image are converted to obtain the coordinates of the grabbing point in the image acquisition component coordinate system.

18. The device according to claim 17, characterized in that The coordinate conversion unit is specifically used for According to the formula The pixel coordinates of the grab point in the environment image are converted to obtain the coordinates of the grab point in the image acquisition component coordinate system, wherein x1 is the coordinate of the grab point on the X axis of the image acquisition component coordinate system, y1 is the coordinate of the grab point on the Y axis of the image acquisition component coordinate system, and z1 is the coordinate of the grab point on the Z axis of the image acquisition component coordinate system. is the preset second transformation matrix, D is the pixel coordinate of the grasping point in the environment image, c x is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction, c y is the distance from the geometric center of the environment image to the image coordinate system of the environment image in the X-axis direction.

19. The device according to any one of claims 12 to 18, characterized in that: The grasping point recognition unit is specifically used to identify the grasping points of the object in the environmental image based on a pre-trained grasping point recognition model when an object is recognized to exist in the environmental image, wherein the grasping point recognition model is trained based on multiple training samples input into the network to be trained, and each of the training samples includes a historical object image marked with a standard grasping point.

20. The device according to any one of claims 12 to 18, characterized in that: The grasping point identification unit is specifically used to determine the grasping point of the object based on a preset machine vision algorithm.

21. A self-propelled cleaning device, characterized in that: It includes a main controller, a body, and a robotic arm, wherein the starting end of the robotic arm is connected to the body, and the end of the robotic arm is provided with a gripper, wherein the main controller is configured to execute to implement an object grasping control method as described in any one of claims 1 to 11.

22. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of a self-moving cleaning device, enables the self-moving cleaning device to implement an object grabbing control method as described in any one of claims 1 to 11.

23. A computer program product, characterized in that It comprises a computer program, which is used to implement the object grabbing control method according to any one of claims 1 to 11 when being executed by a processor.

Images