Object positioning method, robot system control method, device and electronic equipment

Abstract

The present disclosure provides an object positioning method, a control method, a device and an electronic device for a robot system. The object positioning method includes: controlling a point cloud acquisition device to acquire a target point cloud image, wherein the target point cloud image includes a first point cloud of a marker object and a second point cloud of an object to be processed; inputting the first point cloud into a robot model of the robot system for processing to obtain a first pose of the marker object; determining a first difference between the first pose and an actual pose of the marker object; determining the actual pose of the object to be processed based on the first difference, the second point cloud and the robot model, so as to achieve accurate positioning of the object to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an object positioning method, a control method, a device and an electronic device for a robot system. Background Art

[0002] The robot system includes a robot, wherein the robot can perform operations such as clamping, welding and testing. When controlling the robot to perform an operation, the robot system needs to be able to accurately determine the posture of the object to be processed, so as to control the end effector of the robot to move to the position corresponding to the posture to process the object to be processed.

[0003] At present, as the working time of the robot system increases, the environment of the robot system and its own devices will change, and these changes will affect the accuracy of the robot system in determining the position and posture of the object to be processed. Summary of the invention

[0004] Various aspects of the present disclosure provide an object positioning method, a control method, a device and an electronic device for a robot system to solve the problem of inaccurate position and posture of an object to be processed currently determined.

[0005] A first aspect of an embodiment of the present disclosure provides an object positioning method, which is applied to a robot system. The robot system includes a robot and a point cloud acquisition device. The object positioning method includes: controlling the point cloud acquisition device to acquire a target point cloud image, the target point cloud image includes a first point cloud of a marker object and a second point cloud of an object to be processed; inputting the first point cloud into a robot model of the robot system for processing to obtain a first pose of the marker object; determining a first difference between the first pose and an actual pose of the marker object; and determining the actual pose of the object to be processed based on the first difference, the second point cloud and the robot model.

[0006] A second aspect of an embodiment of the present disclosure provides a control method for a robot system, comprising: acquiring an actual posture of an object to be processed, where the actual posture of the object to be processed is determined by the object positioning method of the first aspect; and controlling the end effector of the robot system to move to a position corresponding to the actual posture.

[0007] A third aspect of the embodiments of the present disclosure provides an object positioning device for executing the object positioning method of the first aspect, and is applied to a robot system, wherein the robot system includes a robot and a point cloud acquisition device, and the object positioning device includes:

[0008] A control module, used to control the point cloud acquisition device to acquire a target point cloud image, wherein the target point cloud image includes a first point cloud of a marking object and a second point cloud of an object to be processed;

[0009] A processing module, used for inputting the first point cloud into a robot model of the robot system for processing to obtain a first pose of the marked object;

[0010] A first determination module, used to determine a first difference between the first pose and the actual pose of the marker object;

[0011] The second determination module is used to determine the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model.

[0012] A fourth aspect of an embodiment of the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the object positioning method of the first aspect and / or the robot system control method of the second aspect are implemented.

[0013] A fifth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the object positioning method of the first aspect and / or the control method of the robot system of the second aspect.

[0014] A sixth aspect of an embodiment of the present disclosure provides a computer program product, the program product comprising: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the object positioning method of the first aspect, and / or the control method of the robot system of the second aspect.

[0015] The disclosed embodiments are applied in a scenario where a robot processes an object. A target point cloud image is collected by controlling a point cloud acquisition device, and the target point cloud image includes a first point cloud of a marker object and a second point cloud of an object to be processed. The first point cloud is input into a robot model of a robot system for processing to obtain a first pose of the marker object. A first difference between the first pose and an actual pose of the marker object is determined. The actual pose of the object to be processed is determined based on the first difference, the second point cloud and the robot model, so that the time-varying error generated by the robot system during use can be compensated, thereby achieving accurate positioning of the object to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0017] Figure 1 An application scenario diagram of an object positioning method provided by an exemplary embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a method for locating an object provided by an exemplary embodiment of the present disclosure;

[0019] Figure 3 An application scenario diagram of another object positioning method provided by an exemplary embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of a marking object provided for an exemplary embodiment of the present disclosure;

[0021] Figure 5 A flowchart of another method for locating an object provided by an exemplary embodiment of the present disclosure;

[0022] Figure 6 A flowchart of a control method for a robot system provided by an exemplary embodiment of the present disclosure;

[0023] Figure 7 A structural block diagram of an object positioning device provided for an exemplary embodiment of the present disclosure;

[0024] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0026] When the robot system processes the object to be processed, it is necessary to determine the exact position of the object to be processed so as to move the end effector of the robot system to the corresponding position to process the object to be processed. At present, the robot system is calibrated before leaving the factory, but due to the environment and its own heating during use, the robot system will experience temperature drift, which will affect the positioning accuracy of the robot system for the object to be processed, and thus cause the robot system to be unable to accurately process the object to be processed.

[0027] Based on the above problems, the embodiment of the present disclosure collects the target point cloud image by controlling the point cloud acquisition device, and the target point cloud image includes a first point cloud of the marker object and a second point cloud of the object to be processed; the first point cloud is input into the robot model of the robot system for processing to obtain the first pose of the marker object; the first difference between the first pose and the actual pose of the marker object is determined; the actual pose of the object to be processed is determined according to the first difference, the second point cloud and the robot model, which can compensate for the time-varying error generated by the robot system during use, thereby achieving accurate positioning of the object to be processed.

[0028] In addition, an application scenario of the embodiment of the present disclosure is as follows Figure 1 , Figure 1 The robot system is placed in a room 10, and the robot system includes a point cloud acquisition device 11 and a robot, wherein the robot includes: an end effector 12, a robot base 14 and at least two motion segments 13 installed between the end effector 12 and the robot, and also includes a marker object B and an object to be processed D. The robot system can accurately locate the position of the object D to be processed when it is just turned on. However, after the robot system has been working for a period of time, the indoor temperature changes due to heat dissipation, resulting in temperature drift problems in the structures of the robot and the point cloud acquisition device, which in turn causes deviations in the positioning of the object D to be processed by the robot system. The present disclosure can compensate for the deviation by setting a fixed marker object B to accurately locate the object D to be processed.

[0029] in, Figure 1 This is just an exemplary application scenario, and the embodiment of the present disclosure can be applied to a robot system in any scenario. The embodiment of the present disclosure does not limit the specific application scenario.

[0030] Figure 2 A flowchart of a method for locating an object provided by an exemplary embodiment of the present disclosure. Specifically, the following steps are included:

[0031] S201, controlling the point cloud acquisition device to acquire a target point cloud image.

[0032] The present disclosure is applied to a robot system, wherein a robot system such as Figure 1 As shown, the point cloud acquisition device 11 of the robot system is fixed on the end effector 12 of the robot, and the marker object B is fixed on the outside of the robot. Figure 2 As shown, the robot system includes a point cloud acquisition device 31 and a robot, and the robot includes: an end effector 32 and a robot base 34, and at least two motion segments 33 connected between the end effector 32 and the robot base 34, wherein the point cloud acquisition device 31 is fixedly installed on the outside of the robot, and the marking object B is fixed on the end effector of the robot.

[0033] Among them, the target point cloud image includes a first point cloud of the marker object and a second point cloud of the object to be processed. Specifically, if the distance between the marker object and the object to be processed is less than the distance threshold, so that the marker object and the object to be processed are in the same field of view of the point cloud acquisition device, then the point cloud acquisition device can acquire a frame of target point cloud image, and the frame of target point cloud image includes: the first point cloud of the marker object and the second point cloud of the object to be processed. In addition, if the distance between the marker object and the object to be processed is greater than the distance threshold, so that the marker object and the object to be processed are not in the same field of view of the point cloud acquisition device, then the point cloud acquisition device can acquire two frames of target point cloud images, wherein one frame of the target point cloud image includes: the first point cloud of the marker object, and the other frame includes the second point cloud of the object to be processed.

[0034] Furthermore, the specific process of the point cloud acquisition device acquiring the target point cloud image is that the point cloud acquisition device acquires ordinary images of the marker object and the object to be processed, and obtains the target point cloud image according to the ordinary images and the camera internal parameters of the point cloud acquisition device. The point cloud in the target point cloud image includes multiple points, each of which includes coordinate information, and the coordinate information is the coordinate information of the second coordinate system under the point cloud acquisition device.

[0035] In addition, the first point cloud is a point cloud corresponding to the marker object. If the marker object is a sphere, the first point cloud roughly has the outline of the center of the sphere.

[0036] In the embodiment disclosed in the present invention, the position and posture of the marker object remains unchanged as time goes by, that is, the coordinates of the marker object in the third coordinate system of the robot base remain unchanged. In addition, the marker object can be a sphere or multiple spheres. For example, refer to Figure 4 The marking object can be three Figure 4 The spheres (sphere 41, sphere 42 and sphere 43) are arranged in the manner shown, wherein the line connecting sphere 41 and sphere 42 is perpendicular to the line connecting sphere 42 and sphere 43. The three spheres are in the same field of view of the point cloud acquisition device, and the first difference corresponding to each sphere can be calculated later, and the average value of the first difference of the three spheres is used as the compensation value to compensate the object to be processed.

[0037] S202, inputting the first point cloud into the robot model of the robot system for processing to obtain the first pose of the marked object.

[0038] The robot model refers to a robot system model. The robot model is pre-set and can determine the corresponding position and posture of the object based on the point cloud of the input object.

[0039] Specifically, refer to Figure 1 and Figure 3, both have a second coordinate system (X′Y′Z′) of the point cloud acquisition device and a third coordinate system (X″Y″Z″) of the robot base, wherein the second coordinate system (X′Y′Z′) of the point cloud acquisition device has the point cloud acquisition device as the origin O′ of the coordinate system, the Z′ axis is the direction of the central axis of the field of view of the point cloud acquisition device, and X′Y′ is a plane perpendicular to the central axis of the field of view. In the point cloud collected by the point cloud acquisition device, each point has a coordinate value under the second coordinate system (X′Y′Z′) of the point cloud acquisition device. The third coordinate system (X″Y″Z″) of the robot base is established with the robot base as the origin O, wherein the Z″ axis represents the vertical direction. X″ and Y″ are located in the horizontal plane, and the three coordinate axes are perpendicular to each other.

[0040] In one embodiment, referring to Figure 1 , the point cloud acquisition device 11 is fixed on the end effector 12 of the robot, the marker object B is fixed outside the robot, and the model parameters of the robot model include: the camera extrinsic parameters of the point cloud acquisition device 11 and the robot parameters of the robot. The first point cloud is input into the robot model of the robot system for processing to obtain the first pose of the marker object, including: performing the following steps in the robot model: determining the first execution pose of the marker object in the first coordinate system of the end effector according to the first point cloud and the camera extrinsic parameters, wherein the camera extrinsic parameters include: the transformation relationship from the second coordinate system of the point cloud acquisition device to the first coordinate system; determining the first pose of the marker object in the third coordinate system of the robot base according to the first execution pose and the robot parameters.

[0041] Specifically, refer to Figure 1 , the first coordinate system (XYZ) of the end effector 12, which takes a fixed point on the end effector 12 (such as the center of the flange) as the origin O of the coordinate system, the Z axis is perpendicular to the surface of the flange, and the X axis and the Y axis are perpendicular to each other and parallel to the surface of the flange. In addition, the point cloud acquisition device 11 is fixedly mounted on the end effector 12, so the second coordinate system (X′Y′Z′) of the point cloud acquisition device 11 has a fixed transformation relationship relative to the first coordinate system (XYZ) of the end effector 12. The transformation relationship can be expressed by a matrix, and each component in the matrix can characterize the translation and transformation relationship between the two coordinate systems. According to the transformation relationship, based on the coordinates of a point in the second coordinate system (X′Y′Z′), the coordinates of the point in the first coordinate system (XYZ) can be determined. Figure 1 In the figure, the camera extrinsic parameters of the point cloud acquisition device 11 include the expression of the transformation relationship. Therefore, the posture of the marked object in the first coordinate system (XYZ) of the end effector can be determined according to the camera extrinsic parameters and the first point cloud, which is the first execution posture.

[0042] Furthermore, the robot also has robot parameters, which are determined by the connection relationship between the various motion segments 13 of the robot, the angles between the connected motion segments, the position of each motion segment and the information of the rotation axis. The robot parameters are used to determine the first posture of the marked object in the third coordinate system (X″Y″Z″) according to the first execution posture.

[0043] In another embodiment, referring to Figure 3 , the point cloud acquisition device is fixedly installed on the outside of the robot, the marker object is fixed on the end effector of the robot, and the model parameters of the robot model include: the camera extrinsic parameters of the point cloud acquisition device, the first point cloud is input into the robot model of the robot system for processing, and the first pose of the marker object is obtained, including: performing the following steps in the robot model: determining the first pose of the marker object in the third coordinate system of the robot base according to the first point cloud and the camera extrinsic parameters, wherein the camera extrinsic parameters include: the transformation relationship from the second coordinate system to the third coordinate system of the point cloud acquisition device.

[0044] Specifically, in Figure 3 In the embodiment, the point cloud acquisition device is fixedly installed outside the robot, and the positional relationship between the point cloud acquisition device 31 and the robot base 34 is fixed. Figure 3 The camera extrinsic parameters include: the transformation relationship from the second coordinate system (X′Y′Z′) of the point cloud acquisition device to the third coordinate system (X″Y″Z″), which is also expressed by a matrix. The coordinates of the point in the first point cloud in the second coordinate system (X′Y′Z′) and the camera extrinsic parameters can be determined to determine the coordinates of the point in the third coordinate system (X″Y″Z″), and then the first pose can be determined.

[0045] S203, determining a first difference between the first pose and the actual pose of the marker object.

[0046] In this disclosure, Figure 1 In the case of S202, the first pose is calculated by using the pre-calibrated camera external parameters and robot parameters. Figure 2In the case of, S203 is to calculate the first pose by the pre-calibrated camera extrinsics. Among them, the calibration of the camera extrinsics and the robot parameters is carried out under a preset environment. For example, it is carried out under a preset ambient temperature and / or just after the robot system is turned on. However, during the use of the robot system, the robot system continues to work for several hours after it is turned on. As the working time of the robot system increases, the temperature of the robot system will increase, or the ambient temperature of the environment where the robot system is located (such as indoor 10) will increase due to heat dissipation. In addition, the robot system is made of a large amount of metal materials and / or non-metallic materials, and changes in the environment will cause the performance of the robot system itself to change, such as thermal expansion causing the axial extension of the robot's motion segment, or changes in the angle between the motion segments, etc. Further, since the robot system is shipped, it is time-consuming and labor-intensive to continuously recalibrate the robot system as the environment changes, and it will reduce the processing efficiency of the robot system for the object to be processed. Based on this, the present disclosure determines the first difference between the first pose and the actual pose of the marked object as compensation for the deviation, which can achieve rapid determination of the actual pose of the object to be processed.

[0047] Among them, Figure 1 In the figure, the position of the marker object B relative to the robot base 14 is always unchanged. The actual posture of the marker object B in the third coordinate system (X″Y″Z″) can be determined in advance. Then the calculated posture (first posture) of the marker object B in the third coordinate system (X″Y″Z″) is determined in the above manner. The first difference obtained by subtracting the actual posture from the first posture is used as the compensation value.

[0048] Optionally, in Figure 3 In the figure, the marker object B is fixed on the end effector 32, and its position relative to the end effector 32 is always unchanged. Then, the position of the marker object B in the third coordinate system (X″Y″Z″) can be determined as the actual position based on the robot parameters. Then, after the end effector 32 completes each processing of the object to be processed, it moves to the actual position. In this case, the calculated position (first position) of the marker object B in the third coordinate system (X″Y″Z″) is determined by the above method, and the first difference obtained by subtracting the actual position from the first position is used as the compensation value.

[0049] S204: Determine the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model.

[0050] The second point cloud and the robot model can determine the calculated pose of the object to be processed, and the first difference is used as a compensation value to compensate the calculated pose, thereby determining the actual pose of the object to be processed. The actual pose is the compensated pose, and the compensated pose is used as the actual pose to control the robot system to process the object to be processed.

[0051] In a first optional embodiment, the first point cloud and the second point cloud are included in the same target point cloud image, and the actual posture of the object to be processed is determined according to the first difference, the second point cloud and the robot model, including: inputting the second point cloud into the robot model of the robot system for processing to obtain a second posture of the object to be processed; and determining a second difference between the second posture and the first difference as the actual posture of the object to be processed. Figure 1 and Figure 3 All robot systems can use this method to determine the actual position and posture of the object to be processed.

[0052] Specifically, for Figure 1 In the robot system shown, the model parameters of the robot model include camera extrinsic parameters and robot parameters. The posture H1 of the object to be processed in the first coordinate system of the end effector 12 can be obtained based on the second point cloud and the camera extrinsic parameters, and then the second posture of the object to be processed in the third coordinate system of the robot base 14 can be obtained based on the posture H1 and the robot parameters.

[0053] Specifically, for Figure 3 In the robot system shown, the model parameters of the robot model include camera extrinsics, and the second posture of the object to be processed in the third coordinate system of the robot base 34 can be obtained according to the second point cloud and the camera extrinsics.

[0054] Furthermore, the first difference is calculated by subtracting the actual posture of the marker object from the first posture, and the second difference is calculated by subtracting the first difference from the second posture H3.

[0055] Wherein, the first point cloud and the second point cloud are not included in the same target point cloud image, and the compensation can also be performed by using the first method mentioned above, which is not limited here.

[0056] In a second optional embodiment, the first point cloud and the second point cloud are in different target point cloud images, and the actual posture of the object to be processed is determined according to the first difference, the second point cloud and the robot model, and also includes: inputting the second point cloud into the robot model of the robot system for processing to obtain a third posture of the object to be processed; determining a third difference between the third posture and the first difference; obtaining a fourth posture of the end effector when collecting the target point cloud image containing the first point cloud, and obtaining a fifth posture of the end effector when collecting the target point cloud image containing the second point cloud; determining the transformation matrix from the fourth posture to the fifth posture; and determining the actual posture according to the third difference and the transformation matrix. Among them, for Figure 1 The robot system can use this method to determine the actual position and posture of the object to be processed.

[0057] Specifically, for Figure 1In the robot system shown, the end effector 12 moves to the fourth posture to drive the point cloud acquisition device 11 to move and collect the target point cloud image for the marker object. The target point cloud image only contains the first point cloud of the marker object. At this time, the posture of the end effector is the fourth posture, which is the posture in the third coordinate system of the robot base. Then the end effector 12 moves to the fifth posture to drive the point cloud acquisition device 11 to move and collect the target point cloud image for the object to be processed. The target point cloud image only contains the second point cloud of the object to be processed. At this time, the posture of the end effector is the fifth posture, which is the posture in the third coordinate system of the robot base. Among them, the transformation matrix from the fourth posture to the fifth posture is taken into account in the compensation, so that the posture of the object to be processed can be compensated more accurately. Among them, the actual posture is determined by multiplying the third difference and the transformation matrix.

[0058] Furthermore, for Figure 3 In the robot system shown, the point cloud acquisition device 31 is always fixed. When acquiring the target point cloud image, the object to be processed and the marker object are always in the same field of view of the point cloud acquisition device 31, so the second solution is not applicable.

[0059] In a third optional embodiment, the actual posture of the object to be processed is determined based on the first difference, the second point cloud and the robot model, including: adjusting the model parameters of the robot model based on the first difference; inputting the second point cloud into the adjusted robot model for processing to obtain the actual posture of the marked object.

[0060] Among them, adjusting the model parameters of the robot model is to adjust the external parameters and robot parameters of the point cloud acquisition device 11, so that after the first point cloud is input into the adjusted robot model, the actual position and posture of the marked object can be output. Then, the adjusted robot model can be used to process the second point cloud to obtain the actual position and posture of the object to be processed.

[0061] Furthermore, for Figure 1 The robot system inputs the second point cloud into the adjusted robot model for processing to obtain the actual posture of the marker object, and also includes: adjusting the camera extrinsic parameters according to a fourth difference between the first execution posture and the actual execution posture of the marker object in the first coordinate system.

[0062] Specifically, the first execution posture is the calculated posture of the marker object in the first coordinate system of the end effector, and the actual execution posture is the actual posture of the marker object in the first coordinate system of the end effector. Since the camera extrinsic parameter represents the transformation relationship from the second coordinate system of the point cloud acquisition device 11 to the first coordinate system of the end effector 12, the fourth difference can be used to adjust the camera extrinsic parameter, so that the execution posture determined by the first point cloud and the adjusted camera extrinsic parameter and the actual execution posture can be the same, and then the second point cloud and the adjusted camera extrinsic parameter can be used to determine the posture of the object to be processed in the first coordinate system of the end effector to solve the actual posture of the object to be processed in the first coordinate system of the end effector.

[0063] Figure 5 A flowchart of another object positioning method provided for an exemplary embodiment of the present disclosure specifically includes the following steps:

[0064] S501, when the environment where the robot system is located is a preset environment, controlling the point cloud acquisition device to acquire a reference point cloud image.

[0065] The reference point cloud image includes a reference point cloud of a marker object. In the present disclosure, the preset environment may refer to the robot system being in a shutdown state for a time greater than a first preset time, such as 10 hours or 15 hours, and after the robot system is completely cooled, the robot system is turned on and a reference point cloud image is collected for the marker object within a second preset time, such as 2 minutes or 3 minutes.

[0066] for Figure 1 and / or Figure 3 The robot system shown in the figure inputs the first control parameter into the robot system, and controls the robot to collect the reference point cloud image in the first state. Then, when collecting the target point cloud image containing the marker object later, the first control parameter is input into the robot to control the robot to collect the target point cloud image in the second state. Among them, as the working time of the robot system increases, the temperature of the robot system itself and the ambient temperature will change, which will cause the robot system itself to change. Therefore, when the same first control parameter is input into the robot system at different times, the state of the robot system may have slight differences, which is mainly manifested in the difference in the position of the end effector relative to the robot base, and the present disclosure can also compensate for this difference.

[0067] Among them, when the robot system is in a stable state, the point cloud acquisition device is controlled to collect reference point cloud images. That is, after the robot changes to a given posture, it waits for a stable state, collects reference point cloud images, and then controls the robot to change to another posture. After the robot changes to the specified posture, it takes a short period of time to stop. At this time, the reference point cloud image collected can more accurately determine the actual posture of the marked object.

[0068] S502, inputting the reference point cloud into the robot model for processing to obtain the actual position and posture of the marked object.

[0069] Specifically, for Figure 1 In the robot system shown, the position of the marker object B relative to the robot base is fixed. Figure 3 In the robot system shown, the position of the marker object B relative to the end effector is fixed.

[0070] The reference point cloud is obtained after the robot system is just turned on. Therefore, the pose determined based on the reference point cloud and the robot model can be determined as the actual pose of the marked object.

[0071] S503, controlling the point cloud acquisition device to acquire a target point cloud image.

[0072] For the specific implementation process of step S201, it should be added that, Figure 1 In the robot system shown, the control point cloud acquisition device can acquire a target point cloud image containing both the first point cloud and the second point cloud, or can acquire a target point cloud image containing the first point cloud and another target point cloud image containing the second point cloud. Figure 2 In the robot system shown, only the control point cloud acquisition device can acquire the target point cloud image containing both the first point cloud and the second point cloud.

[0073] In addition, when collecting the target point cloud image, the same first control parameter as that when collecting the reference point cloud image is input to the robot system to control the end effector of the robot system to move to the corresponding position to collect the target point cloud image.

[0074] Furthermore, when the marker object and the object to be processed are in the same field of view, that is, the first point cloud and the second point cloud are in the same target point cloud image, the target point cloud image can be collected when the robot is transformed into a corresponding posture. Specifically, since the marker object and the object to be processed are simultaneously involved in the same vibration of the robot, when the posture of the object to be processed is subsequently compensated, the error of the posture of the object to be processed when the robot is not stopped can be compensated according to the difference between the first posture determined when the robot is not stopped and the actual posture, so the waiting time for the robot to stop can be reduced to a certain extent, thereby improving the processing rhythm of the robot. Among them, when the robot is not stopped, the image collected by the point cloud acquisition device is clear for related calculations.

[0075] S504, inputting the first point cloud into the robot model of the robot system for processing to obtain the first position of the marked object.

[0076] For the specific implementation process of this step, please refer to S202 and will not be repeated here.

[0077] S505: Obtain a fifth difference between the sixth posture of the marker object and the actual posture of the marker object.

[0078] The sixth posture is the calculated posture of the marker object determined when the robot system processes the object before the object to be processed. Specifically, when the object before the object to be processed, the calculated posture of the marker object determined refers to the first posture of the marker object calculated last time.

[0079] Exemplarily, at time T0, the calculated pose of the marker object is determined to be the actual pose C0. At time T1, the object to be processed A1 is processed, and the calculated pose of the marker object is determined to be C1, then the first difference of the marker object at time T1 is "C1-C0". At time T2, the object to be processed A2 is processed, and the calculated pose (first pose) of the marker object is determined to be C2, and the corresponding sixth pose is the calculated pose C1 of the marker object determined when the robot system processes the object A1 before the object to be processed A2. At time T3, the object to be processed A3 is processed, and the calculated pose (first pose) of the marker object is determined to be C3, and the corresponding sixth pose is the calculated pose C2 of the marker object determined when the robot system processes the object A2 before the object to be processed A3. At time T4, the object to be processed A4 is processed, and the calculated pose (first pose) of the marker object is determined to be C4, and the corresponding sixth pose is the calculated pose C3 of the marker object determined when the robot system processes the object A3 before the object to be processed A4, and so on. At time T2, the fifth difference is C1-C0. At time T3, the fifth difference is C2-C0. At time T4, the fifth difference is C3-C0.

[0080] S506, determining a sixth difference between the first pose and the sixth pose.

[0081] In summary, the first pose is the calculated pose of the landmark object at the current moment. At T2, the sixth difference is C2-C1. At T3, the fifth difference is C3-C2. At T4, the fifth difference is C4-C3.

[0082] S507: Determine that the sum of the sixth difference and the fifth difference is the first difference.

[0083] Exemplarily, at time T2, the first difference is C2-C0. At time T3, the fifth difference is C3-C0. At time T4, the fifth difference is C4-C0.

[0084] In the present disclosure, an iterative calculation method is adopted, and the posture obtained by the previous calculation can be directly used to calculate the first difference.

[0085] S508: Determine the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model.

[0086] For the specific implementation process of this step, please refer to S203 and will not be repeated here.

[0087] S509, looping through the above steps to determine if the first preset condition is satisfied, ending the loop, and obtaining actual positions and postures of multiple objects to be processed.

[0088] Specifically, S503 to S508 are executed cyclically. After the above steps are executed cyclically, multiple actual positions and postures of the object to be processed can be obtained.

[0089] Among them, the first preset condition includes: the number of cycles is a preset number, the cycle duration is greater than or equal to the preset duration, and at least one of the ambient temperature of the robot system reaches a stable state. The preset number and the preset duration are both preset. For example, the preset number is 3 times, that is, the above steps are performed 3 times for each object to be processed, and 3 actual postures of the object to be processed are obtained. For another example, the cycle duration is 2 minutes, that is, the actual posture of the object to be processed is calculated multiple times within 2 minutes to obtain multiple actual postures. For another example, the above steps are continuously executed until the ambient temperature reaches a stable state. The ambient temperature reaching a stable state can be understood as the range of change of the ambient temperature within a certain period of time is within the preset temperature range.

[0090] S510: Determine, among the actual postures of the multiple objects to be processed, an actual posture that meets a second preset condition as a target actual posture.

[0091] Among them, the second preset condition includes: the actual posture of the object to be processed with the highest frequency is the target actual posture or the actual posture determined last time is the target actual posture. Specifically, the actual posture determined last time can be understood as being determined when the environment is warmer, so using the actual posture determined last time as the target actual posture can improve accuracy.

[0092] Furthermore, when the first preset condition includes: the number of cycles is a preset number of times and the cycle duration is greater than or equal to the preset duration, the actual postures of multiple objects to be processed can be averaged as the target actual posture.

[0093] In the present disclosure, by determining the first difference of the calibration object, all time-varying rigid errors of the robot system can be compensated, and at the same time, the rhythm and accuracy of the robot system in processing the object to be processed can be improved. Specifically, the compensated rigid errors of the robot include: compensating for errors caused by vibration, temperature drift, etc. of the robot, the point cloud acquisition device, the calibration object and the object to be processed in the same system. Improving the rhythm means that after the end effector of the robot reaches the position to be processed, it needs to stop for a while, and then perform point cloud acquisition on the object to be processed after the end effector of the robot stops shaking. In the present disclosure, the object to be processed can be compensated by calibrating the object to be processed, so there is no need to wait for the end effector to stop shaking before performing point cloud acquisition, and point cloud acquisition can be performed directly after reaching the position to be processed. The present disclosure can improve the processing efficiency of the robot system, as well as improve the positioning accuracy of the robot system for the object to be processed.

[0094] Figure 6 A flowchart of a control method for a robot system provided by an exemplary embodiment of the present disclosure specifically includes the following steps:

[0095] S601, obtaining the actual position and posture of the object to be processed.

[0096] Wherein, the actual position and posture of the object to be processed is determined according to any of the above object positioning methods;

[0097] S602, controlling the end effector of the robot system to move to a position corresponding to the actual posture.

[0098] In this power embodiment, the robot system may have computing control capabilities itself, or it may be connected to a server and use the server to control it to process the object to be processed. After obtaining the actual posture of the object to be processed, according to the application scenario and the actual posture, the corresponding position to which the end effector should move is determined according to a preset determination method, and the end effector of the robot system is controlled to move to the corresponding position, and the object to be processed can be processed subsequently, such as detection, grasping, etc.

[0099] In the present disclosure, since the actual position and posture of the object to be processed can be accurately determined by the above method, in this embodiment, a robot system can be accurately determined based on the actual position and posture of the object to be processed.

[0100] In the embodiments of the present disclosure, refer to Figure 7 In addition to providing an object positioning method, an object positioning device 70 is also provided for use in the above-mentioned object positioning method, including:

[0101] A control module 71, used to control the point cloud acquisition device to acquire a target point cloud image, wherein the target point cloud image includes a first point cloud of a marking object and a second point cloud of an object to be processed;

[0102] A processing module 72, used for inputting the first point cloud into the robot model of the robot system for processing to obtain the first pose of the marked object;

[0103] A first determination module 73, used to determine a first difference between the first pose and the actual pose of the marker object;

[0104] The second determination module 74 is used to determine the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model.

[0105] In an optional embodiment, the first point cloud and the second point cloud are included in the same target point cloud image, and the second determination module 74 is specifically used to: input the second point cloud into the robot model of the robot system for processing to obtain a second posture of the object to be processed; determine the second difference between the second posture and the first difference, which is the actual posture of the object to be processed.

[0106] In an optional embodiment, the first point cloud and the second point cloud are in different target point cloud images, and the second determination module 74 is specifically used to: input the second point cloud into the robot model of the robot system for processing to obtain a third posture of the object to be processed; determine a third difference between the third posture and the first difference; obtain a fourth posture of the end effector when collecting the target point cloud image containing the first point cloud, and obtain a fifth posture of the end effector when collecting the target point cloud image containing the second point cloud; determine the transformation matrix from the fourth posture to the fifth posture; and determine the actual posture based on the third difference and the transformation matrix.

[0107] In an optional embodiment, the second determination module 74 is specifically used to: adjust the model parameters of the robot model according to the first difference; input the second point cloud into the adjusted robot model for processing to obtain the actual position and posture of the marker object.

[0108] In an optional embodiment, the point cloud acquisition device is fixed on the end effector of the robot, the marker object is fixed outside the robot, and the model parameters of the robot model include: the camera external parameters of the point cloud acquisition device and the robot parameters of the robot. The processing module 72 is specifically used to perform the following steps in the robot model:

[0109] Determine the first execution posture of the marker object in the first coordinate system of the end effector according to the first point cloud and the camera extrinsic parameters, wherein the camera extrinsic parameters include: the transformation relationship from the second coordinate system of the point cloud acquisition device to the first coordinate system; determine the first position of the marker object in the third coordinate system of the robot base according to the first execution posture and the robot parameters.

[0110] In an optional embodiment, before the second determination module 74 inputs the second point cloud into the adjusted robot model for processing to obtain the actual posture of the marker object, it is also used to: adjust the camera extrinsic parameters according to the fourth difference between the first execution posture and the actual execution posture of the marker object in the first coordinate system.

[0111] In an optional embodiment, the point cloud acquisition device is fixedly installed outside the robot, the marker object is fixed on the end effector of the robot, and the model parameters of the robot model include: the camera external parameters of the point cloud acquisition device, and the processing module 72 is specifically used to perform the following steps in the robot model:

[0112] The first position of the marker object in the third coordinate system of the robot base is determined according to the first point cloud and the camera external parameters, wherein the camera external parameters include: the transformation relationship from the second coordinate system to the third coordinate system of the point cloud acquisition device.

[0113] In an optional embodiment, the first determination module 73 is specifically used to: obtain the fifth difference between the sixth posture of the marker object and the actual posture of the marker object, the sixth posture being the calculated posture of the marker object determined when the robot system processes an object before the object to be processed; determine the sixth difference between the first posture and the sixth posture; determine the sum of the sixth difference and the fifth difference as the first difference.

[0114] In an optional embodiment, before controlling the point cloud acquisition device to acquire the target point cloud image, the control module 71 is also used to: when the environment in which the robot system is located is a preset environment, control the point cloud acquisition device to acquire a reference point cloud image, the reference point cloud image including a reference point cloud of the marker object; and input the reference point cloud into the robot model for processing to obtain the actual posture of the marker object.

[0115] In an optional embodiment, it also includes: a third determination module (not shown), which is used to cyclically execute the object positioning method; when a first preset condition is met, the loop is ended to obtain the actual postures of multiple objects to be processed, and the first preset condition includes: the number of loops is a preset number, the loop duration is greater than or equal to the preset duration, and the ambient temperature of the robot system reaches at least one of a stable state; among the actual postures of multiple objects to be processed, the actual posture that meets the second preset condition is determined to be the target actual posture, wherein the second preset condition includes: the actual posture of the object to be processed with the highest frequency of occurrence is the target actual posture or the actual posture determined for the last time is the target actual posture.

[0116] The object positioning device provided by the embodiment of the present disclosure collects a target point cloud image by controlling a point cloud acquisition device, wherein the target point cloud image includes a first point cloud of a marker object and a second point cloud of an object to be processed; the first point cloud is input into a robot model of a robot system for processing to obtain a first pose of the marker object; a first difference between the first pose and the actual pose of the marker object is determined; the actual pose of the object to be processed is determined based on the first difference, the second point cloud and the robot model, and the time-varying error generated by the robot system during use can be compensated, thereby achieving accurate positioning of the object to be processed.

[0117] In addition, the embodiment of the present application further provides a control device (not shown) of a robot system, comprising:

[0118] An acquisition module, used for acquiring an actual position and posture of the object to be processed, where the actual position and posture of the object to be processed is determined according to any object positioning method in the above embodiments;

[0119] The control module is used to control the end effector of the robot system to move to a position corresponding to the actual posture.

[0120] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to different types.

[0121] Figure 8 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the electronic device 80 includes: a processor 81, and a memory 82 communicatively connected to the processor 81, and the memory 82 stores computer-executable instructions.

[0122] Among them, the processor executes the computer execution instructions stored in the memory to implement the object positioning method or the robot system control method provided by any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0123] The embodiments of the present disclosure also provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the object positioning method or the robot system control method provided by any of the above method embodiments.

[0124] The embodiments of the present disclosure also provide a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the object positioning method or the control method of the robot system provided by any of the above method embodiments.

[0125] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0128] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0129] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0131] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for locating an object, characterized in that: Applied to a robot system, the robot system includes a robot and a point cloud acquisition device, and the object positioning method includes: Controlling the point cloud acquisition device to acquire a target point cloud image, wherein the target point cloud image includes a first point cloud of a marking object and a second point cloud of an object to be processed; Inputting the first point cloud into the robot model of the robot system for processing to obtain the first pose of the marker object; Determining a first difference between the first posture and an actual posture of the marker object; Determining an actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model; Execute the object positioning method cyclically; When a first preset condition is met, the loop is terminated to obtain the actual positions and postures of the plurality of objects to be processed, wherein the first preset condition includes at least one of: the number of loops is a preset number, the loop duration is greater than or equal to a preset duration, and the ambient temperature of the robot system reaches a stable state; Among the actual postures of the multiple objects to be processed, the actual posture that meets the second preset condition is determined as the target actual posture, wherein the second preset condition includes: the actual posture of the object to be processed with the highest frequency of occurrence is the target actual posture or the actual posture determined for the last time is the target actual posture.

2. The object positioning method according to claim 1, characterized in that: The first point cloud and the second point cloud are included in the same target point cloud image, and determining the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model includes: Inputting the second point cloud into the robot model of the robot system for processing to obtain a second posture of the object to be processed; A second difference between the second posture and the first difference is determined as the actual posture of the object to be processed.

3. The object positioning method according to claim 1, characterized in that: The first point cloud and the second point cloud are in different target point cloud images, and determining the actual position and posture of the object to be processed according to the first difference, the second point cloud and the robot model includes: Inputting the second point cloud into the robot model of the robot system for processing to obtain a third posture of the object to be processed; determining a third difference between the third pose and the first difference; Acquire a fourth pose of the end effector when acquiring a target point cloud image including the first point cloud, and acquire a fifth pose of the end effector when acquiring a target point cloud image including the second point cloud; Determining a transformation matrix from the fourth posture to the fifth posture; The actual posture is determined according to the third difference and the transformation matrix.

4. The object positioning method according to claim 1, characterized in that: Determining the actual posture of the object to be processed according to the first difference, the second point cloud and the robot model includes: According to the first difference, adjusting the model parameters of the robot model; The second point cloud is input into the adjusted robot model for processing to obtain the actual position and posture of the marker object.

5. The object positioning method according to claim 4, characterized in that: The point cloud acquisition device is fixed on the end effector of the robot, the marker object is fixed outside the robot, the model parameters of the robot model include: the camera external parameters of the point cloud acquisition device and the robot parameters of the robot, and the first point cloud is input into the robot model of the robot system for processing to obtain the first pose of the marker object, including: performing the following steps in the robot model: Determine a first execution posture of the marker object in the first coordinate system of the end effector according to the first point cloud and the camera extrinsic parameters, wherein the camera extrinsic parameters include: a transformation relationship from the second coordinate system of the point cloud acquisition device to the first coordinate system; The first posture of the marker object in a third coordinate system of a robot base is determined according to the first execution posture and the robot parameters.

6. The object positioning method according to claim 5, characterized in that: Before inputting the second point cloud into the adjusted robot model for processing to obtain the actual position and posture of the marker object, the method further includes: The camera extrinsic parameter is adjusted according to a fourth difference between the first execution posture and the actual execution posture of the marker object in the first coordinate system.

7. The object positioning method according to any one of claims 1, 2 and 4, characterized in that: The point cloud acquisition device is fixedly mounted on the outside of the robot, the marker object is fixed on the end effector of the robot, the model parameters of the robot model include: the camera external parameters of the point cloud acquisition device, the first point cloud is input into the robot model of the robot system for processing, and the first pose of the marker object is obtained, including: performing the following steps in the robot model: The first posture of the marker object in the third coordinate system of the robot base is determined according to the first point cloud and the camera extrinsic parameters, wherein the camera extrinsic parameters include: a transformation relationship from the second coordinate system of the point cloud acquisition device to the third coordinate system.

8. The object positioning method according to any one of claims 1 to 4, characterized in that: The determining a first difference between the first posture and the actual posture of the marker object comprises: Acquire a fifth difference between a sixth posture of the marker object and the actual posture of the marker object, wherein the sixth posture is a calculated posture of the marker object determined when the robot system processes an object before the object to be processed; determining a sixth difference between the first pose and the sixth pose; A sum of the sixth difference value and the fifth difference value is determined to be the first difference value.

9. The object positioning method according to any one of claims 1 to 4, characterized in that: Before controlling the point cloud acquisition device to acquire the target point cloud image, the method further includes: When the environment in which the robot system is located is a preset environment, controlling the point cloud acquisition device to acquire a reference point cloud image, wherein the reference point cloud image includes a reference point cloud of the marker object; The reference point cloud is input into the robot model for processing to obtain the actual position and posture of the marker object.

10. A control method for a robot system, characterized in that: include: Acquire an actual position and posture of the object to be processed, wherein the actual position and posture of the object to be processed is determined according to the object positioning method according to any one of claims 1 to 9; The end effector of the robot system is controlled to move to a position corresponding to the actual posture.

11. An object positioning device, characterized in that: The object positioning method according to any one of claims 1 to 9 is applied to a robot system, wherein the robot system comprises a robot and a point cloud acquisition device, and the object positioning device comprises: A control module, used to control the point cloud acquisition device to acquire a target point cloud image, wherein the target point cloud image includes a first point cloud of a marking object and a second point cloud of an object to be processed; a processing module, configured to input the first point cloud into a robot model of the robot system for processing to obtain a first pose of the marker object; A first determination module, configured to determine a first difference between the first posture and an actual posture of the marker object; A second determination module, used to determine the actual posture of the object to be processed according to the first difference, the second point cloud and the robot model; A third determining module, configured to execute the object positioning method cyclically; When a first preset condition is met, the loop is terminated to obtain the actual positions and postures of the plurality of objects to be processed, wherein the first preset condition includes at least one of: the number of loops is a preset number, the loop duration is greater than or equal to a preset duration, and the ambient temperature of the robot system reaches a stable state; Among the actual postures of the multiple objects to be processed, the actual posture that meets the second preset condition is determined as the target actual posture, wherein the second preset condition includes: the actual posture of the object to be processed with the highest frequency of occurrence is the target actual posture or the actual posture determined for the last time is the target actual posture.

12. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the object positioning method according to any one of claims 1 to 9 and / or the robot system control method according to claim 10 are implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the object positioning method according to any one of claims 1 to 9, and / or the control method of the robot system according to claim 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, is used to implement the object positioning method according to any one of claims 1 to 9, and / or the control method of the robot system according to claim 10.

Images