Component positioning method, component positioning device, component positioning system and robot

Abstract

The present application discloses a component positioning method, a component positioning device, a component positioning system and a robot, and belongs to the field of image processing technology. The component positioning method comprises: based on the image data of the component to be grasped, determining the first position information of the component to be grasped in the image frame corresponding to the target field of view, the target field of view includes a cross mark, and the center point of the cross mark is located at the center point of the target field of view; when the inclination angle of the component to be grasped relative to the cross mark is greater than the first target threshold, adjusting the rotation angle of the robot arm based on the first position information; when the inclination angle of the component to be grasped relative to the cross mark is not greater than the first target threshold, adjusting the position of the robot arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame. The component positioning method of the present application significantly improves the accuracy, real-time performance and subsequent grasping success rate of positioning, and is suitable for complex or precise grasping environments.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a component positioning method, a component positioning device, a component positioning system and a robot. Background Art

[0002] The environment of the project construction site is complex and changeable. During the installation and transportation of components, it is often necessary to use a robotic arm to carry the components. In the related technology, professional robotic operators are mainly used to operate the robotic arm, and the position of the robotic arm is adjusted by manual remote control devices to gradually adjust the position of the robotic arm to grab the components to be grabbed. Manual fine-tuning of positioning is time-consuming and has large positioning errors. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a component positioning method, a component positioning device, a component positioning system and a robot, which significantly improve the positioning accuracy and the success rate of subsequent grasping, and are suitable for complex or precise grasping environments.

[0004] In a first aspect, the present application provides a component positioning method, the method comprising:

[0005] Based on the image data of the component to be grasped, determining first position information of the component to be grasped in an image frame corresponding to a target field of view, wherein the target field of view includes a cross mark, the center point of the cross mark is located at the center point of the target field of view, and the center point of the cross mark corresponds to the actual position of the robotic arm;

[0006] When the inclination angle of the component to be grasped relative to the cross mark is greater than a first target threshold, adjusting the rotation angle of the robot arm based on the first position information;

[0007] When the inclination angle of the component to be grasped relative to the cross mark is not greater than a first target threshold, the position of the robotic arm is adjusted based on the first position information and the first coordinates of the center point of the cross mark in the image frame.

[0008] According to the component positioning method of the present application, the inclination angle of the component to be grasped in the target field of view is determined by using the pixel position information of the robot arm and the component to be grasped in the image frame corresponding to the target field of view, so as to adjust the rotation angle of the robot arm based on the inclination angle so that the inclination angle is within a reasonable range, and then adjust the position of the robot arm based on the pixel position information of the image frame corresponding to the robot arm and the component to be grasped in the target field of view, so that the robot arm can accurately position the component to be grasped on the basis of being able to work normally, realize real-time positioning, significantly improve the positioning accuracy, positioning real-time performance and subsequent grasping success rate, and solve the technical problems of time-consuming positioning and large positioning errors; and it is suitable for complex or precise grasping environments and different hardware equipment, and solves the technical problem that it cannot be used in precise grasping environments.

[0009] According to an embodiment of the present application, determining first position information of the component to be captured in an image frame corresponding to a target field of view based on image data of the component to be captured includes:

[0010] Binarizing the image data to obtain a binary image corresponding to the component to be captured;

[0011] Traversing the binary image to determine a border area of ​​the component to be captured in the image frame;

[0012] Acquire first position information of at least two pixels in the border area in the image frame.

[0013] According to an embodiment of the present application, adjusting the rotation angle of the robotic arm based on the first position information includes:

[0014] Determine the inclination angle of the component to be grasped relative to the cross mark based on the first position information of the first target pixel point among the at least two pixel points, wherein the first target pixel point is at least two pixel points located on the same side in the frame area;

[0015] The rotation angle is adjusted based on the tilt angle.

[0016] According to an embodiment of the present application, adjusting the position of the robotic arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame includes:

[0017] Based on the first position information of the second target pixel point among the at least two pixel points, determining the second coordinate of the center point of the component to be captured; the second target pixel point is at least two pixel points in the frame area that are centrally symmetrical based on the component to be captured in the image frame;

[0018] Determine the pixel displacement between the component to be grasped and the robotic arm based on the second coordinate and the first coordinate;

[0019] The position of the robotic arm is adjusted based on the pixel displacement.

[0020] According to an embodiment of the present application, adjusting the position of the mechanical arm based on the pixel displacement includes:

[0021] Determining the mechanical displacement of the component to be grasped and the robotic arm based on the pixel displacement and the target conversion coefficient;

[0022] The position of the mechanical arm is adjusted based on the mechanical displacement.

[0023] According to an embodiment of the present application, after adjusting the position of the robotic arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame, the method further includes:

[0024] When the mechanical displacement between the center point of the component to be grasped and the center point of the cross mark is less than a second target threshold, the robotic arm is controlled to grasp the component to be grasped.

[0025] In a second aspect, the present application provides a component positioning device, the device comprising:

[0026] A first processing module, configured to determine, based on image data of a component to be grasped, first position information of the component to be grasped in an image frame corresponding to a target field of view, wherein the target field of view includes a cross mark, a center point of the cross mark is located at a center point of the target field of view, and the center point of the cross mark corresponds to an actual position of the robotic arm;

[0027] A second processing module, configured to adjust a rotation angle of the robot arm based on the first position information when an inclination angle of the component to be grasped relative to the cross mark is greater than a first target threshold;

[0028] The third processing module is used to adjust the position of the robotic arm based on the first position information and the first coordinates of the center point of the cross mark in the image frame when the inclination angle of the component to be grasped relative to the cross mark is not greater than the first target threshold.

[0029] According to the component positioning device of the present application, the inclination angle of the component to be grasped in the target field of view is determined by the pixel position information of the robot arm and the component to be grasped in the image frame in the target field of view, so as to adjust the rotation angle of the robot arm based on the inclination angle so that the inclination angle is within a reasonable range, and then adjust the position of the robot arm based on the pixel position information of the robot arm and the component to be grasped in the image frame in the target field of view, so that the robot arm can accurately position the component to be grasped on the basis of being able to work normally, realize real-time positioning, significantly improve the accuracy of positioning, real-time positioning and the success rate of subsequent grasping, and is suitable for complex or precise grasping environments.

[0030] In a third aspect, the present application provides a component positioning system, comprising:

[0031] A visual system, the visual system is used to collect image data of the component to be grasped and output a cross mark, and the visual system is configured to move synchronously with the robotic arm;

[0032] As described in the second aspect, the component positioning device is electrically connected to the visual system.

[0033] In a fourth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the component positioning method as described in the first aspect above is implemented.

[0034] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the component positioning method as described in the first aspect above is implemented.

[0035] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the component positioning method as described in the first aspect.

[0036] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the component positioning method as described in the first aspect above.

[0037] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0038] The inclination angle of the component to be grasped in the target field of view is determined by the pixel position information of the image frame corresponding to the robot arm and the component to be grasped in the target field of view, so that the rotation angle of the robot arm is adjusted based on the inclination angle so that the inclination angle is within a reasonable range, and then the position of the robot arm is adjusted based on the pixel position information of the image frame corresponding to the robot arm and the component to be grasped in the target field of view, so that the robot arm can accurately position the component to be grasped on the basis of being able to work normally, significantly improving the positioning accuracy and the success rate of subsequent grasping, and solving the technical problems of time-consuming positioning and large positioning errors; and it is suitable for complex or precise grasping environments and different hardware equipment, solving the technical problem that it cannot be used in precise grasping environments.

[0039] Furthermore, by converting the collected image data into a binary image and segmenting the border of the area corresponding to the component to be grasped based on the binary image, the segmentation precision and accuracy can be improved, the error can be reduced, and the accuracy of the acquired first position information can be improved.

[0040] Furthermore, the inclination angle of the component to be grasped relative to the cross mark is determined through the first position information of the first target pixel point among at least two pixel points, and the rotation angle of the robot arm is adjusted based on the inclination angle so that the adjusted inclination angle is within the first target threshold, so as to meet the minimum horizontal angle that the robot arm can adapt to when grasping the component to be grasped, ensuring that the robot arm can normally grasp the component to be grasped, and making the robot arm in a normal working state, thereby helping to improve the grasping efficiency and the grasping success rate.

[0041] Furthermore, the target conversion coefficient is determined by the type of the robotic arm, and the pixel displacement is converted into a mechanical displacement based on the target conversion coefficient. Then, the position of the robotic arm is adjusted based on the mechanical displacement. This is applicable to any type of robotic arm and can further improve the precision and accuracy of control.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0044] Figure 1 This is one of the flowcharts of the component positioning method provided in the embodiment of the present application;

[0045] Figure 2 This is the second flow chart of the component positioning method provided in the embodiment of the present application;

[0046] Figure 3This is one of the principle schematic diagrams of the component positioning method provided in the embodiment of the present application;

[0047] Figure 4 This is the second principle schematic diagram of the component positioning method provided in the embodiment of the present application;

[0048] Figure 5 This is the third principle schematic diagram of the component positioning method provided in the embodiment of the present application;

[0049] Figure 6 is a schematic diagram of the structure of a component positioning device provided in an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of the structure of the robot provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0052] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0053] The component positioning method, component positioning device, component positioning system, robot and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0054] The component positioning method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0055] The terminal includes but is not limited to a robot.

[0056] The component positioning method provided in the embodiment of the present application may be executed by a robot or a functional module or functional entity in the robot that can implement the component positioning method. The component positioning method provided in the embodiment of the present application is described below using a robot as an example of the execution subject.

[0057] It should be noted that the component positioning method is applied in the field of robotic arm grasping components.

[0058] like Figure 1 As shown, the component positioning method includes: step 110, step 120 and step 130.

[0059] Step 110: determining first position information of the component to be grasped in an image frame corresponding to a target field of view based on image data of the component to be grasped, wherein the target field of view includes a cross mark, the center point of the cross mark is located at the center point of the target field of view, and the center point of the cross mark corresponds to the actual position of the robotic arm;

[0060] In this step, the component to be grasped may be a photovoltaic component or other components.

[0061] like Figure 3 As shown, the component 320 to be grasped may include one or more components. In the case of multiple components, the multiple components can be stacked from bottom to top. During the grasping process, the robotic arm grasps them in sequence based on a certain order, such as from top to bottom or from left to right.

[0062] Image data may be acquired by an image sensor.

[0063] It is understandable that in the actual grasping process, the visual system can be set to collect image data and output a cross mark.

[0064] Among them, the cross mark is used to position the robotic arm.

[0065] During the image acquisition process, an image frame can be generated based on the image data in the field of view at each acquisition moment, and the image frame corresponds to the acquisition time one by one.

[0066] Continue to refer Figure 3 , the target field of view 330 is the maximum range that the visual system 310 can observe, and the cross mark is used to determine the position of the gripper of the robot arm, such as Figure 3 The x-axis and y-axis passing through the origin C1 are the cross marks output by the visual system 310 .

[0067] In the actual implementation process, the visual system 310 can be set in front of the component to be captured 320 to collect image data of the image to be captured and output a cross mark.

[0068] In some embodiments, continue to refer to Figure 3 The cross mark can be output by the cross laser 312 in the vision system 310.

[0069] Figure 4An image frame 340 corresponding to a target field of view is illustrated, wherein the center point of the image frame 340 corresponding to the target field of view 330 is point C1, and the cross marks are the horizontal axis x-axis and the vertical axis y-axis with C1 as the origin. The center point C1 of the cross mark coincides with the center point of the target field of view 330 (i.e., the center point of the image frame 340 corresponding to the target field of view 330), and the center point C1 of the cross mark is the projection point of the center point of the gripper of the robot arm on the image frame 340.

[0070] In the actual implementation process, the visual system 310 needs to keep synchronous movement with the robotic arm, that is, when the robotic arm moves, the target field of view 330 and the position of the cross mark will change accordingly.

[0071] The image data is image information corresponding to the projection of the component to be captured 320 on the image frame 340 corresponding to the target field of view 330, including but not limited to color values ​​and pixel positions.

[0072] The first position information is pixel position information of the projection of the component to be captured 320 on the image frame 340 in the image frame 340 .

[0073] Continue to refer Figure 4 For example, rectangle p1p2p4p3 is the projection of the component to be captured 320 on the image frame 340. Based on the image data of the component to be captured 320, the pixel position information corresponding to each pixel point in the projection of the component to be captured 320 on the image frame 340 can be determined.

[0074] like Figure 2 As shown, in some embodiments, step 110 may include:

[0075] Binarize the image data to obtain a binary image corresponding to the component to be captured;

[0076] Traverse the binary image to determine the border area of ​​the component to be captured in the image frame;

[0077] Obtain first position information of at least two pixels in the border area in the image frame.

[0078] In this embodiment, the binarization process converts a 256-bit grayscale image or an RGB color image into a 2-bit black-and-white image.

[0079] After obtaining the binary image, the position area of ​​the component to be captured 320 (i.e., the projection area of ​​the component to be captured 320 on the image frame 340) can be segmented from the image frame 340 corresponding to the target field of view 330, and then the first position information of at least two pixels on the border area of ​​the position area of ​​the component to be captured 320 can be extracted by traversing the binary image row by row and column by column.

[0080] like Figure 4 As shown, the positions p1, p2, p3 and p4 of the top, bottom, left and right non-zero points of the projection area p1p2p4p3 where the component to be captured 320 is located are extracted, and the pixel positions (i.e., the first position information) of p1, p2, p3 and p4 in the image frame 340 corresponding to the target field of view 330 are calculated: p1(x1, y1), p2(x2, y2), p3(x3, y3) and p4(x4, y4).

[0081] According to the component positioning method provided in the embodiment of the present application, by converting the collected image data into a binary image and segmenting the border of the area corresponding to the component 320 to be grasped based on the binary image, the segmentation accuracy and precision can be improved, the error can be reduced, and the accuracy of the first position information obtained can be improved, thereby solving the technical problem of large positioning error.

[0082] Step 120: When the inclination angle of the component to be grasped relative to the cross mark is greater than the first target threshold, adjust the rotation angle of the robot arm based on the first position information;

[0083] In this step, the tilt angle is used to characterize the angle of the component to be grasped 320 relative to the target reference line in the target field of view 330 .

[0084] Target baselines can be user-defined.

[0085] It is understandable that the method for determining the corresponding inclination angle is different for different reference lines.

[0086] Continue to refer Figure 4 The projection of the component to be captured 320 in the image frame 340 is a rectangle p1p2p4p3, and the tilt angle can be expressed as the degree of the angle formed by the straight line formed by p1p2 and the x-axis.

[0087] It is understandable that, in the process of grasping the component, the robot arm needs to rotate along the central axis, and the position of the component to be grasped 320 in the target field of view 330 will also change accordingly, and there will be an uncertain tilt. Accordingly, the position of the projection area of ​​the component to be grasped 320 in the image frame 340 will also be tilted accordingly, such as Figure 4 shown.

[0088] Before grasping, the tilt angle between the component to be grasped 320 and the target field of view 330 needs to be adjusted to be within the first target threshold.

[0089] The first target threshold is the maximum value of the tilt angle of the component to be grasped 320 relative to the target field of view 330 when the robot arm is able to grasp normally.

[0090] The first target threshold can be determined based on the minimum horizontal angle that the robotic arm can adapt to when grasping, such as being set to 1° or 3°, etc., which is not limited in this application.

[0091] When the inclination angle of the component to be grasped 320 relative to the cross mark is greater than the first target threshold, it indicates that the rotation angle of the robot arm needs to be adjusted first to reduce the inclination angle between the target field of view 330 and the component to be grasped 320 so that the adjusted inclination angle is less than the first target threshold.

[0092] In some embodiments, step 120 may include:

[0093] Based on the first position information of the first target pixel point among the at least two pixel points, the tilt angle of the component to be grasped 320 relative to the cross mark is determined, and the first target pixel point is at least two pixel points located on the same side in the frame area;

[0094] Adjust the rotation angle based on the tilt angle.

[0095] In this embodiment, continue to refer to Figure 4 The first target pixel points are at least two pixel points located on the same side in the border area of ​​the area corresponding to the component to be captured 320 in the image frame 340, such as pixel points p1(x1, y1) and p2(x2, y2) corresponding to two vertices on the same side p1p2.

[0096] The tilt angle angle can be calculated by the following formula:

[0097]

[0098] Wherein, angle is the tilt angle; (x1, y1) is the pixel coordinate of pixel point p1, and (x2, y2) is the pixel coordinate of pixel point p2.

[0099] According to the minimum horizontal angle that the robot arm can adapt to when grasping, the first target threshold value threshold is set. When the inclination angle angle of the component to be grasped 320 is greater than the first target threshold value threshold, the rotation angle of the robot arm is adjusted so that the adjusted inclination angle angle is less than the first target threshold value threshold, so that the robot arm can grasp the component to be grasped 320 normally. At this time, the projection area of ​​the component to be grasped 320 in the target field of view 330 is as follows: Figure 5 As shown,.

[0100] Of course, in other embodiments, the tilt angle of the component to be captured 320 relative to the cross mark may also be determined based on other pixel points of the component to be captured 320 in the image frame 340, which is not limited in the present application.

[0101] According to the component positioning method provided in an embodiment of the present application, the inclination angle of the component to be grasped 320 relative to the cross mark is determined through the first position information of the first target pixel point among at least two pixel points, and the rotation angle of the robotic arm is adjusted based on the inclination angle so that the adjusted inclination angle is within the first target threshold value, so as to meet the minimum horizontal angle that the robotic arm can adapt to when grasping the component to be grasped 320, ensure that the robotic arm can normally grasp the component to be grasped 320, and make the robotic arm in a normal working state, thereby helping to improve the grasping efficiency and the grasping success rate.

[0102] Step 130: When the inclination angle of the component to be grasped relative to the cross mark is not greater than the first target threshold, adjust the position of the robot arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame.

[0103] In this step, the first coordinates are the pixel coordinates of the center point of the target field of view 330 in the image frame 340 , that is, the pixel coordinates of the center point C1 of the cross mark in the image frame 340 corresponding to the target field of view 330 .

[0104] The first position information may be expressed as the pixel coordinates of the center point C2 of the area where the component to be captured 320 is located in the image frame 340 .

[0105] Continue to refer Figure 5 When the inclination angle of the component to be grasped 320 relative to the cross mark is not greater than the first target threshold, the position of the robotic arm can be adjusted based on the first position information and the first coordinate so that the first coordinate of the center point C1 of the adjusted cross mark approximately coincides with the pixel point coordinate of the center point C2 of the area where the component to be grasped 320 is located in the image frame 340.

[0106] It should be noted that before executing step 130 , it is necessary to ensure that the inclination angle of the component to be grasped 320 relative to the cross mark is not greater than the first target threshold.

[0107] In some embodiments, continue to refer to Figure 2 In the case where the inclination angle of the component to be grasped 320 relative to the cross mark is greater than the first target threshold, steps 110 and 120 can be repeated to readjust the rotation angle of the robot arm until the adjusted inclination angle of the component to be grasped 320 relative to the cross mark is not greater than the first target threshold, and then step 130 is entered.

[0108] In this step, after adjusting the inclination angle of the component to be grasped 320 relative to the cross mark to within the first target threshold, the position of the robotic arm is adjusted based on the first position information and the first coordinate of the center point, so that the robotic arm can maintain a normal working state while accurately positioning, thereby improving the grasping accuracy and the grasping success rate and efficiency.

[0109] The inventor discovered during the research and development process that in the related technology, there is a method for determining the offset of a robotic arm from a preset standard position based on a predetermined distance formula and image data corresponding to a specific mark on the collected component to be grasped, and adjusting the posture of the robotic arm based on the offset. However, this method requires determining a distance formula based on a predetermined standard position, and requires setting a specific feature mark on the component to be grasped. The steps are relatively complicated, the workload is large, and it cannot be applied to conventional components to be grasped that do not have specific marks.

[0110] In the related technology, there is another method of determining the relative position of the robot arm and the target object by obtaining the coordinates of the target object under the coordinates of the tracking marker in real time, and determining the movement vector based on the relative position to control the movement of the robot arm. However, this method cannot adjust the rotation angle of the robot arm, and the control accuracy that can be achieved in the end is not high, and it cannot be applied to complex grasping environments.

[0111] In the present application, the pixel position information of the gripper of the robotic arm and the component to be grasped 320 in the target field of view 330 is obtained, and the real-time position relationship between the robotic arm and the component to be grasped 320 is determined based on the pixel position information. The rotation angle and displacement of the robotic arm are adjusted based on the real-time position relationship. There is no need to preset parameters in advance. The implementation process is simple and fast, suitable for any grasping conditions, and can timely adjust the position of the robotic arm based on the real-time position changes of the component to be grasped, thereby realizing real-time positioning and solving the technical problem that positioning can only be performed once in related technologies.

[0112] In addition, in the present application, the inclination angle of the component to be grasped 320 in the target field of view 330 is determined based on the image data of the component to be grasped 320, and the rotation angle of the robot arm is adjusted based on the inclination angle so that the inclination angle is within a reasonable range. On this basis, the position of the robot arm is adjusted based on the first position information and the first coordinate of the center point. This can significantly improve the control accuracy and grasping success rate, and is suitable for any complex or precise grasping environment, thereby solving the technical problem that the related technology cannot be used for precise grasping environments.

[0113] According to the component positioning method provided in the embodiment of the present application, the tilt angle of the component to be grasped 320 in the target field of view 330 is determined by using the pixel position information of the image frame 340 corresponding to the robotic arm and the component to be grasped 320 in the target field of view 330, so as to adjust the rotation angle of the robotic arm based on the tilt angle so that the tilt angle is within a reasonable range, and then adjust the position of the robotic arm based on the pixel position information of the image frame 340 corresponding to the robotic arm and the component to be grasped 320 in the target field of view 330, so that the robotic arm can accurately locate the component to be grasped 320 on the basis of being able to work normally, and real-time positioning can be achieved without manual control by the user, which significantly improves the accuracy, efficiency and timeliness of positioning and the success rate of subsequent grasping, and solves the technical problems of time-consuming positioning and large positioning errors; and it is suitable for complex or precise grasping environments and different hardware equipment, and solves the technical problem that it cannot be used in precise grasping environments.

[0114] In some embodiments, step 130 may include:

[0115] Based on the first position information of the second target pixel point among the at least two pixel points, the second coordinate of the center point of the component to be grasped is determined; the second target pixel point is at least two pixel points in the frame area that are centrally symmetrical based on the component to be grasped;

[0116] Determine the pixel displacement between the component to be grasped and the robotic arm based on the second coordinate and the first coordinate;

[0117] Adjust the position of the robot based on pixel displacement.

[0118] In this embodiment, if Figure 5 As shown, the second target pixel point is the border area of ​​the area formed by the component to be captured 320 in the image frame 340, and is at least two pixel points that are centrally symmetric based on the area where the component to be captured 320 is located, such as the diagonals p1 (x1, y1) and p4 (x4, y4).

[0119] Then, based on the formula: C2=(x1+(x4-x1) / 2, y1+(y4-y1) / 2), the second coordinate of the center point C2 of the area where the component to be captured 320 is located in the image frame 340 corresponding to the target field of view 330 can be determined.

[0120] The first coordinate of the center point C1 of the cross mark can also be determined based on a similar processing method, which will not be described in detail here.

[0121] Continue to refer Figure 2 After the first coordinate and the second coordinate are obtained, the pixel displacement can be determined based on the difference between the horizontal and vertical coordinates of the first coordinate and the second coordinate.

[0122] For example, the pixel displacement can be determined based on the following formula:

[0123] Dx=C1(x)-C2(x)

[0124] Dy=C1(y)-C2(y)

[0125] Wherein, Dx is the lateral displacement in the pixel displacement, Dy is the longitudinal displacement in the pixel displacement, C1 is the first coordinate, and C2 is the second coordinate.

[0126] It should be noted that the pixel displacement includes displacement size and displacement direction. The displacement size is the pixel distance, and the displacement direction can be represented by positive and negative signs. For example, the positive and negative signs can be used to indicate that the robot arm needs to be adjusted up, down, left, or right.

[0127] After obtaining the pixel displacement, based on the displacement direction and displacement size corresponding to the pixel displacement, the robotic arm can be controlled to adjust the corresponding displacement size in the corresponding displacement direction, so that the first coordinate of the center point of the cross mark approximately coincides with the pixel point coordinate of the center point of the area where the component 320 to be grasped is located in the image frame 340.

[0128] According to the component positioning method provided in the embodiment of the present application, the pixel displacement of the robotic arm and the component to be grasped 320 in the image frame 340 corresponding to the target field of view 330 is determined by the second coordinate and the first coordinate of the center point of the component to be grasped 320, so as to adjust the position of the robotic arm based on the pixel displacement, with high control accuracy, thereby solving the technical problem of large positioning error.

[0129] In some embodiments, adjusting the position of the robotic arm based on the pixel displacement may include:

[0130] Based on the pixel displacement and the target conversion coefficient, the mechanical displacement between the component to be grasped and the robotic arm is determined;

[0131] Adjust the position of the robot arm based on the mechanical displacement.

[0132] In this embodiment, the pixel displacement is the displacement between pixel points in the image frame corresponding to the target field of view 330 .

[0133] The mechanical displacement is the actual displacement of the robot arm and the component to be grasped 320 in the three-dimensional space.

[0134] The target conversion factor is a conversion factor for converting pixel displacement to mechanical displacement.

[0135] The target conversion coefficient is determined based on the type of robot arm, and different types of robot arms have different conversion coefficients.

[0136] For example, the mechanical displacement can be determined based on the following formula:

[0137] Rx=Dx*r_x

[0138] Ry=Dy*r_y

[0139] Wherein, R(x, y) represents mechanical displacement, D(x, y) represents pixel displacement, Dx represents the conversion coefficient in the horizontal direction, and Dy represents the conversion coefficient in the vertical direction.

[0140] According to the component positioning method provided in the embodiment of the present application, the target conversion coefficient is determined by the type of the robotic arm, and the pixel displacement is converted into a mechanical displacement based on the target conversion coefficient. Then, the position of the robotic arm is adjusted based on the mechanical displacement. This method is applicable to any type of robotic arm and can further improve the precision and accuracy of control.

[0141] In some embodiments, after step 130, the method may further include: controlling the robot arm to grasp the component to be grasped when the mechanical displacement between the center point of the component to be grasped and the center point of the cross mark is less than a second target threshold.

[0142] In this embodiment, the second target threshold is used to characterize the maximum allowable range of deviation between the center point of the gripper of the robot arm and the center point of the component to be grasped 320 under normal grasping conditions, and the second target threshold can be user-defined.

[0143] For example, after obtaining the mechanical displacement (Rx, Ry), the size relationship between the mechanical displacement (Rx, Ry) and the second target threshold (threshold_x, threshold_y) is compared. When it is determined that Rx is less than threshold_x and Ry is less than threshold_y, the positioning is determined to be successful, and the robotic arm is controlled to grasp the component to be grasped 320.

[0144] In some embodiments, when the mechanical displacement between the center point of the component to be grasped 320 and the center point of the cross mark is not less than the second target threshold, steps 110 to 130 are repeated to readjust the rotation angle and / or position of the robot arm.

[0145] According to the component positioning method provided in the embodiment of the present application, by adjusting the rotation angle of the robotic arm so that the tilt angle is within a reasonable range, and then further adjusting the position of the robotic arm, on the basis of ensuring that the robotic arm is accurately positioned and can grasp normally, the robotic arm is controlled to grasp the component to be grasped 320, which can significantly improve the grasping efficiency and grasping success rate.

[0146] The component positioning method provided in the embodiment of the present application can be executed by a component positioning device. In the embodiment of the present application, the component positioning device executing the component positioning method is taken as an example to illustrate the component positioning device provided in the embodiment of the present application.

[0147] An embodiment of the present application also provides a component positioning device.

[0148] like Figure 6 As shown, the component positioning device includes: a first processing module 610 , a second processing module 620 and a third processing module 630 .

[0149] A first processing module 610 is used to determine first position information of the component to be grasped in an image frame corresponding to a target field of view based on image data of the component to be grasped, wherein the target field of view includes a cross mark, the center point of the cross mark is located at the center point of the target field of view, and the center point of the cross mark corresponds to the actual position of the robot arm;

[0150] A second processing module 620 is used to adjust the rotation angle of the robot arm based on the first position information when the inclination angle of the component to be grasped relative to the cross mark is greater than the first target threshold;

[0151] The third processing module 630 is used to adjust the position of the robot arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame when the inclination angle of the component to be grasped relative to the cross mark is not greater than the first target threshold.

[0152] According to the component positioning device provided in the embodiment of the present application, the inclination angle of the component to be grasped in the target field of view is determined by using the pixel position information of the robot arm and the component to be grasped in the image frame corresponding to the target field of view, so as to adjust the rotation angle of the robot arm based on the inclination angle so that the inclination angle is within a reasonable range, and then adjust the position of the robot arm based on the pixel position information of the image frame corresponding to the robot arm and the component to be grasped in the target field of view, so that the robot arm can accurately position the component to be grasped on the basis of being able to work normally, realize real-time positioning, significantly improve the positioning accuracy, positioning real-time performance and subsequent grasping success rate, and solve the technical problems of time-consuming positioning and large positioning errors; and it is suitable for complex or precise grasping environments and different hardware equipment, and solves the technical problem that it cannot be used in precise grasping environments.

[0153] In some embodiments, the first processing module 610 may be used to:

[0154] Binarize the image data to obtain a binary image corresponding to the component to be captured;

[0155] Traverse the binary image to determine the border area of ​​the component to be captured in the image frame;

[0156] Obtain first position information of at least two pixels in the border area in the image frame.

[0157] In some embodiments, the second processing module 620 may be used to:

[0158] Based on first position information of a first target pixel point among at least two pixel points, determining an inclination angle of the component to be grasped relative to the cross mark, wherein the first target pixel point is at least two pixel points located on the same side in the frame area;

[0159] Adjust the rotation angle based on the tilt angle.

[0160] In some embodiments, the third processing module 630 may be used to:

[0161] Based on the first position information of the second target pixel point among the at least two pixel points, the second coordinate of the center point of the component to be captured is determined; the second target pixel point is at least two pixel points in the frame area that are centrally symmetrical based on the component to be captured in the image frame;

[0162] Determine the pixel displacement between the component to be grasped and the robotic arm based on the second coordinate and the first coordinate;

[0163] Adjust the position of the robot based on pixel displacement.

[0164] In some embodiments, the third processing module 630 may be used to:

[0165] Based on the pixel displacement and the target conversion coefficient, the mechanical displacement between the component to be grasped and the robotic arm is determined;

[0166] Adjust the position of the robot arm based on the mechanical displacement.

[0167] In some embodiments, the device may further include a fourth processing module for controlling the robot arm to grasp the component to be grasped when the mechanical displacement between the center point of the component to be grasped and the center point of the cross mark is less than a second target threshold.

[0168] The component positioning device in the embodiment of the present application may be a robot, or a component in the robot, such as an integrated circuit or a chip, and the embodiment of the present application does not make any specific limitation.

[0169] The component positioning device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0170] The component positioning device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0171] An embodiment of the present application also provides a component positioning system.

[0172] like Figure 3As shown, the component positioning system includes a vision system 310 and a component positioning device as described in any of the above embodiments.

[0173] The visual system 310 is used to collect image data of the component to be grasped 320 and provide a target field of view 330 . The visual system 310 is configured to move synchronously with the robot arm.

[0174] The component positioning device is electrically connected to the visual system 310, and is used to receive image data collected by the visual system 310, and execute the component positioning method described in any of the above embodiments based on the image data.

[0175] According to the component positioning system provided in the embodiment of the present application, by setting a visual system 310 to collect image data of the component to be grasped 320 and provide a target field of view 330 and a cross mark for characterizing the position of the robotic arm, the component positioning device can determine the pixel position information of the robotic arm and the component to be grasped 320 in the image frame 340 where the target field of view 330 is located based on the image data, and further determine the inclination angle of the component to be grasped 320 in the target field of view 330, so as to adjust the rotation angle of the robotic arm based on the inclination angle so that the inclination angle is within a reasonable range, and then adjust the position of the robotic arm based on the pixel position information of the robotic arm and the component to be grasped 320 in the image frame 340 where the target field of view 330 is located, so that the robotic arm can accurately locate the component to be grasped 320 on the basis of being able to work normally, thereby realizing real-time positioning, significantly improving the accuracy of positioning, real-time positioning and the success rate of subsequent grasping, and is suitable for complex or precise grasping environments.

[0176] Continue to refer Figure 3 In some embodiments, the vision system 310 may include: a cross laser 312 and an image sensor 311 .

[0177] In this embodiment, the cross laser 312 is used to output a cross mark, and the cross mark is located at the center point of the target field of view 330 .

[0178] The image sensor 311 is used to provide a target field of view 330 and to collect image data.

[0179] The image sensor 311 may be a visual camera.

[0180] Continue to refer Figure 3 In some embodiments, the cross laser 312 and the image sensor 311 can be arranged relatively parallel, and in actual application, they are placed in front of the component to be grasped 320.

[0181] In some embodiments, Figure 7As shown, an embodiment of the present application also provides a robot 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned component positioning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0182] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0183] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned component positioning method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0185] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned component positioning method when executed by a processor.

[0186] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0187] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned component positioning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0188] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0189] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0191] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0192] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0193] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A component positioning method, characterized in that: include: Based on the image data of the component to be grasped, determining first position information of the component to be grasped in an image frame corresponding to a target field of view, wherein the target field of view includes a cross mark, the center point of the cross mark is located at the center point of the target field of view, and the center point of the cross mark corresponds to the actual position of the robotic arm; When the inclination angle of the component to be grasped relative to the cross mark is greater than a first target threshold, adjusting the rotation angle of the robot arm based on the first position information; When the inclination angle of the component to be grasped relative to the cross mark is not greater than a first target threshold, adjusting the position of the robot arm based on the first position information and a first coordinate of a center point of the cross mark in the image frame; The determining, based on the image data of the component to be grasped, first position information of the component to be grasped in the image frame corresponding to the target field of view includes: Binarizing the image data to obtain a binary image corresponding to the component to be captured; Traversing the binary image to determine a border area of ​​the component to be captured in the image frame; Acquire first position information of at least two pixels in the border area in the image frame; The adjusting the rotation angle of the robotic arm based on the first position information includes: Determine the inclination angle of the component to be grasped relative to the cross mark based on the first position information of the first target pixel point among the at least two pixel points, wherein the first target pixel point is at least two pixel points located on the same side in the frame area; adjusting the rotation angle based on the tilt angle; The adjusting the position of the robotic arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame includes: Based on the first position information of the second target pixel point among the at least two pixel points, determining the second coordinate of the center point of the component to be captured; the second target pixel point is at least two pixel points in the frame area that are centrally symmetrical based on the component to be captured in the image frame; Determine the pixel displacement between the component to be grasped and the robotic arm based on the second coordinate and the first coordinate; The position of the robotic arm is adjusted based on the pixel displacement.

2. The component positioning method according to claim 1, characterized in that: The adjusting the position of the mechanical arm based on the pixel displacement comprises: Determining the mechanical displacement of the component to be grasped and the robotic arm based on the pixel displacement and the target conversion coefficient; The position of the mechanical arm is adjusted based on the mechanical displacement.

3. The component positioning method according to any one of claims 1 to 2, characterized in that: After adjusting the position of the robotic arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame, the method further includes: When the mechanical displacement between the center point of the component to be grasped and the center point of the cross mark is less than a second target threshold, the robotic arm is controlled to grasp the component to be grasped.

4. A component positioning device, characterized in that: include: A first processing module, configured to determine, based on image data of a component to be grasped, first position information of the component to be grasped in an image frame corresponding to a target field of view, wherein the target field of view includes a cross mark, a center point of the cross mark is located at a center point of the target field of view, and the center point of the cross mark corresponds to an actual position of the robotic arm; A second processing module, configured to adjust a rotation angle of the robot arm based on the first position information when an inclination angle of the component to be grasped relative to the cross mark is greater than a first target threshold; A third processing module is used to adjust the position of the robot arm based on the first position information and the first coordinate of the center point of the cross mark in the image frame when the inclination angle of the component to be grasped relative to the cross mark is not greater than a first target threshold; The first processing module is further used to perform binarization processing on the image data to obtain a binary image corresponding to the component to be captured; traverse the binary image to determine the border area of ​​the component to be captured in the image frame; Acquire first position information of at least two pixels in the border area in the image frame; The second processing module is further used to determine the inclination angle of the component to be grasped relative to the cross mark based on the first position information of the first target pixel point among the at least two pixel points, wherein the first target pixel point is at least two pixel points located on the same side in the border area; adjusting the rotation angle based on the tilt angle; The third processing module is also used to determine the second coordinates of the center point of the component to be captured based on the first position information of the second target pixel point among the at least two pixel points; the second target pixel point is at least two pixel points in the frame area that are centrally symmetrical based on the component to be captured in the image frame; Determine the pixel displacement between the component to be grasped and the robotic arm based on the second coordinate and the first coordinate; The position of the robotic arm is adjusted based on the pixel displacement.

5. A component positioning system, characterized in that: include: A visual system, the visual system is used to collect image data of the component to be grasped and output a cross mark, and the visual system is configured to move synchronously with the robotic arm; The component positioning device as claimed in claim 4, wherein the component positioning device is electrically connected to the vision system.

6. The component positioning system according to claim 5, characterized in that: The visual system comprises: An image sensor, the image sensor is used to provide a target field of view and collect the image data; A cross laser, the cross laser is used to output the cross mark, and the cross mark is located at the center point of the target field of view.

7. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the component positioning method as described in any one of claims 1-3 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the component positioning method according to any one of claims 1 to 3 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the component positioning method according to any one of claims 1 to 3 is implemented.

Images