A visual calibration method, terminal and computer storage medium

Abstract

The present application relates to a visual calibration method, terminal, and computer storage medium, wherein the calibration method includes: determining the visual state of a machine based on the first visual coordinates of a first identifier; when the machine is in a visual deviation state, determining the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine; and performing visual calibration on the machine based on the visual deviation value of the machine. The visual calibration method, terminal, and computer storage medium provided in the present application can determine the visual state of a machine based on the first visual coordinates of the first identifier, thereby improving the detection efficiency and accuracy of machine vision; can also determine the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, and perform visual calibration on the machine based on the visual deviation value of the machine, thereby improving the calibration efficiency and accuracy of machine vision.

Description

Technical Field

[0001] The present application belongs to the field of automated production technology, and in particular relates to a visual calibration method, a terminal, and a computer storage medium. Background Art

[0002] The rapid development of the warehousing and logistics industry has brought significant convenience to people's lives. Robotic arms, due to their operational flexibility, have been widely used in the industry. Among the components of a robotic arm, the vision element is a key component. As the eyes of the robotic arm, the performance of the vision element directly determines the accuracy of the robotic arm's grasping and movement.

[0003] Currently, robotic arms locate and grasp objects based on color-depth images captured by visual sensors. After a grasp, the robotic arm uses the visual sensor to capture a second image, obtaining a new color-depth image. This information is then used to optimize the grasping process, and the grasping process is updated iteratively. However, this method does not check the robotic arm's visual state before operation, making it difficult to eliminate visual deviations.

[0004] Therefore, how to improve the detection efficiency and accuracy of machine vision, as well as the calibration efficiency and accuracy of machine vision has become an urgent problem to be solved. Summary of the Invention

[0005] In response to the above technical problems, the present application provides a vision calibration method, a terminal and a computer storage medium to improve the detection efficiency and accuracy of machine vision, as well as the calibration efficiency and accuracy of machine vision.

[0006] The present application provides a visual calibration method, including: determining the visual state of a machine based on the first visual coordinates of a first identifier; when the machine is in a visual deviation state, determining the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine; and performing visual calibration on the machine based on the visual deviation value of the machine.

[0007] In one embodiment, before the step of determining the visual state of the machine based on the first visual coordinates of the first identifier, the step includes: when the machine is in a first preset position, obtaining image information of the first identifier, wherein the image information of the first identifier includes depth information of the pixel points of the first identifier; and determining the first visual coordinates of the first identifier based on the depth information of the characteristic pixel points of the first identifier and the size information of the first identifier.

[0008] In one embodiment, the step of determining the visual state of the machine based on the first visual coordinates of the first identifier includes: if the first visual coordinates meet a first preset condition, the machine is in a visually intact state; if the first visual coordinates meet a second preset condition, the machine is in a first visual deviation state; if the first visual coordinates meet a third preset condition, the machine is in a second visual deviation state.

[0009] In one embodiment, before determining the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, it includes: if the machine is in a first visual deviation state, determining the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine; if the machine is in a second visual deviation state, outputting a first alarm prompt information.

[0010] In one embodiment, the visual deviation value of the machine is determined based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, including: obtaining the second visual coordinates of the first identifier when the machine is in a second preset position; determining the first visual deviation value of the machine based on the spatial coordinates of the machine and the second visual coordinates of the first identifier, combined with a first preset function equation.

[0011] In one embodiment, the visual deviation value of the machine is determined based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, and also includes: when the machine is in a second preset position, obtaining the third visual coordinates of the second identifier; and determining the second visual deviation value of the machine based on the spatial coordinates of the machine and the third visual coordinates of the second identifier, combined with a second preset function equation.

[0012] In one embodiment, determining the visual deviation value of the machine includes: obtaining an average visual deviation value of the machine; and using the average visual deviation value as the visual deviation value of the machine; wherein the average visual deviation value of the machine is an average of a first visual deviation value of the machine and a second visual deviation value of the machine.

[0013] In one embodiment, the step of performing visual calibration on the machine based on the visual deviation value of the machine includes: if the difference between the visual deviation value of the machine and the initial value of the visual deviation of the machine is less than or equal to a deviation threshold, updating the visual deviation storage value of the machine; if the difference between the visual deviation value of the machine and the initial value of the visual deviation of the machine is greater than the deviation threshold, outputting a second alarm prompt message.

[0014] The present application also provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned calibration method when executing the computer program.

[0015] The present application also provides a computer storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned calibration method are implemented.

[0016] The present application provides a visual calibration method, terminal, and computer storage medium, which can determine the visual state of a machine based on the first visual coordinates of a first identifier, thereby improving the detection efficiency and accuracy of the machine's vision; can also determine the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, and perform visual calibration on the machine based on the visual deviation value of the machine, thereby improving the calibration efficiency and accuracy of the machine's vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of a calibration method according to an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of the calibration method provided in the embodiment of the present application;

[0019] Figure 3 Schematic diagram of the calibration method according to the first embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the specific flow of the calibration method provided in Example 2 of the present application;

[0021] Figure 5 This is a schematic diagram of the structure of the terminal provided in Example 3 of the present application. DETAILED DESCRIPTION

[0022] The technical solution of this application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used herein, "and / or" includes any and all combinations of one or more of the relevant listed items.

[0023] Figure 1 Schematic diagram of the calibration method provided in the embodiment of the present application. Figure 1As shown, the upper left corner O of the space where the robot is located is used as the coordinate origin to establish a three-dimensional space coordinate system; the visual element 1 is fixed at the end of the robot arm, and the center point of the visual element is used as the coordinate origin to establish a three-dimensional visual coordinate system; the first marker m1 is fixed on the upper platform surface of the robot arm base, and its shape is as follows: Figure 2 (a); The second mark m2 is fixed on the lower platform surface of the robot arm base, and its shape is as follows: Figure 2 (b) It's worth noting that the first marker m1 and the second marker m2 can be any other regularly shaped objects with clear pixel information, such as QR codes, rectangular blocks, and so on. The relative positions of the first marker m1 and the second marker m2 can also be adjusted, but for the same visual element, they must have different depth information. Movement of the end of the robotic arm will also cause the visual element to move. When the end point of the robotic arm is at different positions, the visual element captures the marker image information, which is used to detect and calibrate the robotic arm's visual state.

[0024] Figure 3 This is a flow chart of the calibration method provided in Example 1 of this application. Figure 3 As shown, the calibration method of the present application may include the following steps:

[0025] Step S101: determining the visual state of the machine according to the first visual coordinates of the first identifier;

[0026] In one embodiment, before step S101, the following steps are included:

[0027] When the machine is at a first preset position, acquiring image information of a first marker, wherein the image information of the first marker includes depth information of a pixel point of the first marker;

[0028] The first visual coordinates of the first marker are determined according to the depth information of the characteristic pixel point of the first marker and the size information of the first marker.

[0029] Taking a robotic arm as an example, the first preset position of the robotic arm is the zero position of the end point of the robotic arm in the spatial coordinate system after the robotic arm is powered on or the robotic arm completes a single grasping task; when the robotic arm is at the zero position, a color depth image of the first identifier is obtained by a visual element (such as a camera, a camera, etc.) installed at the end of the robotic arm, and the identity of the first identifier is confirmed by the color information of the image. The first visual coordinates of the first identifier in the visual coordinate system are determined by the depth information of the central pixel point of the first identifier, the actual length of any side of the first identifier, and the length in the image. Similarly, when the end point of the robotic arm is at the zero position and the robotic arm is in the initial state after visual debugging is completed, the above method can be used to determine the first visual initial coordinates of the first identifier in the visual coordinate system. Among them, the visual coordinates in the visual coordinate system are 6-dimensional coordinates, including coordinate values ​​in the three directions of x, y, and z, and rotation angles in the three directions of x, y, and z.

[0030] In one embodiment, step S101 includes:

[0031] If the first visual coordinate meets the first preset condition, the machine is in a visually intact state;

[0032] If the first visual coordinates meet the second preset condition, the machine is in a first visual deviation state;

[0033] If the first visual coordinates meet the third preset condition, the machine is in a second visual deviation state.

[0034] Optionally, the first preset condition is that the distances between the first visual coordinate and the first visual initial coordinate in all directions are all less than or equal to a first distance threshold, and the differences between the rotation angles of the first visual coordinate and the first visual initial coordinate in all directions are all less than or equal to a first angle threshold;

[0035] Optionally, the second preset condition is that the distance between the first visual coordinate and the first visual initial coordinate in any direction is greater than a first distance threshold and less than a second distance threshold, and the distances between the first visual coordinate and the first visual initial coordinate in each direction are all less than the second distance threshold, and the differences in the rotation angles between the first visual coordinate and the first visual initial coordinate in each direction are all less than the second angle threshold; or, the differences in the rotation angles between the first visual coordinate and the first visual initial coordinate in any direction are greater than the first angle threshold and less than the second angle threshold, and the differences in the rotation angles between the first visual coordinate and the first visual initial coordinate in each direction are all less than the second angle threshold, and the distances between the first visual coordinate and the first visual initial coordinate in each direction are all less than the second distance threshold;

[0036] Optionally, the third preset condition is that the distance between the first visual coordinate and the first visual initial coordinate in any direction is greater than or equal to a second distance threshold, or the difference between the rotation angle of the first visual coordinate and the first visual initial coordinate in any direction is greater than or equal to a second angle threshold.

[0037] Optionally, the first distance threshold is 5 mm, the second distance threshold is 10 mm, the first angle threshold is 1 degree, and the second angle threshold is 2 degrees.

[0038] Step S102: when the machine is in a visual deviation state, determining the visual deviation value of the machine according to the second visual coordinates of the first identifier and the spatial coordinates of the machine;

[0039] In one embodiment, before determining the visual deviation value of the machine based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, the method includes:

[0040] If the machine is in the first visual deviation state, determining the visual deviation value of the machine according to the second visual coordinates of the first identifier and the spatial coordinates of the machine;

[0041] If the machine is in the second visual deviation state, the first alarm prompt information is output.

[0042] In one embodiment, determining the visual deviation value of the machine based on the second visual coordinates of the first marker and the spatial coordinates of the machine includes:

[0043] When the machine is at a second preset position, obtaining a second visual coordinate of the first identifier;

[0044] A first visual deviation value of the machine is determined according to the spatial coordinates of the machine and the second visual coordinates of the first identifier in combination with a first preset function equation.

[0045] In one embodiment, determining the visual deviation value of the machine based on the second visual coordinates of the first marker and the spatial coordinates of the machine further includes:

[0046] When the machine is at the second preset position, obtaining a third visual coordinate of the second marker;

[0047] According to the spatial coordinates of the machine and the third visual coordinates of the second identifier, combined with the second preset function equation, a second visual deviation value of the machine is determined.

[0048] In one embodiment, determining a visual deviation value of a machine includes:

[0049] Get the average visual deviation of the machine;

[0050] The average visual deviation is used as the visual deviation value of the machine;

[0051] Optionally, the average visual deviation value of the machine is an average value of the first visual deviation value of the machine and the second visual deviation value of the machine.

[0052] Optionally, the first preset function equation and the second preset function equation are two different solutions to the hand-eye calibration function. Since the hand-eye calibration function includes 6 unknown parameters, solving the equation requires at least 6 different sets of data.

[0053] Taking a robotic arm as an example, the spatial coordinates of the robotic arm's end point can be determined based on the spatial coordinates of the robotic arm's base center point, the angles of each robotic arm joint, and the lengths of each robotic arm joint, combined with the robotic arm's kinematic formula. In the spatial coordinate system, a position of the robotic arm's end point that can simultaneously capture both the first and second markers is selected as a second preset position. At least six different second preset positions are selected. When the robotic arm's end point reaches each second preset position, the visual element simultaneously acquires color and depth image information of the first and second markers. The method in step S101 is used to determine the second visual coordinates of the first marker and the third visual coordinates of the second marker. The six sets of data, including the spatial coordinates of the robotic arm's end point and the second visual coordinates of the first marker, are then substituted into a first preset function equation to determine a first visual deviation value for the robotic arm. The six sets of data, including the spatial coordinates of the robotic arm's end point and the third visual coordinates of the second marker, are then substituted into a second preset function equation to determine a second visual deviation value for the robotic arm. The average of the first and second visual deviation values ​​is taken as the visual deviation value for the robotic arm.

[0054] Step S103: Perform visual calibration on the machine according to the visual deviation value of the machine.

[0055] In one embodiment, step S103 includes:

[0056] If the difference between the machine's visual deviation value and the machine's initial visual deviation value is less than or equal to the deviation threshold, then the machine's visual deviation storage value is updated;

[0057] If the difference between the machine's visual deviation value and the machine's initial visual deviation value is greater than the deviation threshold, a second alarm prompt message is output.

[0058] Optionally, the initial value of the visual deviation is the distance value between the machine and the visual element when the machine vision debugging is completed; taking the robotic arm as an example, the initial value of the visual deviation of the robotic arm is the distance value between the end point of the robotic arm and the midpoint of the visual element when the robotic arm vision debugging is completed.

[0059] Furthermore, if the difference between the machine's visual deviation value and the machine's initial visual deviation value is less than or equal to the deviation threshold, the stored visual deviation value of the machine is updated to the machine's current visual deviation value; wherein the current visual deviation value of the machine is the visual deviation value determined in step S102. Optionally, the deviation threshold is 5 mm.

[0060] The calibration method provided in Example 1 of the present application determines the visual state of the machine based on the first visual coordinates of the first identifier, thereby improving the detection efficiency and accuracy of the machine vision; in addition, the visual deviation value of the machine is determined based on the second visual coordinates of the first identifier and the spatial coordinates of the machine, and the machine is visually calibrated based on the visual deviation value of the machine, thereby improving the calibration efficiency and accuracy of the machine vision.

[0061] Figure 4 This is a schematic diagram of the specific flow of the calibration method provided in Example 2 of this application. Figure 4 As shown, the calibration method of the present application may include the following steps:

[0062] Step S201: determining the visual state of the machine according to the first visual coordinates of the first identifier;

[0063] Step S202: determining whether the visual state of the machine is in a good visual state;

[0064] If the visual state of the machine is in a good state, the process returns to step S201;

[0065] If the visual state of the machine is not the visually intact state, then step S203: determining whether the visual state of the machine is the first visual deviation state;

[0066] If the vision state of the machine is the first vision deviation state, step S204 is executed: determining the current value of the vision deviation of the machine according to the second vision coordinates of the first identifier and the spatial coordinates of the machine;

[0067] If the visual state of the machine is not the first visual deviation state, executing step S205: outputting a first alarm prompt message;

[0068] Step S206: determining whether the difference between the current value of the machine's visual deviation and the initial value of the visual deviation is less than or equal to the deviation threshold;

[0069] If the difference between the current value of the machine's visual deviation and the initial value of the visual deviation is less than or equal to the deviation threshold, executing step S207: updating the stored value of the machine's visual deviation to the current value of the visual deviation;

[0070] If the difference between the current value of the machine's visual deviation and the initial value of the visual deviation is greater than the deviation threshold, step S208 is executed: outputting a second alarm prompt message.

[0071] The calibration method provided in Example 2 of the present application further subdivides the visual deviation state, calculates the visual deviation value and automatically performs visual calibration for the state where the visual deviation is controllable, and before automatic calibration, further compares the current value of the visual deviation with the initial value of the visual deviation to perform a secondary detection of the controllable state of the visual deviation; directly outputs an alarm prompt message for the state where the visual deviation is uncontrollable, and performs manual intervention in time, thereby improving the efficiency and accuracy of visual detection and calibration.

[0072] Figure 5 1 is a schematic diagram of the structure of the terminal provided in the third embodiment of the present application. The terminal of the present application includes: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. When the processor 110 executes the computer program 112, the steps in the above-mentioned calibration method embodiment are implemented, for example Figure 3 Steps S101 to S103 are shown.

[0073] The terminal may include, but is not limited to, a processor 110 and a memory 111. Those skilled in the art will appreciate that Figure 5 These are merely examples of terminals and do not constitute a limitation on the terminals. The terminals may include more or fewer components than shown in the figures, or a combination of certain components, or different components. For example, the terminals may also include input and output devices, network access devices, buses, etc.

[0074] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0075] The memory 111 can be an internal storage unit of the terminal, such as the terminal's hard drive or memory. The memory 111 can also be an external storage device of the terminal, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 111 can include both the terminal's internal storage unit and an external storage device. The memory 111 is used to store computer programs and other programs and data required by the terminal. The memory 111 can also be used to temporarily store data that has been output or is about to be output.

[0076] The present application also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned calibration method are implemented.

[0077] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0079] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A visual calibration method, characterized in that: include: determining a visual state of the machine according to the first visual coordinate of the first identifier; When the machine is in a visual deviation state, a visual deviation value of the machine is determined according to the second visual coordinates of the first identifier and the spatial coordinates of the machine; wherein, before determining the visual deviation value of the machine according to the second visual coordinates of the first identifier and the spatial coordinates of the machine, the method includes: If the machine is in a first visual deviation state, determining a visual deviation value of the machine according to the second visual coordinates of the first identifier and the spatial coordinates of the machine; If the machine is in a second visual deviation state, outputting a first alarm prompt message; Determining the visual deviation value of the machine according to the second visual coordinates of the first identifier and the spatial coordinates of the machine includes: When the machine is in a second preset position, obtaining a second visual coordinate of the first marker; determining a first visual deviation value of the machine based on the spatial coordinate of the machine and the second visual coordinate of the first marker in combination with a first preset functional equation; and, when the machine is in the second preset position, obtaining a third visual coordinate of the second marker; determining a second visual deviation value of the machine based on the spatial coordinate of the machine and the third visual coordinate of the second marker in combination with a second preset functional equation; Performing visual calibration on the machine according to the visual deviation value of the machine; wherein determining the visual deviation value of the machine includes: Obtain an average visual deviation value of the machine; and use the average visual deviation value as the visual deviation value of the machine; wherein the average visual deviation value of the machine is an average value of a first visual deviation value of the machine and a second visual deviation value of the machine.

2. The calibration method according to claim 1, wherein: Before the step of determining the visual state of the machine according to the first visual coordinates of the first identifier, the method includes: When the machine is at a first preset position, acquiring image information of the first marker, wherein the image information of the first marker includes depth information of pixels of the first marker; The first visual coordinates of the first marker are determined according to the depth information of the characteristic pixel point of the first marker and the size information of the first marker.

3. The calibration method according to claim 1, wherein: The step of determining the visual state of the machine according to the first visual coordinates of the first identifier includes: If the first visual coordinates meet a first preset condition, the machine is in a visually intact state; If the first visual coordinates meet a second preset condition, the machine is in a first visual deviation state; If the first visual coordinates meet a third preset condition, the machine is in a second visual deviation state.

4. The calibration method according to claim 1, wherein: The step of performing visual calibration on the machine according to the visual deviation value of the machine includes: If the difference between the vision deviation value of the machine and the initial vision deviation value of the machine is less than or equal to the deviation threshold, updating the vision deviation storage value of the machine; If the difference between the vision deviation value of the machine and the initial value of the vision deviation of the machine is greater than the deviation threshold, a second alarm prompt message is output.

5. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the calibration method according to any one of claims 1 to 4 are implemented.

6. A computer storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the calibration method according to any one of claims 1 to 4 are implemented.

Images