Vision-based working point position correction method and device, equipment and storage medium

Abstract

The invention relates to a vision-based working point position correction method and device, equipment and a storage medium. The method comprises the steps that when a target workpiece is controlled to move to a reference point position, an imaging device is used for collecting a target image corresponding to the target workpiece; performing feature recognition on the target image, and determining first pixel coordinates corresponding to feature points in the target image; acquiring a second pixel coordinate corresponding to the feature point in the reference image; wherein the reference image is obtained by carrying out image acquisition on a reference workpiece through an imaging device when the reference workpiece in the standard posture is at the reference point location; determining a correction movement amount of the target workpiece according to the first pixel coordinate and the second pixel coordinate; and the correction movement amount is adopted to correct the working point position of the target workpiece, and the corrected working point position is obtained. According to the scheme provided by the invention, the position of the workpiece can be corrected, and the experiment precision and the experiment safety are improved.

Description

Technical Field

[0001] The present application relates to the field of visual detection technology, and particularly to a method, device, equipment and storage medium for correcting working positions based on vision. Background Art

[0002] In an automated laboratory, to meet the experimental requirements, it is often necessary for each instrument and equipment to operate with high precision. For example, in an automated filtration operation, a robotic arm can be used to control an electric gripper to pick up a syringe and move it above a filter head or above a container containing liquid, so as to perform operations such as fixing the filter head, sucking or releasing liquid.

[0003] To avoid the situation of not being able to pick up the filter head or liquid, or the liquid dripping outside the container, the syringe needs to be kept completely vertical so that the syringe can be aligned with the container mouth in the vertical direction. However, there are problems such as the processing error of the syringe, the processing error of the filter head, the clamping error of the electric gripper, and the error of the syringe fixing the filter head, resulting in the syringe not being completely vertical. Therefore, it is easy to cause damage to the clamped object, waste of liquid, and pollution caused by the liquid dripping outside the container, thus affecting the experimental accuracy and safety. Summary of the Invention

[0004] To solve or partially solve the problems existing in the related art, the present application provides a method, device, equipment and storage medium for correcting working positions based on vision, which can correct the position of a workpiece when the position of the workpiece deviates due to processing errors, operation errors, etc., and improve the experimental accuracy and experimental safety.

[0005] The first aspect of the present application provides a method for correcting working positions based on vision, and the method includes:

[0006] When controlling a target workpiece to move to a reference position, using an imaging device to collect a target image corresponding to the target workpiece;

[0007] Performing feature recognition on the target image, and determining first pixel coordinates corresponding to feature points in the target image;

[0008] Obtaining second pixel coordinates corresponding to feature points in a reference image; wherein, the reference image is obtained by collecting an image of a reference workpiece at the reference position in a standard posture through the imaging device;

[0009] Determining a correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates;

[0010] Using the correction movement amount to correct the working position of the target workpiece to obtain a corrected working position.

[0011] In one embodiment, the feature recognition of the target image and the determination of the first pixel coordinates corresponding to the feature points in the target image include:

[0012] Match the target image with at least one template image to obtain the target region image of the target workpiece; wherein, the target region image contains the target structure of the target workpiece;

[0013] Process the target region image to obtain the pixel coordinates of the target structure, and determine the pixel coordinates of the target structure as the first pixel coordinates of the feature points in the target image.

[0014] In one embodiment, the at least one template image includes a first template image and a second template image; the matching of the target image with at least one template image to obtain the target region image of the target workpiece includes:

[0015] Match the target image with the first template image to determine the initial region of the target workpiece in the target image, and intercept the initial region image of the target workpiece from the target image according to the initial region; wherein, the structure of the target workpiece contained in the initial region image is more than just the target structure;

[0016] Match the initial region image with the second template image to determine the target region of the target structure of the target workpiece in the initial region image, and intercept the target region image of the target workpiece from the initial region image according to the target region.

[0017] In one embodiment, the processing of the target region image to obtain the pixel coordinates of the target structure includes:

[0018] Perform a Hough transform on the target region image to determine the geometric center of the target structure;

[0019] Determine the pixel coordinates of the geometric center as the pixel coordinates of the target structure.

[0020] In one embodiment, the determination of the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates includes:

[0021] Calculate the difference between the first pixel coordinates and the second pixel coordinates to obtain the first pixel offset of the target workpiece;

[0022] Determine the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image.

[0023] In one embodiment, the method further includes:

[0024] Calibrating the imaging device to obtain the pixel equivalent of the imaging device.

[0025] In one embodiment, the calibrating the imaging device to obtain the pixel equivalent of the imaging device includes:

[0026] Controlling the robotic arm to carry the reference workpiece to move according to preset reference points in the same reference plane; wherein, the reference plane is perpendicular to the direction of the first axis of the robotic arm coordinate system, and the reference point is located within the reference plane;

[0027] Whenever the robotic arm moves to any preset reference point, controlling the imaging device to collect an image of the reference workpiece to obtain multiple calibration images;

[0028] Determining the third pixel coordinates of the feature points of the reference workpiece in each calibration image;

[0029] Calculating the difference between the second pixel coordinates and each third pixel coordinate respectively to obtain the second pixel offset corresponding to each calibration image; and calculating the difference between the robotic arm coordinates of the reference point and the robotic arm coordinates of the preset reference point corresponding to each calibration image respectively to obtain the corresponding actual offset; wherein, the second pixel offset is the pixel distance between the feature points of the reference workpiece in the reference image and in the calibration image;

[0030] Calculating the included angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system according to the second pixel offset corresponding to each calibration image and the corresponding actual offset; wherein, the second axis direction of the robotic arm coordinate system is perpendicular to the first axis direction;

[0031] Calculating the pixel equivalent of the imaging device according to the included angle between the coordinate axes, the second pixel offset corresponding to each calibration image, and the corresponding actual offset.

[0032] In one embodiment, the calculating the pixel equivalent of the imaging device according to the included angle between the coordinate axes, the second pixel offset corresponding to each calibration image, and the corresponding actual offset includes:

[0033] Substituting the included angle between the coordinate axes, the second pixel offset corresponding to each calibration image, and the corresponding actual offset into the following two-dimensional coordinate rotation transformation formula to calculate the pixel equivalent corresponding to each calibration image, and then averaging the pixel equivalents to finally obtain the pixel equivalent of the imaging device:

[0034]

[0035] where x' = Δx i ·coef, y' = Δy i ·coef, x = Δx a , y = Δy a ;

[0036] wherein, the Δx i represents the x coordinate of the second pixel offset, the Δy i represents the y coordinate of the second pixel offset, the coef represents the pixel equivalent of the imaging device, and the Δx a represents the x coordinate of the actual offset, and the Δy i represents the y coordinate of the actual offset.

[0037] In one embodiment, determining the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device includes:

[0038] Substituting the first pixel offset, the pixel equivalent of the imaging device, and the coordinate axis included angle into the following formula to calculate the correction movement amount of the target workpiece:

[0039]

[0040] wherein, the Δx i represents the x coordinate of the first pixel offset, the Δy i represents the y coordinate of the first pixel offset, the coef represents the pixel equivalent of the imaging device, the Δx a represents the x coordinate of the correction movement amount, and the Δy i represents the y coordinate of the correction movement amount.

[0041] In one embodiment, the optical axis of the camera in the imaging device is parallel to the direction of the first axis of the robotic arm coordinate system; the direction of the first axis of the robotic arm coordinate system is the Z-axis direction of the robotic arm coordinate system, and the direction of the second axis of the robotic arm coordinate system is the X-axis direction or the Y-axis direction of the robotic arm coordinate system.

[0042] The second aspect of the present application provides a vision-based working point correction device, including:

[0043] A target image acquisition module, configured to collect a target image corresponding to the target workpiece by using an imaging device when controlling the target workpiece to move to a reference point;

[0044] The first pixel coordinate determination module is configured to perform feature recognition on the target image and determine the first pixel coordinates corresponding to the feature points in the target image;

[0045] The second pixel coordinate acquisition module is configured to acquire the second pixel coordinates corresponding to the feature points in the reference image; wherein, the reference image is obtained by collecting an image of the reference workpiece at the reference position by the imaging device when the reference workpiece is in a standard posture;

[0046] The correction movement amount determination module is configured to determine the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates;

[0047] The working position correction module is configured to correct the working position of the target workpiece by using the correction movement amount to obtain the corrected working position.

[0048] A third aspect of the present application provides an electronic device, including:

[0049] A processor; and

[0050] A memory having executable code stored thereon, which when executed by the processor, causes the processor to execute the method as described above.

[0051] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon, which when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0052] The technical solution provided by the present application may include the following beneficial effects:

[0053] The solution provided by the present application, when controlling the target workpiece to move to the reference position, uses the imaging device to collect the target image corresponding to the target workpiece; performs feature recognition on the target image and determines the first pixel coordinates corresponding to the feature points in the target image; acquires the second pixel coordinates corresponding to the feature points in the reference image; wherein, the reference image is obtained by collecting an image of the reference workpiece at the reference position by the imaging device when the reference workpiece is in a standard posture; determines the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates; corrects the working position of the target workpiece by using the correction movement amount to obtain the corrected working position. The present application collects the target image of the target workpiece and the reference image of the reference workpiece at the same reference position through the imaging device, and determines the correction movement amount of the target workpiece through the first pixel coordinates of the feature points in the target image and the second pixel coordinates of the feature points in the reference image, so as to correct the working position of the target workpiece, and can perform position correction on the workpiece when the position of the workpiece deviates due to processing errors, operation errors, etc., improving the experimental accuracy and experimental safety.

[0054] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] By describing the exemplary embodiments of this application in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of this application will become more apparent, wherein, in the exemplary embodiments of this application, the same reference numerals generally represent the same components.

[0056] Figure 1 is a schematic flowchart of a vision-based working point correction method shown in an embodiment of this application;

[0057] Figure 2 is another schematic flowchart of a vision-based working point correction method shown in an embodiment of this application;

[0058] Figure 3 is a schematic diagram of the positional relationship among a robotic arm, a target workpiece, and an imaging device shown in an embodiment of this application;

[0059] Figure 4 is a diagram of the positional relationship among a robotic arm, an electric gripper, and a target workpiece;

[0060] Figures 5A - 5B is a template image of a syringe;

[0061] Figure 6 is an image of a general syringe intercepted;

[0062] Figure 7 is a reference image of a reference workpiece collected at a reference point;

[0063] Figure 8 is a schematic diagram of a reference plane;

[0064] Figure 9 is one of the calibration images of the reference workpiece;

[0065] Figure 10 are multiple calibration images of the reference workpiece collected at different preset reference point positions;

[0066] Figure 11 is a calibration flowchart of the robotic arm for gripping the reference workpiece;

[0067] Figure 12 is a correction flowchart of the robotic arm for gripping the target workpiece;

[0068] Figure 13 is a schematic structural diagram of a vision-based working point correction device shown in an embodiment of this application;

[0069] Figure 14It is a schematic structural diagram of an electronic device shown in an embodiment of the present application. Detailed implementation manners

[0070] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application will be more thorough and complete, and can fully convey the scope of the present application to those skilled in the art.

[0071] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0072] It should be understood that although the terms "first", "second", "third", etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality" means two or more unless otherwise specifically defined.

[0073] In an automated laboratory, in order to meet the experimental requirements, it is often necessary for each instrument and device to operate with high precision. For example, in an automated filtration operation, a robotic arm can be used to control an electric gripper to pick up a syringe and then move it above a filter head or above a container (such as a test tube) containing a liquid to perform operations such as installing the filter head, sucking or releasing the liquid.

[0074] In order to avoid the situation of not being able to obtain the filter head or the liquid, or the liquid dripping outside the container, the syringe needs to be aligned with the container mouth in the vertical direction. In the related art, a coordinate of the robotic arm is calibrated above the container mouth as the movement destination, so that each time the electric gripper picks up the syringe, the robotic arm moves to this movement destination to perform the filtration operation.

[0075] When the syringe is kept completely vertical in the above operation mode, the error sources are the error in calibrating the coordinates of the robotic arm and the movement error of the robotic arm. Generally, the movement error of the robotic arm is relatively small, while the error in calibrating the coordinates of the robotic arm is mainly related to manual calibration, and its error can be reduced by multiple calibrations. Therefore, the total error can generally be controlled relatively small. However, in the related art, there are the following reasons that cause the syringe not to be completely vertical:

[0076] (1) There are machining errors in the machining of the syringe itself, and it is not completely vertical itself;

[0077] (2) There is an error when the electric claw grabs the syringe, making the posture of the syringe not completely vertical;

[0078] (3) When performing the filtering operation, a filter head can be installed on the nozzle of the syringe, and there may also be machining errors and / or installation errors in the filter head.

[0079] It can be seen that any of the above reasons will cause the syringe not to be completely vertical. Therefore, it will cause the syringe not to be aligned with the filter head, the syringe not to be aligned with the container mouth, and the filter head not to be aligned with the container mouth, affecting the experimental accuracy, and it is also easy to cause damage to the clamped object, waste of liquid, and liquid dripping outside the container to cause pollution, and even endanger the safety of the experimental personnel.

[0080] In view of the above problems, the embodiment of the present application provides a vision-based working point correction method. By using an imaging device to collect a target image of a target workpiece and a reference image of a reference workpiece at the same reference point, and determining the correction movement amount of the target workpiece through the first pixel coordinates of the feature points in the target image and the second pixel coordinates of the feature points in the reference image, the working point of the target workpiece is corrected, which can correct the position of the workpiece when the position of the workpiece deviates due to machining errors, operation errors, etc., and improve the experimental accuracy and experimental safety.

[0081] The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0082] Figure 1 is a schematic flowchart of the vision-based working point correction method shown in the embodiment of the present application.

[0083] See Figure 1 , the vision-based working point correction method of the present application includes:

[0084] S110, when controlling the target workpiece to move to the reference point, use the imaging device to collect the target image corresponding to the target workpiece.

[0085] The imaging device can be a camera, and the embodiments of this application have relatively high hardware requirements for the imaging device. For example, in order to improve the positioning accuracy, an industrial camera with high resolution can be used as the imaging device to collect high-definition images of the target workpiece through the industrial camera. Among them, in order to reduce the errors introduced by teaching errors and the processing errors of the target workpiece, and at the same time to be compatible with target workpieces of different models and sizes, a lens with a larger depth of field and smaller distortion can be used. In addition, in order to meet the requirement of a larger depth of field of the lens, a lighting device can be used to provide lighting function for the target workpiece, where the lighting device can be an annular light source with a relatively long working distance.

[0086] An electric gripper can be installed at the end of the robotic arm, and the robotic arm can control the electric gripper to pick up the target workpiece, where the target workpiece can be a syringe or a syringe with a filter head installed. During the calibration process, the embodiments of this application can pick up the target workpiece through the robotic arm and control the robotic arm to move to the reference point position so that the target workpiece is located at the reference point position.

[0087] The reference point position is related to the installation position of the imaging device. For example, the imaging device can be installed below the robotic arm, and the position directly above the imaging device can be selected as the reference point position. Therefore, when controlling the target workpiece to move to the reference point position, the target image corresponding to the target workpiece can be collected by the imaging device below. Another example is that the imaging device can be installed above the robotic arm and can take pictures of the target workpiece from top to bottom. At this time, the position directly below the imaging device can be selected as the reference point position. It can be understood that according to actual needs, the shooting direction of the imaging device can also be in the horizontal direction or other preset directions, and the shooting direction of the imaging device is not limited here. For example, the imaging device takes pictures of the target workpiece in the horizontal direction from left to right. Another example is that the imaging device takes pictures of the target workpiece at an angle of 45° downward from top to bottom.

[0088] S111, perform feature recognition on the target image and determine the first pixel coordinates corresponding to the feature points in the target image.

[0089] During the calibration process, the target image can be subjected to feature recognition through an image processing algorithm to determine the first pixel coordinates of the feature points of the target workpiece in the target image. For example, by applying image processing algorithms such as Gaussian filtering, color space transformation, template matching, gamma transformation, and Hough transformation to the target image, the first pixel coordinates of the feature points of the target workpiece in the target image can be determined.

[0090] S112, obtain the second pixel coordinates corresponding to the feature points in the reference image; where the reference image is obtained by collecting an image of the reference workpiece at the reference point position through the imaging device when the reference workpiece is in the standard posture.

[0091] During the calibration process, the robotic arm can be controlled to pick up the reference workpiece and move it to the reference point position so that the reference workpiece is located at the reference point position. Then, the imaging device can be used to collect the reference image corresponding to the reference workpiece. Among them, the reference workpiece can be the target workpiece in a standard posture (such as the vertical direction, horizontal direction, etc.), or the same kind of workpiece with the same specifications as the target workpiece in a standard posture. Further, the machining error of the reference workpiece can be much smaller than that of the target workpiece. For example, a straight syringe can be selected as the reference workpiece to reduce the machining error of the syringe itself. In order to improve the calibration accuracy, a metal processing tooling can also be used to replace the syringe as the reference workpiece.

[0092] After obtaining the reference image, the reference image can be subjected to feature recognition through an image processing algorithm to determine the second pixel coordinates of the feature points of the reference workpiece in the reference image. For example, the second pixel coordinates of the feature points of the reference workpiece in the reference image can be determined by applying image processing algorithms such as Gaussian filtering, color space transformation, template matching, gamma transformation, and Hough transformation to the reference image.

[0093] It should be noted that the reference image can be collected and stored once during the initial calibration so that no matter how many corrections are performed subsequently, the embodiments of the present application can directly obtain the stored reference image. In addition, the second pixel coordinates of the feature points in the reference image can be calculated by the same method as the target image each time the reference image is obtained, or the second pixel coordinates can be calculated and stored during the initial calibration so that the stored second pixel coordinates can be directly obtained during each subsequent correction operation.

[0094] S113. Determine the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates.

[0095] After obtaining the first pixel coordinates of the feature points of the target workpiece in the target image and the second pixel coordinates of the feature points of the reference workpiece in the reference image, the deviation between the target workpiece and the reference workpiece can be determined according to the first pixel coordinates and the second pixel coordinates, and the correction movement amount of the target workpiece can be determined based on the deviation between the target workpiece and the reference workpiece.

[0096] S114. Correct the working point position of the target workpiece by using the correction movement amount to obtain the corrected working point position.

[0097] After obtaining the correction movement amount, the correction movement amount can be used to correct the working point position of the target workpiece, so that when the position of the workpiece deviates due to machining errors, operation errors, etc., the position of the workpiece can be corrected, improving the experimental accuracy and experimental safety.

[0098] Among them, the working point position is different from the reference point position. For example, the reference point position can be the position directly above the imaging device (industrial camera), and its function is to take pictures of the syringe through the industrial camera below this position. The working point position is the position directly above the target container (such as a test tube), and its function is to control the syringe to perform the target operation after the robotic arm moves to this position. The target operation can include filtering operations, pipetting operations, etc.

[0099] As can be seen from this example, in the solution provided by the present application, when controlling the target workpiece to move to the reference point position, the imaging device is used to collect the target image corresponding to the target workpiece; feature recognition is performed on the target image, and the first pixel coordinates corresponding to the feature points in the target image are determined; the second pixel coordinates corresponding to the feature points in the reference image are obtained; wherein, the reference image is obtained by collecting an image of the reference workpiece through the imaging device when the reference workpiece is in the standard posture at the reference point position; according to the first pixel coordinates and the second pixel coordinates, the correction movement amount of the target workpiece is determined; the correction movement amount is used to correct the working point position of the target workpiece to obtain the corrected working point position. By collecting the target image of the target workpiece and the reference image of the reference workpiece at the same reference point position through the imaging device, and determining the correction movement amount of the target workpiece through the first pixel coordinates of the feature points in the target image and the second pixel coordinates of the feature points in the reference image, the working point position of the target workpiece is corrected in this way, which can correct the position of the workpiece when the position of the workpiece deviates due to processing errors, operation errors, etc., and improve the experimental accuracy and experimental safety.

[0100] Figure 2 It is another schematic flowchart of the vision-based working point position correction method shown in the embodiments of the present application.

[0101] See Figure 2 , the vision-based working point position correction method of the present application includes:

[0102] S210, when controlling the target workpiece to move to the reference point position, use the imaging device to collect the target image corresponding to the target workpiece.

[0103] Figure 3 It is a schematic diagram of the positional relationship among the robotic arm, the target workpiece, and the imaging device. As Figure 3As shown, the robotic arm can be a robotic arm with multiple degrees of freedom, such as a four-axis robotic arm, a six-axis robotic arm, a seven-axis robotic arm, etc.; the robotic arm can also be a multi-axis translation mechanism, such as an XYZ-axis translation mechanism. When the robotic arm is a robotic arm with multiple degrees of freedom, the base of the robotic arm can be fixed to the imaging device on the same installation plane. Among them, in order to improve the positioning accuracy, a high-resolution industrial camera can be used as the imaging device to take pictures of the reference workpiece / target workpiece; in order to reduce the errors introduced by the teaching error and the machining error of the target workpiece, and at the same time to be compatible with target workpieces of different models and sizes, a lens with a larger depth of field and less distortion can be used; in order to meet the requirement of a larger depth of field of the lens, a ring light source with a longer working distance can be used as the lighting device to provide lighting for the reference workpiece / target workpiece during photographing.

[0104] Such as Figure 3 As shown, the robotic arm is equipped with an electric gripper, and the robotic arm can control the electric gripper to pick up the reference workpiece / target workpiece. Among them, the target workpiece can be a syringe or a syringe fixed with a filter head.

[0105] The reference point is related to the setting position of the imaging device. In one implementation, the imaging device can be set below the robotic arm (such as Figure 3 as shown), and the position at a preset height directly above the imaging device can be calibrated as the reference point. In another implementation, the imaging device can also be set on the side of the robotic arm (such as the left side or the right side), and the position at a preset distance directly beside the imaging device can be calibrated as the reference point. Among them, the preset height or the preset distance can both be related to the focal length of the imaging device. For example, based on the current focal length of the imaging device, the preset height or the preset distance can be set to 20 cm. Therefore, the robotic arm can be controlled to move to the position 20 cm directly above the imaging device, or the robotic arm can be controlled to move to the position 20 cm directly beside the imaging device.

[0106] During the calibration process, after the robotic arm controls the electric gripper to pick up the target workpiece, the robotic arm can be controlled to move to the previously calibrated reference point. If the imaging device is set below the robotic arm, then at this time the electric gripper and the target workpiece are directly above the imaging device (industrial camera), that is, the shooting direction of the industrial camera is upward. Therefore, the industrial camera can take a bottom image of the target workpiece from below, and the bottom image of the target workpiece can be used as the target image of the target workpiece; if the imaging device is set on the side of the robotic arm (such as the left side or the right side), then at this time the electric gripper and the target workpiece are directly beside the imaging device (industrial camera), that is, the shooting direction of the industrial camera is to the right or to the left. The industrial camera can take a side image of the target workpiece from both sides, and the side image of the target workpiece can be used as the target image of the target workpiece.

[0107] As an example, during the calibration process, the robotic arm can be used to control the electric claw to grip the target workpiece. For example, Figure 4 as shown Figure 4 the target workpiece in is a randomly gripped syringe fixed with a filter head. The robotic arm can be controlled to move to the previously recorded reference point, and then the imaging device can be controlled to collect the bottom image or side image of the target workpiece. The collected bottom image or side image of the target workpiece is used as the target image.

[0108] S211, match the target image with at least one template image to obtain the target region image of the target workpiece; wherein, the target region image contains the target structure of the target workpiece.

[0109] Among them, the target structure can be the structure of a specific area on the target workpiece, such as the lowermost structure of the target workpiece, the uppermost structure of the target workpiece, etc., which is not limited here.

[0110] Before performing template matching on the target image, Gaussian filtering and color space transformation can be performed on the target image first. Exemplarily, Gaussian filtering can be performed on the target image to weaken the noise interference in the target image, and then color space transformation can be performed on the target image. For example, the target image can be converted from a color image to a grayscale image to reduce the computational complexity of subsequent algorithm analysis and facilitate the subsequent use of the template matching method. If the imaging device is a black-and-white industrial camera, the target image collected by it is already a grayscale image, so the color space transformation step can be skipped.

[0111] After performing Gaussian filtering and color space transformation on the target image, the template matching step can be entered. For example, the target image can be matched with at least one template image to obtain the target region image of the target workpiece, where the target region image can contain the target structure of the target workpiece.

[0112] It should be noted that the embodiments of the present application do not limit the execution order of the Gaussian filtering step and the color space transformation step, that is, the embodiments of the present application can perform the Gaussian filtering step first and then the color space transformation step, or perform the color space transformation step first and then the Gaussian filtering step.

[0113] It should be noted that template matching is a technology for finding the most matching (similar) part of another template image in an image. The template here refers to a known small image, and template matching is to search for the target in a large image. In other words, the template is the known target to be found in the image, and the target has the same or similar size, direction and image as the template. The target can be found in the image through a certain algorithm, and the coordinate position of the target can be determined.

[0114] In one embodiment, at least one template image includes a first template image and a second template image; matching the target image with at least one template image to obtain the target region image of the target workpiece may include:

[0115] Matching the target image with the first template image to determine the initial region of the target workpiece in the target image, and cropping the initial region image of the target workpiece from the target image according to the initial region; wherein, the structure of the target workpiece included in the initial region image is more than just the target structure; matching the initial region image with the second template image to determine the target region of the target structure of the target workpiece in the initial region image, and cropping the target region image of the target workpiece from the initial region image according to the target region.

[0116] In the embodiments of the present application, at least one template image may include a first template image T1 and a second template image T2. Wherein, the size of the first template image T1 may be larger than the size of the second template image T2. For example, the width and height of the first template image T1 may be respectively larger than the width and height of the second template image T2. In addition, the size of the first template image T1 and the size of the second template image T2 may be the same, as long as it is ensured that the structure information of the target workpiece included in the first template image T1 is more than the structure information included in the second template image T2.

[0117] Reference Figures 5A - 5B , Figure 5A is the first template image T1 of the syringe, Figure 5B is the second template image T2 of the syringe. In terms of the structure information of the target workpiece included in the two template images, the first template image T1 may include the second template image T2, but the first template image T1 and the second template image T2 are not necessarily cropped from the same template image, and the first template image T1 and the second template image T2 may not be in a relationship of equal proportion scaling.

[0118] Wherein, the measured workpiece corresponding to the acquisition of the first template image T1 and the second template image T2 needs to have the characteristics of being clean, flat, burr-free, small machining error, and no hanging liquid droplets. The measured workpiece may be the aforementioned reference workpiece. Reference Figure 5B and Figure 6 , Figure 5B is the second template image T2 of the syringe, Figure 6 is the image of a general syringe cropped. It can be seen that Figure 5B the syringe in Figure 6 has the characteristics of being clean, flat, burr-free, small machining error, and no hanging liquid droplets, while Figure 6 the syringe in

[0119] During the template matching process, the first template image T1 ( Figure 5A ) can be used to perform the first template matching. In a specific implementation, based on the first template image T1, it slides in the target image at a step size to calculate the first matching score, and then based on the first matching score, the initial region of the target workpiece in the target image is determined. Then, the initial region image of the target workpiece is intercepted from the target image according to the initial region. Among them, the initial region image can be an image of the approximate position of the target workpiece (syringe or filter head) in the target image, similar to Figure 5A the first template image T1 shown, that is, the structure of the target workpiece included in the initial region image can be more than just the target structure.

[0120] After obtaining the initial region image, the second template image T2 ( Figure 5B ) can be used to perform the second template matching. In a specific implementation, based on the second template image T2, it slides in the initial region image at a step size to calculate the second matching score, and then based on the second matching score, the target region of the target structure of the target workpiece in the initial region image is determined. Then, the target region image of the target workpiece is intercepted from the initial region image according to the target region. Among them, the target region image can be an image of the relatively accurate position of the target workpiece (syringe or filter head) in the initial region image, similar to Figure 5B the second template image T2 shown, that is, the target region image includes the target structure of the target workpiece.

[0121] In the embodiment of the present application, two template matchings can prevent matching errors, thereby improving the positioning accuracy. At the same time, the size of the image to be matched (target image / reference image / calibration image) can be reduced. Since the template matching uses the sliding window method, that is, the template image slides in the image to be matched at a step size to calculate the matching score. Since the template image becomes smaller, the image to be matched will also become smaller. Therefore, the calculation amount can be greatly reduced, thereby improving the matching efficiency.

[0122] It should be noted that the embodiment of the present application can also be adjusted to one template matching or more template matchings according to the actual situation, and the embodiment of the present application does not limit the number of matchings.

[0123] S212. Process the target region image to obtain the pixel coordinates of the target structure, and determine the pixel coordinates of the target structure as the first pixel coordinates of the feature points in the target image.

[0124] Before processing the target region image, the target region image can be subjected to gamma transformation first. Exemplarily, an adaptive gamma transformation can be performed on the target region image of the target workpiece so as to adjust the overall brightness of the target region image and prevent it from being too bright or too dark.

[0125] After performing gamma transformation on the target region image, the target region image can be processed to obtain the pixel coordinates of the target structure of the target workpiece in the target region image. In the embodiments of the present application, the pixel coordinates of the target structure can be determined as the first pixel coordinates of the feature points of the target workpiece in the target image.

[0126] In one embodiment, processing the target region image to obtain the pixel coordinates of the target structure may include:

[0127] Performing Hough transformation on the target region image to determine the geometric center of the target structure; determining the pixel coordinates of the geometric center as the pixel coordinates of the target structure.

[0128] The shape of the target structure may include a circle, an ellipse, a polygon, etc. If the shape of the target structure is a circle, the center of the circle (i.e., the center of the circle) can be used as the geometric center of the target structure. Therefore, the pixel coordinates of the center of the circle can be determined as the pixel coordinates of the target structure. If the shape of the target structure is an ellipse, the center of the ellipse can be used as the geometric center of the target structure. Therefore, the pixel coordinates of the center of the ellipse can be determined as the pixel coordinates of the target structure. If the shape of the target structure is a polygon, the center of the polygon can be used as the geometric center of the target structure. Therefore, the pixel coordinates of the center of the polygon can be determined as the pixel coordinates of the target structure.

[0129] In a specific implementation, the edge image of the target region image after gamma transformation can be calculated. The edge image can be an image of the edge of the target structure of the target workpiece. For example, the edge image can be an image of the bottom ring edge of the target workpiece. In the edge image, the bottom ring of the target workpiece contains multiple inner circles. Due to machining errors of the target workpiece, such as the syringe nozzle or the filter head nozzle being machined crooked, and clamping errors when the electric claw clamps the target workpiece, such as the electric claw clamping the target workpiece crookedly, the multiple inner circles in the bottom ring of the target workpiece are not necessarily concentric circles. Therefore, in the embodiments of the present application, the target circle with the highest score in the edge image can be determined based on Hough transformation. For example, the inner circles found by Hough transformation can be used as candidate circles, and each candidate circle has a corresponding accumulator value. The candidate circle corresponding to the maximum accumulator value can be determined as the target circle with the highest score. This target circle is the inner circle of the bottom ring of the target workpiece (such as a syringe or a filter head), that is, the circumference of the hole in the bottom ring of the target workpiece. Therefore, the center of the target circle can represent the center of the target structure, and thus the pixel coordinates of the center of the target circle can be determined as the pixel coordinates of the target structure.

[0130] It should be noted that Hough transformation is a fitting strategy that realizes the fitting of straight lines and curves by transforming the coordinates in the image space to the parameter space.

[0131] S213. Obtain the second pixel coordinates corresponding to the feature points in the reference image, where the reference image is obtained by collecting an image of the reference workpiece at the reference position by an imaging device when the reference workpiece is in a standard posture.

[0132] The reference workpiece can be the target workpiece in a standard posture (such as the vertical direction, horizontal direction, etc.), or the same kind of workpiece with the same specifications as the target workpiece in a standard posture. Further, the machining error of the reference workpiece can be much smaller than that of the target workpiece. For example, a straight syringe can be selected as the reference workpiece to reduce the machining error of the syringe itself. To improve the calibration accuracy, a metal machining tooling can also be used to replace the syringe as the reference workpiece.

[0133] During the calibration process, after the robotic arm controls the gripper to pick up the reference workpiece, the robotic arm can be controlled to move to the pre-calibrated reference position. If the imaging device is arranged below the robotic arm (as shown in Figure 3 ), then at this time, the gripper and the reference workpiece are directly above the imaging device (industrial camera), that is, the shooting direction of the industrial camera is upward. Therefore, the industrial camera can take the bottom image of the reference workpiece from below, and the bottom image of the reference workpiece can be used as the reference image of the reference workpiece. If the imaging device is arranged on the side of the robotic arm (such as the left side or the right side), then at this time, the gripper and the reference workpiece are directly on the side of the imaging device (industrial camera), that is, the shooting direction of the industrial camera is to the right or to the left. The industrial camera can take the side image of the reference workpiece from both sides, and the side image of the reference workpiece can be used as the reference image of the reference workpiece.

[0134] As an example, during the calibration process, the robotic arm can be controlled to pick up the reference workpiece by the gripper, and then the robotic arm can be controlled to move to a position at a preset height directly above the imaging device, record the coordinates of the robotic arm at this time, use these coordinates as the reference position, and the imaging device can be controlled to collect the bottom image of the reference workpiece, as shown in Figure 7 shown, Figure 7 is the bottom image of the reference workpiece collected by the imaging device at the reference position, and the collected bottom image of the reference workpiece is used as the reference image.

[0135] After obtaining the reference image, the reference image can be subjected to feature recognition through an image processing algorithm to determine the second pixel coordinates of the feature points of the reference workpiece in the reference image. For example, the second pixel coordinates of the feature points of the reference workpiece in the reference image can be determined through image processing algorithms such as Gaussian filtering, color space transformation, template matching, gamma transformation, and Hough transformation on the reference image.

[0136] It should be noted that the step of determining the second pixel coordinates of the feature points of the reference workpiece in the reference image during the calibration process is the same as the implementation manner of steps S211 to S212 above for determining the first pixel coordinates of the feature points of the target workpiece in the target image during the calibration process. Specifically, reference can be made to steps S211 to S212 above, which will not be elaborated here.

[0137] It should be noted that the reference image can be collected and stored once during the initial calibration so that no matter how many calibrations are performed subsequently, the embodiments of the present application can directly obtain the stored reference image. In addition, the second pixel coordinates of the feature points in the reference image can be calculated by the same method as the target image each time the reference image is obtained, or the second pixel coordinates can be calculated and stored during the initial calibration so that the stored second pixel coordinates can be directly obtained during each subsequent calibration operation.

[0138] It should be noted that the acquisition orientation of the target image is the same as that of the reference image. For example, when the reference image is the bottom image of the reference workpiece, the target image should also be the bottom image of the target workpiece; when the reference image is the side image of the reference workpiece, the target image should also be the side image of the target workpiece.

[0139] S214. Calculate the difference between the first pixel coordinates and the second pixel coordinates to obtain the first pixel offset of the target workpiece.

[0140] After obtaining the first pixel coordinates of the feature points of the target workpiece in the target image and the second pixel coordinates of the feature points of the reference workpiece in the reference image, the difference between the first pixel coordinates and the second pixel coordinates can be calculated, and this difference can be used as the first pixel offset of the target workpiece.

[0141] Among them, the first pixel offset can represent the deviation between the target workpiece and the reference workpiece in the camera coordinate system. Since the reference workpiece is used as the standard for calibration, the larger the first pixel offset, the greater the deviation between the target workpiece and the reference workpiece in the camera coordinate system; the smaller the first pixel offset, the smaller the deviation between the target workpiece and the reference workpiece in the camera coordinate system.

[0142] Let the first pixel offset be Δx i and Δy i , where, Δx i refers to the x coordinate of the first pixel offset, and Δy i refers to the y coordinate of the first pixel offset.

[0143] S215. Determine the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device; where the pixel equivalent is the actual physical distance represented by one pixel in the target image.

[0144] The position and orientation of an object are uniquely determined according to a reference coordinate system. When describing the pose of the same object in different reference coordinate systems, the pose representation will also change accordingly. Therefore, coordinate transformation is needed to relate the descriptions of the object in different reference coordinate systems. Among them, the coordinate transformation between different coordinate systems can include: translation transformation, rotation transformation, and composite transformation.

[0145] In the embodiments of the present application, the first pixel offset is based on the parameters of the camera coordinate system, while the correction of the working point position of the target workpiece requires the use of parameters based on the robot arm coordinate system. Therefore, it is necessary to determine the conversion relationship between the pixel coordinate transformation amount of the camera coordinate system and the coordinate transformation amount of the robot arm coordinate system, so as to perform coordinate transformation on the first pixel offset based on the conversion relationship to obtain the correction movement amount of the target workpiece, where the correction movement amount is based on the parameters of the robot arm coordinate system. When the robot arm is a multi-degree-of-freedom robot arm, the robot arm coordinate system is the robot arm base coordinate system; when the robot arm is an XYZ-axis translation mechanism, the robot arm coordinate system can be a coordinate system constructed by the XYZ axes of the robot arm.

[0146] In a specific implementation, the pixel equivalent coef of the imaging device can be obtained. The pixel equivalent coef can characterize the relationship between the distance of one pixel in the target image converted to the distance in the robot arm coordinate, that is, the pixel equivalent coef is the actual physical distance represented by one pixel in the target image. Therefore, in the embodiments of the present application, the correction movement amount of the target workpiece can be determined according to the first pixel offset and the pixel equivalent of the imaging device.

[0147] In one embodiment, the method may further include:

[0148] Calibrate the imaging device to obtain the pixel equivalent of the imaging device.

[0149] During the calibration process, the imaging device can be calibrated to obtain the pixel equivalent coef of the imaging device, and then the pixel equivalent coef is stored, so that the pixel equivalent coef of the imaging device can be directly obtained during the subsequent correction process.

[0150] It should be noted that the robot arm and the imaging device can form an imaging system. For the same imaging system, after the pixel equivalent coef is calibrated once, the pixel equivalent coef is stored and there is no need to calibrate it multiple times.

[0151] In one embodiment, calibrating the imaging device to obtain the pixel equivalent of the imaging device may include:

[0152] Control the robotic arm to carry the reference workpiece and move it according to the preset reference points in the same reference plane; wherein, the reference plane is perpendicular to the direction of the first axis of the robotic arm coordinate system, and the reference point is located within the reference plane; whenever the robotic arm moves to any preset reference point, control the imaging device to collect an image of the reference workpiece to obtain multiple calibration images; determine the third pixel coordinates of the feature points of the reference workpiece in each calibration image; calculate the difference between the second pixel coordinates and each third pixel coordinate respectively to obtain the second pixel offset corresponding to each calibration image; and calculate the difference between the robotic arm coordinates of the reference point and the robotic arm coordinates of the preset reference point corresponding to each calibration image respectively to obtain the corresponding actual offset; wherein, the second pixel offset is the pixel distance between the feature points of the reference workpiece in the reference image and in the calibration image; according to the second pixel offset corresponding to each calibration image and the corresponding actual offset, calculate the axis angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system; wherein, the second axis direction of the robotic arm coordinate system is perpendicular to the first axis direction; according to the axis angle, the second pixel offset corresponding to each calibration image and the corresponding actual offset, calculate the pixel equivalent of the imaging device.

[0153] During the calibration process, in addition to collecting the reference image of the reference workpiece at the reference point, multiple calibration images of the reference workpiece at different positions can also be collected.

[0154] Specifically, during the calibration process, a reference plane can be established with the reference point as the center and perpendicular to the direction of the first axis of the robotic arm coordinate system, such as Figure 8 shown, the reference plane can include multiple grid points, and the distance between each two adjacent grid points is the same. For example, the distance between two adjacent grid points can be set to 1 mm. In the embodiments of the present application, each grid point can be used as a preset reference point, and then the robotic arm can be controlled to carry the reference workpiece and move according to the preset reference points in the same reference plane. It can be understood that Figure 8 the situation shown is only an example, and the arrangement manner of the preset reference points on the reference plane can also be other forms, which is not limited here.

[0155] Since the robotic arm moves in both positive and negative directions in the reference plane, that is, with the reference point as the center, the robotic arm moves along the positive x-axis direction, negative x-axis direction, positive y-axis direction, and negative y-axis direction in the reference plane. For the convenience of the robotic arm movement, the position farthest from the reference point (point O) among the relative positions to be moved can be used as the starting point. For example, point M can be used as the starting point, and the robotic arm can be controlled to move along Figure 8 the arrow direction of.

[0156] Whenever the robotic arm moves to any preset reference point position, the imaging device (industrial camera) can be controlled to capture an image of the reference workpiece, so that multiple calibration images can be obtained. That is to say, the images of the reference workpiece captured at each preset reference point position can be used as calibration images. For example, the robotic arm can be controlled to move several times along the x-axis direction with a step size of 1 mm and several times along the y-axis direction with a step size of 1 mm. After each movement, it pauses, and then the industrial camera under the robotic arm is controlled to take a picture of the bottom of the reference workpiece to obtain the calibration image of the reference workpiece, as Figure 9 shown, Figure 9 Figure Figure 9 is the calibration image of the reference workpiece captured by the industrial camera after the robotic arm moves 1 mm along the y-axis direction.

[0157] As an example, in the reference plane of Figure 8 , assuming that the preset step size is 1 mm, that is, the distance between two adjacent preset reference points in the preset moving direction is set to 1 mm, the coordinates of point M are (-3 mm, -3 mm). The imaging device can be controlled to capture the bottom image of the reference workpiece at the position of point M to obtain the calibration image M; starting from point M, the robotic arm can be controlled to move once along the positive y-axis direction with a step size of 1 mm to reach point N (-3 mm, -2 mm), and then the imaging device can be controlled to capture the bottom image of the reference workpiece at the position of point N to obtain the calibration image N;... and so on. Therefore, multiple calibration images captured at different preset reference point positions can be obtained, as Figure 10 shown, Figure 10 Figure Figure 10 are multiple calibration images of the reference workpiece when the robotic arm moves to different preset reference point positions captured by the industrial camera.

[0158] It should be noted that since the preset reference point at the center (point O) belongs to the reference point position, in order to facilitate the distinction between the calibration image and the reference image, the image of the reference workpiece captured at the preset reference point position at the center (point O) can be used as the reference image, and the images of the reference workpiece captured at other preset reference point positions are all used as calibration images.

[0159] It should be noted that during the calibration process, the height of the robotic arm remains unchanged, and the reference workpiece cannot exceed the image edge. In other words, the reference plane and the reference point position are at the same height, and the reference plane is within the shooting range of the industrial camera.

[0160] After obtaining multiple calibration images, the third pixel coordinates of the feature points of the reference workpiece in each calibration image can be determined. This step is the same as the implementation method of steps S211 - S212 for determining the first pixel coordinates of the feature points of the target workpiece in the target image during the correction process. For details, please refer to steps S211 - S212 above and will not be elaborated here.

[0161] After obtaining the third pixel coordinates of the feature points of the reference workpiece in each calibration image, the differences between the second pixel coordinates and each third pixel coordinate can be calculated respectively to obtain a plurality of differences, and each difference can be used as the second pixel offset corresponding to each calibration image. Among them, the second pixel offset can characterize the pixel distance between the feature points of the reference workpiece in the reference image and in the calibration image.

[0162] Let the second pixel offset be Δx i and Δy i , where Δx i refers to the x coordinate of the second pixel offset, and Δy i refers to the y coordinate of the second pixel offset.

[0163] Since the robotic arm coordinates of the reference point position and the robotic arm coordinates of each preset reference point are known, the actual offset corresponding to each calibration image can be obtained by calculating the differences between the robotic arm coordinates of the reference point position and the robotic arm coordinates of the preset reference point corresponding to each calibration image respectively. Exemplarily, for calibration image M and the reference image, let the coordinates of point M be (x a1 , y a1 ), the coordinates of point O be (x a2 , y a2 ), and the actual offset corresponding to calibration image M be (Δx a , Δy a ). Assuming that the robotic arm moves from point M (x a1 , y a1 ) to point O (x a2 , y a2 ), then the actual offset corresponding to calibration image M is:

[0164] Δx a = x a2 - x a1 , Δy a = y a2 - y a1 Equation 1

[0165] The actual offset (Δx a , Δy a ) corresponding to each calibration image is calculated according to the above Equation 1.

[0166] From the inner product formula of plane vectors: A·B = |A||B|cosθ, it can be known that the cosine of the included angle θ is:

[0167]

[0168] In the two-dimensional case, let A = (x 1 , y 1 ), B = (x 2, y 2 ), then we can obtain:

[0169]

[0170] In the embodiments of the present application, except for the reference image, the difference between the pixel coordinates of the feature points in each of the other images (calibration image / target image) and the pixel coordinates of the feature points in the reference image constitutes a vector (vector A), that is, both the first pixel offset and the second pixel offset can be represented in the form of a vector. If it is to be converted into an actual physical distance, the pixel equivalent coef needs to be considered. Therefore, vector A = (Δx i ·coef, Δy i ·coef). The actual offset corresponding to each calibration image can be directly calculated according to the above formula 1, then vector B = (Δx a , Δy a ). Substituting vector A and vector B into the above formula 3, we can obtain the arccosine function:

[0171]

[0172] Substituting the second pixel offset (Δx i , Δy i ) corresponding to each calibration image and the corresponding actual offset (Δx a , Δy a ) into the above cosine function (formula 4), and then through the arccosine function, multiple axis angles θ' can be calculated. Among them, each calibration image can correspond to an axis angle θ'. The embodiments of the present application can calculate the average value of multiple axis angles θ' Taking this average value as the final axis angle θ. Among them, the axis angle θ can represent the axis angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system. In the robotic arm coordinate system, the second axis direction is perpendicular to the first axis direction.

[0173] After obtaining the axis angle θ, the second pixel offset (Δx i , Δy i ) corresponding to each calibration image, and the corresponding actual offset (Δx a , Δy a ), the embodiments of the present application can calculate the pixel equivalent coef of the imaging device according to the axis angle θ, the second pixel offset (Δx i , Δy i ) corresponding to each calibration image, and the corresponding actual offset (Δx a , Δy a ).

[0174] In one embodiment, the optical axis of the camera in the imaging device is parallel to the direction of the first axis of the robotic arm coordinate system; the direction of the first axis of the robotic arm coordinate system is the Z-axis direction of the robotic arm coordinate system, and the direction of the second axis of the robotic arm coordinate system is the X-axis direction or the Y-axis direction of the robotic arm coordinate system.

[0175] It can be seen therefrom that the optical axis of the camera in the imaging device is parallel to the Z-axis direction of the robotic arm coordinate system, and the coordinate axis angle θ can be the coordinate axis angle between the X-axis direction of the camera coordinate system of the imaging device and the X-axis direction of the robotic arm coordinate system, or the coordinate axis angle between the Y-axis direction of the camera coordinate system of the imaging device and the Y-axis direction of the robotic arm coordinate system.

[0176] In one embodiment, calculating the pixel equivalent of the imaging device according to the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset may include:

[0177] Substitute the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset into the following two-dimensional coordinate rotation transformation formula to calculate the pixel equivalent corresponding to each calibration image, and then average the pixel equivalents to finally obtain the pixel equivalent of the imaging device:

[0178]

[0179] Where x' = Δx i ·coef, y' = Δy i ·coef, x = Δx a , y = Δy a ;

[0180] Where Δx i represents the x coordinate of the second pixel offset, Δy i represents the y coordinate of the second pixel offset, coef represents the pixel equivalent of the imaging device, Δx a represents the x coordinate of the actual offset, and Δy i represents the y coordinate of the actual offset.

[0181] Specifically, substituting x' = Δx i ·coef, y' = Δy i ·coef, x = Δx a and y = Δy a into the above two-dimensional coordinate rotation transformation formula (Equation 5), we can obtain:

[0182]

[0183] Through the above formula 6, multiple pixel equivalents coef' can be calculated. Among them, each calibration image can correspond to two pixel equivalents coef', and the embodiments of the present application can calculate the average value of multiple pixel equivalents coef'. Take this average value and determine it as the final pixel equivalent coef.

[0184] In one embodiment, determining the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device may include:

[0185] Substitute the first pixel offset, the pixel equivalent of the imaging device, and the coordinate axis included angle into the following formula to calculate the correction movement amount of the target workpiece:

[0186]

[0187] where, Δx i represents the x coordinate of the first pixel offset, Δy i represents the y coordinate of the first pixel offset, coef represents the pixel equivalent of the imaging device, Δx a represents the x coordinate of the correction movement amount, and Δy i represents the y coordinate of the correction movement amount.

[0188] The correction movement amount (Δx a , Δy a ) of the target workpiece can be calculated from the above formula 7.

[0189] It should be noted that the parameter symbols corresponding to the first pixel offset and the second pixel offset are the same, both being Δx i and Δy i . Specifically, during the calibration process, Δx i represents the x coordinate of the second pixel offset, Δy i represents the y coordinate of the second pixel offset, Δx a represents the x coordinate of the actual offset, and Δy i represents the y coordinate of the actual offset; during the correction process, Δx i represents the x coordinate of the first pixel offset, Δy i represents the y coordinate of the first pixel offset, Δx a represents the x coordinate of the correction movement amount, and Δy i represents the y coordinate of the correction movement amount.

[0190] Among them, the second pixel offset is used to characterize the pixel distance of the feature points of the reference workpiece in the reference image and in the calibration image during the calibration process, and its function is to determine the pixel coordinate transformation amount (Δx i , Δyi )The coordinate transformation amount to the robot arm coordinate system (Δx a , Δy a ), that is, to determine the included angle θ of the coordinate axes and the pixel equivalent coef of the imaging device. The first pixel offset is used to characterize the deviation between the target workpiece and the reference workpiece in the camera coordinate system during the calibration process. Its function is based on the conversion relationship obtained by calibration (the included angle θ of the coordinate axes and the pixel equivalent coef of the imaging device), and perform coordinate transformation on the first pixel offset (Δx i , Δy i ) to obtain the coordinate transformation amount to the robot arm coordinate system (Δx a , Δy a ). This coordinate transformation amount to the robot arm coordinate system (Δx a , Δy a ) is the calibration movement amount of the target workpiece.

[0191] S216. Use the calibration movement amount to calibrate the working point position of the target workpiece to obtain the calibrated working point position.

[0192] During the calibration process, the position where the robot arm performs the target operation (such as a filtering operation / pipetting operation, etc.) can also be determined, that is, to determine the working point position. Specifically, during the calibration process, the robot arm can be controlled to clamp the reference workpiece with the electric gripper, and then the robot arm is controlled to move to the position directly above the target container (such as a test tube), and the robot arm coordinates at this time are recorded, and this coordinate is used as the working point position.

[0193] Through the first pixel offset (Δx i , Δy i ), the pixel equivalent coef of the imaging device, and the included angle θ of the coordinate axes, after calculating the calibration movement amount of the target workpiece (Δx a , Δy a ), since the calibration movement amount (Δx a , Δy a ) characterizes the deviation between the target workpiece and the reference workpiece in the robot arm coordinate system, the calibration movement amount can be used to calibrate the working point position of the target workpiece.

[0194] Specifically, the working point position coordinates can be the coordinates in the robot arm base coordinate system, and its expression form is (X a , Y a , Z a , RX a , RY a , RZ a ), that is, including the position and attitude. The (X a , Y a ) in the working point position coordinates and the calibration movement amount (Δx a , Δya ) and the other coordinates in the working point position coordinates remain unchanged. Therefore, the working point position coordinates obtained after calibration are (X a +Δx a , Y a +Δy a , Z a , RX a , RY a , RZ a ). After the robotic arm moves to the calibrated working point position coordinates (X a +Δx a , Y a +Δy a , Z a , RX a , RY a , RZ a ), the bottom of the target workpiece (syringe or filter head) can be exactly above the target container. Thus, when the workpiece deviates from its position due to processing errors, operation errors, etc., the position of the workpiece can be corrected, improving the experimental accuracy and experimental safety.

[0195] In summary, in the related art, there is a lack of necessary means to check whether the movement is accurate after the robotic arm moves. Therefore, it is difficult to avoid the influence of processing errors of syringes, processing errors of filter heads, clamping errors of electric claws, and errors in fixing the filter head by the syringe. Therefore, in the embodiments of the present application, by adding a vision-based correction means after the robotic arm moves, the alignment problems between the syringe and the filter head, between the syringe and the container mouth, and between the filter head and the container mouth in automated target operations can be solved.

[0196] It can be understood that the robotic arm can pick up the target workpiece through an electric claw, or can also fix the target workpiece by plugging, magnetic attraction or other means, which is not limited here. The embodiments of the present application can be applied to calibrate the working point positions of syringes or filter heads in automated filtration operations to align the syringes or filter heads with the container mouth; can also be applied to calibrate the working point positions of pipette tips in automated pipetting processes to align the tips with the container mouth; and can also be applied to other experimental operations, which is not limited in the present application.

[0197] As can be seen from this example, in the solution provided by this application, when controlling the target workpiece to move to the reference point position, the imaging device is used to collect the target image corresponding to the target workpiece; the target image is matched with at least one template image to obtain the target area image of the target workpiece; wherein, the target area image contains the target structure of the target workpiece; the target area image is processed to obtain the pixel coordinates of the target structure, and the pixel coordinates of the target structure are determined as the first pixel coordinates of the feature points in the target image; the second pixel coordinates corresponding to the feature points in the reference image are obtained; wherein, the reference image is obtained by collecting an image of the reference workpiece at the reference point position by the imaging device when the reference workpiece is in the standard posture; the difference between the first pixel coordinates and the second pixel coordinates is calculated to obtain the first pixel offset of the target workpiece; according to the first pixel offset and the pixel equivalent of the imaging device, the correction movement amount of the target workpiece is determined; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image; the working point position of the target workpiece is corrected by using the correction movement amount to obtain the corrected working point position. In this application, the target image of the target workpiece and the reference image of the reference workpiece at the same reference point position are collected by the imaging device. Through the first pixel coordinates of the feature points in the target image and the second pixel coordinates of the feature points in the reference image, the first pixel offset of the target workpiece can be determined, and the first pixel offset can characterize the deviation between the target workpiece and the reference workpiece in the camera coordinate system. Further, in this application, through the pixel equivalent of the imaging device, the conversion relationship from the coordinate transformation amount in the camera coordinate system to the coordinate transformation amount in the robotic arm coordinate system can be determined. Therefore, based on the conversion relationship, the first pixel offset can be converted into the correction movement amount to correct the working point position of the target workpiece, and the position of the workpiece can be corrected when the position of the workpiece deviates due to processing errors, operation errors, etc., improving the experimental accuracy and experimental safety.

[0198] Furthermore, in this application, multiple template matches can improve the positioning accuracy and matching efficiency.

[0199] Furthermore, in this application, the calibration of the imaging system is simplified to only require the axis angle and pixel equivalent to describe the conversion relationship from the coordinate transformation amount in the camera coordinate system to the coordinate transformation amount in the robotic arm coordinate system, thereby improving the correction accuracy and efficiency.

[0200] To enable those skilled in the art to better understand the embodiments of this application, the following two examples are used to illustrate the embodiments of this application.

[0201] Figure 11 is the calibration flow chart of the robotic arm for gripping the reference workpiece. The calibration process is as follows:

[0202] S1101, after controlling the robotic arm holding the reference workpiece to move to the reference point position, control the imaging device to collect an image of the reference workpiece to obtain a reference image; wherein, the imaging device can be arranged below the robotic arm, and the reference point position can be the position directly above the imaging position.

[0203] S1102, control the robotic arm holding the reference workpiece to move according to the preset reference points in the same reference plane; wherein, the reference plane is perpendicular to the direction of the first axis of the robotic arm coordinate system, and the reference point position is located within the reference plane.

[0204] S1103, whenever the robotic arm moves to any preset reference point, control the imaging device to collect an image of the reference workpiece to obtain multiple calibration images.

[0205] S1104, perform Gaussian filtering on the reference image and each calibration image respectively.

[0206] S1105, perform color space transformation on the reference image and each calibration image respectively. For example, the reference image can be converted from a color image to a grayscale image, and each calibration image can be converted from a color image to a grayscale image respectively.

[0207] S1106, match the reference image after color space transformation with the first template image T1 to determine the initial region of the reference workpiece in the reference image, and intercept the initial region image of the reference workpiece from the reference image according to the initial region; and, match each calibration image after color space transformation with the first template image T1 to determine the initial region of the reference workpiece in each calibration image, and intercept the initial region image of the reference workpiece from each calibration image respectively according to the initial region.

[0208] S1107, match each initial region image of the reference workpiece with the second template image T2 respectively to determine the target region of the reference workpiece in each initial region image, and intercept the target region image of the reference workpiece from each initial region image respectively according to the target region.

[0209] S1108, perform adaptive gamma transformation on each target region image of the reference workpiece respectively.

[0210] S1109, perform Hough transformation on each target region image of the reference workpiece after gamma transformation to obtain the second pixel coordinates of the feature points of the reference workpiece in the reference image and the third pixel coordinates in each calibration image.

[0211] S1110, calculate the differences between the second pixel coordinates and each third pixel coordinate respectively to obtain the second pixel offsets corresponding to each calibration image; and calculate the differences between the robotic arm coordinates of the reference point and the robotic arm coordinates of the preset reference point corresponding to each calibration image respectively to obtain the corresponding actual offsets; wherein, the second pixel offset is the pixel distance of the feature point of the reference workpiece in the reference image and in the calibration image.

[0212] S1111, according to the second pixel offsets and the corresponding actual offsets corresponding to each calibration image, calculate the axis angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system; wherein, the second axis direction of the robotic arm coordinate system is perpendicular to the first axis direction.

[0213] S1112, according to the axis angle, the second pixel offsets and the corresponding actual offsets corresponding to each calibration image, calculate the pixel equivalent of the imaging device; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image.

[0214] Figure 12 It is the calibration flowchart of the robotic arm for gripping the target workpiece. The calibration process is as follows:

[0215] S1201, after controlling the robotic arm gripping the target workpiece to move to the reference point, control the imaging device to collect the image of the target workpiece to obtain the target image; wherein, the imaging device can be set below the robotic arm, and the reference point can be the position directly above the imaging position.

[0216] S1202, perform Gaussian filtering on the target image.

[0217] S1203, perform color space transformation on the target image. For example, the target image can be converted from a color image to a grayscale image.

[0218] S1204, match the target image after color space transformation with the first template image T1 to determine the initial region of the target workpiece in the target image, and intercept the initial region image of the target workpiece from the target image according to the initial region.

[0219] S1205, match the initial region image of the target workpiece with the second template image T2 to determine the target region of the target workpiece in the initial region image, and intercept the target region image of the target workpiece from the initial region image according to the target region.

[0220] S1206, perform adaptive gamma transformation on the target region image of the target workpiece.

[0221] S1207. Perform a Hough transform on the target area image of the target workpiece after gamma transformation to obtain the first pixel coordinates of the feature points of the target workpiece in the target image.

[0222] S1208. Obtain the second pixel coordinates of the feature points of the reference workpiece in the reference image.

[0223] S1209. Calculate the difference between the first pixel coordinates and the second pixel coordinates to obtain the first pixel offset of the target workpiece; wherein, the first pixel offset can characterize the deviation between the target workpiece and the reference workpiece in the camera coordinate system.

[0224] S1210. Obtain the pixel equivalent and the axis angle of the imaging device; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image, and the axis angle is the angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system.

[0225] S1211. Calculate the correction movement amount of the target workpiece according to the first pixel offset, the pixel equivalent of the imaging device, and the axis angle.

[0226] S1212. Use the correction movement amount to correct the working point position of the target workpiece to obtain the corrected working point position.

[0227] Corresponding to the foregoing application function implementation method embodiments, the present application also provides a vision-based working point position correction device, an electronic device, a computer-readable storage medium, and corresponding embodiments.

[0228] Figure 13 is a schematic structural diagram of the vision-based working point position correction device shown in the embodiments of the present application.

[0229] See Figure 13 , a vision-based working point position correction device provided by the present application, the device may include:

[0230] A target image acquisition module 1301, configured to use the imaging device to acquire a target image corresponding to the target workpiece when controlling the target workpiece to move to the reference point position.

[0231] A first pixel coordinate determination module 1302, configured to perform feature recognition on the target image and determine the first pixel coordinates corresponding to the feature points in the target image.

[0232] A second pixel coordinate acquisition module 1303, configured to acquire the second pixel coordinates corresponding to the feature points in the reference image; wherein, the reference image is an image acquired by the imaging device for the reference workpiece when the reference workpiece is at the reference point position in the standard posture.

[0233] A calibration movement amount determination module 1304, configured to determine a calibration movement amount of a target workpiece according to a first pixel coordinate and a second pixel coordinate;

[0234] A working point position calibration module 1305, configured to calibrate the working point position of the target workpiece by using the calibration movement amount to obtain a calibrated working point position.

[0235] In an embodiment, the first pixel coordinate determination module 1302 may include:

[0236] A template matching sub-module, configured to match the target image with at least one template image to obtain a target region image of the target workpiece; wherein, the target region image includes a target structure of the target workpiece;

[0237] A first pixel coordinate determination sub-module, configured to process the target region image to obtain pixel coordinates of the target structure, and determine the pixel coordinates of the target structure as the first pixel coordinates of the feature points in the target image.

[0238] In an embodiment, the at least one template image includes a first template image and a second template image; the template matching sub-module may include:

[0239] A first template matching unit, configured to match the target image with the first template image to determine an initial region of the target workpiece in the target image, and intercept an initial region image of the target workpiece from the target image according to the initial region; wherein, the structure of the target workpiece included in the initial region image is more than the target structure;

[0240] A second template matching unit, configured to match the initial region image with the second template image to determine a target region of the target structure of the target workpiece in the initial region image, and intercept a target region image of the target workpiece from the initial region image according to the target region.

[0241] In an embodiment, the first pixel coordinate determination sub-module may include:

[0242] A Hough transform unit, configured to perform a Hough transform on the target region image to determine a geometric center of the target structure;

[0243] A pixel coordinate determination unit of the target structure, configured to determine the pixel coordinates of the geometric center as the pixel coordinates of the target structure.

[0244] In an embodiment, the calibration movement amount determination module 1304 may include:

[0245] A first pixel offset amount determination sub-module, configured to calculate a difference between the first pixel coordinate and the second pixel coordinate to obtain a first pixel offset amount of the target workpiece;

[0246] The calibration movement amount determination module sub-module is used to determine the calibration movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image.

[0247] In one embodiment, the device may further include:

[0248] The calibration processing module is used to perform calibration processing on the imaging device to obtain the pixel equivalent of the imaging device.

[0249] In one embodiment, the calibration processing module may include:

[0250] The movement sub-module is used to control the robotic arm to carry the reference workpiece to move according to the preset reference points in the same reference plane; wherein, the reference plane is perpendicular to the direction of the first axis of the robotic arm coordinate system, and the reference point is located within the reference plane;

[0251] The calibration image acquisition sub-module is used to control the imaging device to acquire images of the reference workpiece whenever the robotic arm moves to any preset reference point, so as to obtain multiple calibration images;

[0252] The third pixel coordinate determination sub-module is used to determine the third pixel coordinates of the feature points of the reference workpiece in each calibration image;

[0253] The calculation sub-module of the second pixel offset and the actual offset is used to calculate the difference between the second pixel coordinate and each third pixel coordinate respectively to obtain the second pixel offset corresponding to each calibration image; and calculate the difference between the robotic arm coordinates of the reference point and the robotic arm coordinates of the preset reference point corresponding to each calibration image respectively to obtain the corresponding actual offset; wherein, the second pixel offset is the pixel distance between the feature points of the reference workpiece in the reference image and in the calibration image;

[0254] The coordinate axis angle calculation sub-module is used to calculate the coordinate axis angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system according to the second pixel offset and the corresponding actual offset corresponding to each calibration image; wherein, the second axis direction of the robotic arm coordinate system is perpendicular to the first axis direction;

[0255] The pixel equivalent calculation sub-module is used to calculate the pixel equivalent of the imaging device according to the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset.

[0256] In one embodiment, the pixel equivalent calculation sub-module may include:

[0257] A pixel equivalent calculation unit, which is configured to substitute the included angle between coordinate axes, the second pixel offset corresponding to each calibration image, and the corresponding actual offset into the following two-dimensional coordinate rotation transformation formula to calculate the pixel equivalent corresponding to each calibration image, and then average the pixel equivalents to finally obtain the pixel equivalent of the imaging device:

[0258]

[0259] where x' = Δx i ·coef, y' = Δy i ·coef, x = Δx a , y = Δy a ;

[0260] where, Δx i represents the x coordinate of the second pixel offset, Δy i represents the y coordinate of the second pixel offset, coef represents the pixel equivalent of the imaging device, Δx a represents the x coordinate of the actual offset, Δy i represents the y coordinate of the actual offset.

[0261] In one embodiment, the sub-module of the correction movement amount determination module may include:

[0262] A correction movement amount determination module unit, which is configured to substitute the first pixel offset, the pixel equivalent of the imaging device, and the included angle between coordinate axes into the following formula to calculate the correction movement amount of the target workpiece:

[0263]

[0264] where, Δx i represents the x coordinate of the first pixel offset, Δy i represents the y coordinate of the first pixel offset, coef represents the pixel equivalent of the imaging device, Δx a represents the x coordinate of the correction movement amount, Δy i represents the y coordinate of the correction movement amount.

[0265] In one embodiment, the optical axis of the camera in the imaging device is parallel to the direction of the first axis of the robotic arm coordinate system; the direction of the first axis of the robotic arm coordinate system is the Z-axis direction of the robotic arm coordinate system, and the direction of the second axis of the robotic arm coordinate system is the X-axis direction or the Y-axis direction of the robotic arm coordinate system.

[0266] As can be seen from this example, in the solution provided by this application, when controlling the target workpiece to move to the reference point position, the imaging device is used to collect the target image corresponding to the target workpiece; feature recognition is performed on the target image, and the first pixel coordinates corresponding to the feature points in the target image are determined; the second pixel coordinates corresponding to the feature points in the reference image are obtained; wherein, the reference image is obtained by collecting an image of the reference workpiece at the reference point position by the imaging device when the reference workpiece is in the standard posture; according to the first pixel coordinates and the second pixel coordinates, the correction movement amount of the target workpiece is determined; the working point position of the target workpiece is corrected by using the correction movement amount to obtain the corrected working point position. In this application, the imaging device is used to collect the target image of the target workpiece and the reference image of the reference workpiece at the same reference point position. The correction movement amount of the target workpiece is determined by the first pixel coordinates of the feature points in the target image and the second pixel coordinates of the feature points in the reference image, and the working point position of the target workpiece is corrected accordingly. When the position of the workpiece deviates due to processing errors, operation errors, etc., the position of the workpiece can be corrected, improving the experimental accuracy and experimental safety.

[0267] Regarding the vision-based working point position correction device in the above embodiment, the specific manners in which each module performs operations have been described in detail in the embodiment related to this method, and will not be elaborated here.

[0268] Figure 14 It is a schematic structural diagram of an electronic device shown in an embodiment of this application.

[0269] See Figure 14 , the electronic device 1400 includes a memory 1410 and a processor 1420.

[0270] The processor 1420 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0271] The memory 1410 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM can store static data or instructions required by the processor 1420 or other modules of the computer. The permanent storage device can be a readable and writable storage device. The permanent storage device can be a non-volatile storage device that does not lose the stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during operation. In addition, the memory 1410 can include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks can also be used. In some embodiments, the memory 1410 can include a removable storage device that is readable and / or writable, such as a compact disc (CD), read-only digital versatile disc (such as DVD-ROM, dual-layer DVD-ROM), read-only Blu-ray disc, super density disc, flash memory card (such as SD card, min SD card, Micro-SD card, etc.), magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and instantaneous electronic signals transmitted wirelessly or wired.

[0272] Executable code is stored on the memory 1410, and when the executable code is processed by the processor 1420, it can cause the processor 1420 to execute some or all of the methods described above.

[0273] In addition, the method according to the present application can also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the above steps of the method according to the present application.

[0274] Alternatively, the present application can also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium), on which executable code (or a computer program or computer instruction code) is stored. When the executable code (or the computer program or computer instruction code) is executed by a processor of an electronic device (or a server, etc.), it causes the processor to execute some or all of the steps of the above method according to the present application.

[0275] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A vision-based working point correction method, characterized in that, the method includes: When controlling the target workpiece to move to the reference point, using an imaging device to collect the target image corresponding to the target workpiece; Performing feature recognition on the target image and determining the first pixel coordinates corresponding to the feature points in the target image; Obtaining the second pixel coordinates corresponding to the feature points in the reference image; wherein, the reference image is obtained by collecting an image of the reference workpiece at the reference point when the reference workpiece is in a standard posture through the imaging device; Determining the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates; Using the correction movement amount to correct the working point of the target workpiece to obtain the corrected working point.

2. The method according to claim 1, characterized in that, the performing feature recognition on the target image and determining the first pixel coordinates corresponding to the feature points in the target image includes: Matching the target image with at least one template image to obtain the target region image of the target workpiece; wherein, the target region image contains the target structure of the target workpiece; Processing the target region image to obtain the pixel coordinates of the target structure, and determining the pixel coordinates of the target structure as the first pixel coordinates of the feature points in the target image.

3. The method according to claim 2, characterized in that, the at least one template image includes a first template image and a second template image; the matching the target image with at least one template image to obtain the target region image of the target workpiece includes: Matching the target image with the first template image to determine the initial region of the target workpiece in the target image, and intercepting the initial region image of the target workpiece from the target image according to the initial region; wherein, the structure of the target workpiece contained in the initial region image is more than the target structure; Matching the initial region image with the second template image to determine the target region of the target structure of the target workpiece in the initial region image, and intercepting the target region image of the target workpiece from the initial region image according to the target region.

4. The method according to claim 2, characterized in that, the processing the target region image to obtain the pixel coordinates of the target structure includes: Performing a Hough transform on the target region image to determine the geometric center of the target structure; Determining the pixel coordinates of the geometric center as the pixel coordinates of the target structure.

5. The method according to any one of claims 1-4, characterized in that, the determining the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates includes: Calculating the difference between the first pixel coordinates and the second pixel coordinates to obtain the first pixel offset of the target workpiece; Determine the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device; wherein, the pixel equivalent is the actual physical distance represented by one pixel in the target image.

6. The method according to claim 5, wherein, the method further includes: Performing a calibration process on the imaging device to obtain the pixel equivalent of the imaging device.

7. The method according to claim 6, wherein, The performing a calibration process on the imaging device to obtain the pixel equivalent of the imaging device includes: Controlling the robotic arm to carry the reference workpiece to move according to preset reference points in the same reference plane; wherein, the reference plane is perpendicular to the direction of the first axis of the robotic arm coordinate system, and the reference point is located within the reference plane; Whenever the robotic arm moves to any preset reference point, controlling the imaging device to collect an image of the reference workpiece to obtain multiple calibration images; Determine the third pixel coordinates of the feature points of the reference workpiece in each calibration image; Respectively calculate the difference between the second pixel coordinates and each third pixel coordinate to obtain the second pixel offset corresponding to each calibration image; and respectively calculate the difference between the robotic arm coordinates of the reference point and the robotic arm coordinates of the preset reference point corresponding to each calibration image to obtain the corresponding actual offset; wherein, the second pixel offset is the pixel distance between the feature points of the reference workpiece in the reference image and in the calibration image; According to the second pixel offset corresponding to each calibration image and the corresponding actual offset, calculate the coordinate axis angle between the second axis direction of the camera coordinate system of the imaging device and the second axis direction of the robotic arm coordinate system; wherein, the second axis direction of the robotic arm coordinate system is perpendicular to the first axis direction; Calculate the pixel equivalent of the imaging device according to the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset.

8. The method according to claim 7, wherein, The calculating the pixel equivalent of the imaging device according to the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset includes: Substitute the coordinate axis angle, the second pixel offset corresponding to each calibration image, and the corresponding actual offset into the following two-dimensional coordinate rotation transformation formula to calculate the pixel equivalent corresponding to each calibration image, and then average the pixel equivalents to finally obtain the pixel equivalent of the imaging device: where x' = Δx i ·coef, y' = Δy i ·coef, x = Δx a , y = Δy a ; wherein, the Δx i represents the x coordinate of the second pixel offset, the Δy i represents the y coordinate of the second pixel offset, the coef represents the pixel equivalent of the imaging device, and the Δx a represents the x coordinate of the actual offset, and the Δy i represents the y coordinate of the actual offset.

9. The method according to claim 7, wherein, The determining the correction movement amount of the target workpiece according to the first pixel offset and the pixel equivalent of the imaging device includes: Substitute the first pixel offset, the pixel equivalent of the imaging device, and the coordinate axis angle into the following formula to calculate the correction movement amount of the target workpiece: Among them, the Δx i represents the x coordinate of the first pixel offset, and the Δy i represents the y coordinate of the first pixel offset. The coef represents the pixel equivalent of the imaging device, and the Δx a represents the x coordinate of the correction movement amount, and the Δy i represents the y coordinate of the correction movement amount.

10. The method according to claim 7, wherein, The optical axis of the camera in the imaging device is parallel to the direction of the first axis of the robotic arm coordinate system; the direction of the first axis of the robotic arm coordinate system is the Z-axis direction of the robotic arm coordinate system, and the direction of the second axis of the robotic arm coordinate system is the X-axis direction or the Y-axis direction of the robotic arm coordinate system.

11. A vision-based working point correction device Characterized in that It includes: A target image acquisition module, configured to use an imaging device to acquire a target image corresponding to the target workpiece when controlling the target workpiece to move to a reference point; A first pixel coordinate determination module, configured to perform feature recognition on the target image and determine the first pixel coordinates corresponding to the feature points in the target image; A second pixel coordinate acquisition module, configured to acquire the second pixel coordinates corresponding to the feature points in the reference image; wherein, the reference image is obtained by using the imaging device to perform image acquisition on the reference workpiece when the reference workpiece is at the reference point in a standard posture; A correction movement amount determination module, configured to determine the correction movement amount of the target workpiece according to the first pixel coordinates and the second pixel coordinates; A working point correction module, configured to correct the working point of the target workpiece by using the correction movement amount to obtain a corrected working point.

12. An electronic device Characterized in that It includes: A processor; And A memory, on which executable code is stored, and when the executable code is executed by the processor, the processor is caused to execute the method according to any one of claims 1-10.

13. A computer-readable storage medium, on which executable code is stored, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1-10.

Images