Machine vision labeling system and control method thereof, labeling device and storage medium

Abstract

The application discloses a machine vision labeling system and a control method thereof, a labeling device and a storage medium. The system comprises an execution mechanism, a shooting mechanism, a conveying mechanism and a labeling mechanism. The shooting mechanism and the labeling mechanism are detachably installed on the execution mechanism. The shooting mechanism is configured to collect image information. The conveying mechanism is configured to convey an object to be labeled. The method comprises the following steps: acquiring image information of the object to be labeled during movement of the execution mechanism following the object to be labeled; acquiring pose information of a labeling area of the object to be labeled according to the image information; controlling the execution mechanism to perform dynamic tracking according to pose information of the execution mechanism, pose information of the conveying mechanism and the pose information of the labeling area; and controlling the labeling mechanism to label the labeling area when the labeling mechanism is synchronized with the labeling area. In this way, dynamic tracking and labeling can be completed without stopping the conveying mechanism, and the efficiency and automation level of visual guidance labeling are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual positioning, and particularly relates to a control method of a machine vision labeling system, a machine vision labeling system, a computer readable storage medium and a labeling device. BACKGROUND

[0002] In the related art, the visual guidance labeling scene is a step-by-step scene, and the conveying belt needs to be frequently stopped, which is only applicable to static scenes, and this will cause low labeling efficiency. SUMMARY

[0003] The present application aims to at least solve one of the technical problems in the related art. To this end, a first object of the present application is to provide a control method of a machine vision labeling system, which controls an execution mechanism to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of a conveying mechanism and the pose information of a region to be labeled, and controls a labeling mechanism to label the region to be labeled when the labeling mechanism is synchronized with the region to be labeled, so that labeling can be completed without stopping the conveying mechanism, the efficiency of visual guidance labeling is optimized, the automation level is improved, and significant economic value is brought.

[0004] A second object of the present application is to provide a machine vision labeling system.

[0005] A third object of the present application is to provide a computer readable storage medium.

[0006] A fourth object of the present application is to provide a labeling device.

[0007] To achieve the above objects, a control method of a machine vision labeling system is provided in a first aspect of the present application, the system comprising an execution mechanism, a shooting mechanism, a conveying mechanism and a labeling mechanism, the shooting mechanism and the labeling mechanism being detachably installed on the execution mechanism, the shooting mechanism being configured to collect image information, the conveying mechanism being configured to convey an object to be labeled, the method comprising: acquiring image information of the object to be labeled in a process in which the execution mechanism follows movement of the object to be labeled; acquiring pose information of a region to be labeled of the object to be labeled according to the image information; controlling the execution mechanism to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism and the pose information of the region to be labeled, and controlling the labeling mechanism to label the region to be labeled when the labeling mechanism is synchronized with the region to be labeled.

[0008] In some embodiments, the execution mechanism is controlled to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area, including: determining pose error information of the labeling area and the labeling mechanism according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area; generating a motion control instruction according to the pose error information, the pose information of the execution mechanism, and the pose information of the conveying mechanism; and controlling the execution mechanism to perform dynamic tracking according to the motion control instruction.

[0009] In some embodiments, the execution mechanism is controlled to perform dynamic tracking according to the motion control instruction, including: performing dynamic tracking by adjusting three degrees of freedom of the execution mechanism in the horizontal direction X, Y, and Rz.

[0010] In some embodiments, the pose error information of the labeling area and the labeling mechanism is determined by the following formula:

[0011] errDist k =L0+Δy t +(l 2k -l 2t )-(l 1k -l 1t )

[0012] errDist k represents the pose error information of the labeling area and the labeling mechanism along the conveying direction at time k, L0represents the installation position deviation of the center of the shooting mechanism and the center of the labeling mechanism along the conveying direction, Δy t represents the pose error information of the center of the shooting mechanism and the center of the labeling area along the conveying direction at time t, l 2k represents the pose information of the execution mechanism at time k, l 2t represents the pose information of the execution mechanism at time t, l 1k represents the pose information of the conveying mechanism at time k, l 1t represents the pose information of the conveying mechanism at time t.

[0013] In some embodiments, in the case of external interference, the pose error information of the labeling area and the labeling mechanism is determined by the following formula:

[0014] errDist (k+1) =L0+Δy′ t+1 +(l 2(k+1) -l 2(t+1) )-(l 1(k+1) -l 1(t+1 ))

[0015] errDist (k+1)represents the pose error information of the labeling region and the labeling mechanism along the conveying direction at time k+1, Δy' t+1 represents the pose error information of the center of the imaging mechanism and the center of the labeling region along the conveying direction at time t+1, l 2(k+1) represents the pose information of the execution mechanism at time k+1, l 2(t+1) represents the pose information of the execution mechanism at time t+1, l 1(k+1) represents the pose information of the conveying mechanism at time k+1, l 1(t+1) represents the pose information of the conveying mechanism at time t+1.

[0016] In some embodiments, the pose information of the labeling region of the object to be labeled is obtained according to the image information, including: performing feature extraction on the image information to obtain a region of interest of the object to be labeled; performing feature extraction on the region of interest to obtain coordinates of feature points in the region of interest; determining a plane equation of the region of interest based on the coordinates of the feature points; determining a target feature point according to the preset geometric constraint condition and the plane equation; and determining the pose of the labeling region in the image information according to the coordinates of the target feature point.

[0017] In some embodiments, the image information is matched and tracked based on a template matching method in the process of dynamic tracking.

[0018] In some embodiments, the target feature point is determined according to the preset geometric constraint condition and the plane equation, including: in the case that the preset geometric constraint condition is a distance constraint, obtaining the distance from the feature points to the plane equation, and taking the feature points with a distance less than a preset distance as the target feature points; and in the case that the preset geometric constraint condition is a proportion constraint, obtaining the distance from the feature points to the plane equation, and arranging all distances in ascending order, and taking the first preset proportion of the feature points as the target feature points.

[0019] In some embodiments, the pose information of the labeling region is obtained at a first preset servo frequency, and the pose information of the execution mechanism and the pose information of the conveying mechanism are obtained at a second preset servo frequency; wherein the first preset servo frequency is less than the second preset servo frequency.

[0020] In some embodiments, the labeling mechanism and the labeling region are determined to be synchronized in the case that the error between the pose information of the labeling mechanism and the pose information of the labeling region is less than a preset error threshold, and the movement speed of the execution mechanism is the same as the movement speed of the object to be labeled.

[0021] To achieve the above object, the second aspect of the present application provides a machine vision labeling system, comprising an execution mechanism, a shooting mechanism, a conveying mechanism, a labeling mechanism and a controller, the shooting mechanism and the labeling mechanism are detachably installed on the execution mechanism, the conveying mechanism is configured to convey an object to be labeled, wherein the shooting mechanism is configured to acquire image information of the object to be labeled during the execution mechanism following the movement of the object to be labeled; the controller is configured to acquire pose information of a labeling area of the object to be labeled according to the image information, and control the execution mechanism to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism and the pose information of the labeling area, and control the labeling mechanism to label the labeling area when the labeling mechanism is synchronized with the labeling area.

[0022] To achieve the above object, the third aspect of the present application provides a computer readable storage medium, which stores a control program of a machine vision labeling system, and the control program of the machine vision labeling system is executed by a processor to implement the control method of the machine vision labeling system.

[0023] To achieve the above object, the fourth aspect of the present application provides a labeling device, comprising the machine vision labeling system.

[0024] According to the control method of the machine vision labeling system, the image information of the object to be labeled is acquired during the execution mechanism following the movement of the object to be labeled; the pose information of the labeling area of the object to be labeled is acquired according to the image information; the execution mechanism is controlled to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism and the pose information of the labeling area, and the labeling mechanism is controlled to label the labeling area when the labeling mechanism is synchronized with the labeling area. In this way, the object to be labeled is dynamically tracked based on machine vision, and labeling can be completed without stopping the conveying mechanism, thereby optimizing the efficiency of vision-guided labeling, improving the automation level and bringing significant economic value. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the application and its specification, and do not constitute improper limitations to the present application. In the drawings:

[0026] Figure 1 is a structural schematic diagram of a machine vision labeling system according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of a control method of a machine vision labeling system according to an embodiment of the present application;

[0028] Figure 3is a block diagram of a labeling device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to an embodiment of the present application, a control method of a machine vision labeling system is provided.

[0033] Figure 1 is a structural schematic diagram of a machine vision labeling system according to an embodiment of the present application, with reference to Figure 1 The machine vision labeling system includes an execution mechanism 11, a shooting mechanism 12, a conveying mechanism 13 and a labeling mechanism 14. The shooting mechanism 12 and the labeling mechanism 14 are detachably mounted on the execution mechanism 11. The shooting mechanism 12 is configured to collect image information, and the conveying mechanism 13 is configured to convey objects to be labeled. The execution mechanism 11 can include an execution mechanism body and an execution mechanism end effector.

[0034] In some embodiments, with continued reference to Figure 1, the conveying mechanism 13 comprises a conveying belt, and the machine vision labeling system can further comprise at least one photoelectric sensor 15 arranged on the conveying belt, and when the number of photoelectric sensors 15 is two, the photoelectric sensors 15 can be symmetrically arranged on two sides of the conveying belt, and used for detecting whether the object to be labeled exists on the conveying mechanism 13. The machine vision labeling system can further comprise a visual motion controller 16 and a servo driver 17.

[0035] Specifically, the conveying mechanism 13 is configured to convey the object to be labeled, for example, a conveying belt, which can move smoothly in a horizontal direction after being started. The photoelectric sensor 15 is arranged on the conveying mechanism 13, and the number of photoelectric sensors 15 can be one or more. The execution mechanism 11, for example, a gantry device or a SCARA (Selective Compliance Assembly Robot Arm, a robot arm applied to assembly work) robot, is used to dynamically track the object to be labeled, and the tracking direction is the same as the movement direction of the conveying mechanism 13. The shooting mechanism 12, for example, an industrial camera, is used to obtain image information of the object to be labeled during the movement of the execution mechanism 11 following the object to be labeled. The labeling mechanism 14 can move up and down in a vertical direction and is controlled by the execution mechanism 11, and is used to perform a labeling operation when the labeling mechanism 14 is synchronized with the labeling area. The shooting mechanism 12 and the labeling mechanism 14 are detachably mounted on the execution mechanism 11, wherein there is a height difference between the shooting mechanism 12 and the labeling mechanism 14, the labeling mechanism 14 is installed closer to the conveying mechanism 13, and the vertical distance between the shooting mechanism 12 and the object to be labeled remains unchanged.

[0036] The execution mechanism 11 is in a servo state at an enabling position, waits for the arrival of the object to be labeled, and when the photoelectric sensor 15 detects the object to be labeled, outputs a trigger signal to the visual motion controller 16. The visual motion controller 16 generates a periodic trigger signal to the shooting mechanism 12, triggers the shooting mechanism 12 to shoot an image, and the visual motion controller 16 obtains an image stream from the shooting mechanism 12 and processes the obtained image to obtain the 6D pose of the object to be labeled. The visual motion controller 16 obtains the pose information of the execution mechanism 11 and the conveying mechanism 13 from the servo driver 17 through an EtherCAT bus, plans a trajectory according to the 6D pose of the object to be labeled and the pose information of the execution mechanism 11 and the conveying mechanism 13, generates motion control information, and drives the end effector of the execution mechanism to drive the labeling mechanism to label the object to be labeled through the servo driver 17.

[0037] Figure 2 A flowchart of a control method of a machine vision labeling system according to an embodiment of the present application is shown. Referring to Figure 2 The control method of the machine vision labeling system can comprise:

[0038] S110, acquiring image information of the object to be labeled during the movement of the execution mechanism following the object to be labeled.

[0039] Specifically, the shooting mechanism can be an industrial camera, which is detachably mounted on the execution mechanism, follows the movement of the object to be labeled, and acquires image information of the object to be labeled in real time.

[0040] For example, the execution mechanism is in a servo state at the starting position, waits for the arrival of the object to be labeled, triggers the shooting mechanism to shoot and acquire image information of the object to be labeled after the photoelectric sensor detects the object to be labeled, and feeds back image information of the object to be labeled in real time during the movement of the execution mechanism following the object to be labeled, while counting the number of objects to be labeled.

[0041] S120, acquiring pose information of the labeling area of the object to be labeled according to the image information.

[0042] Specifically, the image information of the object to be labeled is image-processed, and the pose information of the labeling area of the object to be labeled, such as 6D pose information of the labeling area, is acquired according to the image processing result.

[0043] S130, controlling the execution mechanism to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area, and controlling the labeling mechanism to label the labeling area when the labeling mechanism is synchronized with the labeling area.

[0044] Specifically, corresponding encoders can be arranged on the conveying belt, and the pose information of the conveying belt can be obtained by reading the readings of the encoders. Similarly, corresponding encoders can be arranged on the end effector of the execution mechanism, and the pose information of the execution mechanism can be obtained by reading the readings of the encoders. The pose information of the execution mechanism (such as 6D pose information of the execution mechanism) and the pose information of the conveying mechanism (such as 6D pose information of the conveying mechanism) can be acquired in real time through an EtherCAT (Ethernet Control Automation Technology, real-time Ethernet technology) bus, and trajectory planning can be performed in combination with the pose information of the labeling area to control the execution mechanism to perform dynamic tracking according to the planned motion trajectory with the labeling mechanism until the labeling mechanism is synchronized with the labeling area, at which time the labeling mechanism is controlled to be pressed quickly to label the labeling area.

[0045] In this way, labeling can be completed without stopping the conveying mechanism, the efficiency of visual guidance labeling is optimized, the automation level is improved, and significant economic value is brought.

[0046] In some embodiments, the execution mechanism is controlled to perform dynamic tracking according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area, including: determining pose error information of the labeling area and the labeling mechanism according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area; generating a motion control instruction according to the pose error information, the pose information of the execution mechanism, and the pose information of the conveying mechanism; and controlling the execution mechanism to perform dynamic tracking according to the motion control instruction.

[0047] In some embodiments, the execution mechanism is controlled to perform dynamic tracking according to the motion control instruction, including: performing dynamic tracking by adjusting three degrees of freedom of the execution mechanism in the horizontal direction X, Y, and Rz.

[0048] Specifically, when the pose error of the labeling area and the labeling mechanism is close to 0, it means that the center of the labeling mechanism is basically aligned with the center of the labeling area of the object to be labeled. Therefore, the pose error information of the labeling area and the labeling mechanism is determined according to the pose information of the execution mechanism, the pose information of the conveying mechanism, and the pose information of the labeling area. Based on the pose error information, and in combination with the pose information of the execution mechanism and the pose information of the conveying mechanism, a motion control instruction is generated. Since the labeling is planar labeling, the execution mechanism is controlled to adjust in three degrees of freedom in the horizontal direction X, Y, and Rz until the pose error of the labeling area and the labeling mechanism is close to 0, so as to realize tracking of the labeling area of the object to be labeled.

[0049] For example, when the pose error information of the labeling area and the labeling mechanism indicates a certain difference in the horizontal direction, the moving speed of the labeling mechanism can be adjusted so that the center of the labeling mechanism and the center of the labeling area are on the same perpendicular line; when the pose error information of the labeling area and the labeling mechanism indicates that the labeling mechanism forms a certain angle with the horizontal plane, the angle between the labeling mechanism and the horizontal plane can be adjusted to make the labeling mechanism and the labeling area parallel.

[0050] In some embodiments, the pose information of the labeling area is obtained at a first preset servo frequency, and the pose information of the execution mechanism and the pose information of the conveying mechanism are obtained at a second preset servo frequency; wherein the first preset servo frequency is less than the second preset servo frequency.

[0051] Exemplarily, the shooting mechanism shoots and acquires image information of the moving object to be labeled in the tracking process, processes the image information of the object to be labeled, and calculates the pose information of the labeling area at a first preset servo frequency, while acquiring the pose information of the execution mechanism and the pose information of the conveying mechanism in real time at a second preset servo frequency through the EtherCAT bus. The pose information of the labeling area calculated at the first preset servo frequency, the pose information of the execution mechanism, and the pose information of the conveying mechanism acquired at the second preset servo frequency are fused to obtain the pose error information of the labeling area and the labeling mechanism. At this time, based on the pose information of the labeling area, the pose information of the execution mechanism, and the pose information of the conveying mechanism, a motion control instruction is generated at the second preset servo frequency, and the execution mechanism adjusts in three degrees of freedom of the horizontal directions X, Y, and Rz based on the motion control instruction to dynamically track the labeling area of the object to be labeled until the pose error of the labeling area and the labeling mechanism approaches 0.

[0052] It should be noted that the first preset servo frequency is the frequency at which the shooting mechanism shoots images, the sampling frequency is low, and the acquired images are accurate, the second preset servo frequency is the frequency at which the encoder acquires, the sampling frequency is fast, and the measured data is less. The two data are fused to obtain pose error information with higher frequency and higher accuracy. Generally, the first preset servo frequency is 30 Hz, and the second preset servo frequency is 1 kHz.

[0053] In some embodiments, the pose error information of the labeling area and the labeling mechanism is determined by the following formula:

[0054] errDist k =L0+Δy t +(l 2k -l 2t )-(l 1k -l 1t )

[0055] Wherein, errDist k represents the pose error information of the labeling area and the labeling mechanism along the conveying direction at time k, L0 represents the installation position deviation of the center of the shooting mechanism and the center of the labeling mechanism along the conveying direction, Δy t represents the pose error information of the center of the shooting mechanism and the center of the labeling area along the conveying direction at time t, l 2k represents the pose information of the execution mechanism at time k, l 2t represents the pose information of the execution mechanism at time t, l 1k represents the pose information of the conveying mechanism at time k, l 1t represents the pose information of the conveying mechanism at time t.

[0056] Specifically, based on the foregoing analysis, the pose error information of the labeling region and the labeling mechanism is obtained by calculating the pose information of the labeling region at a first preset servo frequency, obtaining the pose information of the execution mechanism and the pose information of the conveying mechanism at a second preset servo frequency, and fusing at the second preset servo frequency. Wherein, the first preset servo frequency is less than the second preset servo frequency, so that obtaining the pose information of the execution mechanism and the pose information of the conveying mechanism at the second preset servo frequency will be shorter than the interval time of calculating the pose information of the labeling region at the first preset servo frequency.

[0057] Specifically, in the dynamic tracking process, the imaging mechanism captures the i-1th frame of the image of the object to be labeled at t-1 time, and captures the i th frame of the image of the object to be labeled at t time. The i-1th frame image and the i th frame image are matched and tracked, and the pose information of the i th frame (t time) labeling region is calculated at a first preset servo frequency. K time is between t time and t+1 time, so when the pose information of the i th frame (t time) labeling region is obtained at the first preset servo frequency, the pose information l 2t of the execution mechanism at t time can be obtained at a second preset servo frequency. 1t And before obtaining the pose information of the i+1th frame (t+1 time) labeling region, the pose information l 2k of the execution mechanism at k time can also be obtained. 1k The pose information l 2t of the execution mechanism at t time and the pose information l 1t of the execution mechanism at k time are fused at the second preset servo frequency, and the pose information l 2k of the execution mechanism at k time is obtained at the second preset servo frequency. 1k The pose information l k of the execution mechanism at k time is fused at the second preset servo frequency, and the pose error information errDist t of the labeling region and the labeling mechanism can be obtained, which realizes the timestamp alignment of the labeling region of the object to be labeled and the conveying mechanism and the execution mechanism.

[0058] For example, based on the pose information of the labeling region and the pose information of the execution mechanism, the pose error information Δy 2k of the center of the imaging mechanism and the center of the labeling region along the conveying direction can be calculated. 2t The change amount of the pose information of the execution mechanism between t time and k time is represented by l 1k The change amount of the pose information of the execution mechanism between t time and k time is represented by l 1tThis represents the change in pose information of the transmission mechanism from time t to time k. During dynamic tracking, the actuator is continuously adjusted in the three degrees of freedom (X, Y, and Rz) in the horizontal direction to ensure that the pose error Δy between the center of the imaging mechanism and the center of the area to be labeled along the transmission direction is constant. t It is almost zero, while making the change in pose information of the actuator almost equal to the change in pose information of the transmission mechanism, even if (l 2k -l 2t )-(l 1k -l 1t The value of ) is almost 0. In addition, the shooting mechanism and labeling mechanism are detachably mounted on the actuator. By default, the center of the shooting mechanism and the center of the labeling mechanism should be basically aligned. Therefore, during dynamic tracking, the installation position deviation L0 between the center of the shooting mechanism and the center of the labeling mechanism along the conveying direction is continuously adjusted until it is almost zero. This reduces the pose error errDist between the labeling area and the labeling mechanism. k Close to 0.

[0059] In this way, the pose information of the area to be labeled, the pose information of the actuator, and the pose information of the conveying mechanism are fused to obtain the pose error information of the area to be labeled and the labeling mechanism along the conveying direction. During the tracking process, the actuator is continuously adjusted in the three degrees of freedom of the horizontal direction (X, Y, and Rz) to achieve dynamic tracking of the area to be labeled on the object without stopping the conveying mechanism.

[0060] In some embodiments, in the presence of external interference, the pose error information between the labeling area and the labeling mechanism is determined by the following formula:

[0061] errDist (k+1) =L0+Δy′ t+1 +(l 2(k+1) -l 2(t+1) )-(l 1(k+1) -l 1(t+1) )

[0062] Among them, errDist (k+1) This represents the pose error information of the labeling area and the labeling mechanism along the conveying direction at time k+1, Δy′. t+1 This represents the pose error information between the center of the imaging mechanism and the center of the area to be labeled at time t+1 along the conveying direction. 2(k+1) This represents the pose information of the actuator at time k+1, l 2(t+1) This represents the pose information of the actuator at time t+1, l 1(k+1) This represents the pose information of the transmission mechanism at time k+1, l 1(t+1) This represents the pose information of the transmission mechanism at time t+1.

[0063] Specifically, assuming that at time k during the dynamic tracking process, the system is subjected to an external disturbance, such as the actuator being moved manually, then at time t+1 (the (i+1)th frame), compensation for this disturbance is performed. This involves matching tracking between the i-th and i+1th frames, calculating the pose information of the area to be labeled in the i+1th frame (time t+1) using a first preset servo frequency. Since time k+1 falls between time t+1 and time t+2, when the pose information of the area to be labeled in the i+1th frame (time t+1) is obtained using the first preset servo frequency, the pose information of the actuator at time t+1 can be obtained using a second preset servo frequency. 2(t+1) and the pose information of the transmission mechanism 1(t+1) Furthermore, before obtaining the pose information of the area to be labeled in the (i+2)th frame (time t+2), the pose information of the actuator at time k+1 can also be obtained. 2(k+1) and the pose information of the transmission mechanism 1(k+1) Then, using the second preset servo frequency, the pose information of the area to be labeled in the (i+1)th frame (time t+1) and the pose information of the actuator at time t+1 are... 2t and the pose information of the transmission mechanism 1t The pose information of the actuator at time k+1 2k and the pose information of the transmission mechanism 1k By fusing the data, the pose error information between the labeling area and the labeling mechanism can be obtained (errDist). (k+1) This achieves alignment of the labeling area of ​​the object to be labeled with the timestamps of the conveying mechanism and the execution mechanism.

[0064] For example, based on the pose information of the area to be labeled and the pose information of the actuator, the pose error information Δy′ between the center of the shooting mechanism and the center of the area to be labeled along the conveying direction is calculated. t+1 Δy t Updated to Δy′ t+1 To compensate for external disturbances. 2(k+1) -l 2(t+1) This represents the change in the pose information of the actuator between time t+1 and time k+1; l 1(k+1) -l 1(t+1) This represents the change in pose information of the transmission mechanism from time t+1 to time k+1. During dynamic tracking, the actuator is continuously adjusted in the three degrees of freedom (X, Y, and Rz) in the horizontal direction to ensure the pose error Δy′ between the center of the imaging mechanism and the center of the area to be labeled along the transmission direction is corrected. t+1 It is almost zero, while making the change in pose information of the actuator almost equal to the change in pose information of the transmission mechanism, even if (l 2(k+1) -l 2(t+1) )-(l 1(k+1) -l1(t+1) ) is almost 0. In addition, the installation position deviation L0 of the center of the shooting mechanism and the center of the labeling mechanism in the conveying direction is also readjusted to be almost 0. In this way, the pose error errDist of the region to be labeled and the labeling mechanism is almost 0. (k+1) close to 0.

[0065] In this way, when external disturbances occur in the tracking stage, the external disturbances can be compensated for at the next moment, thereby improving the robustness of the system.

[0066] In some embodiments, the pose information of the region to be labeled of the object to be labeled is acquired according to the image information, including: performing feature extraction on the image information to obtain a region of interest of the object to be labeled; performing feature extraction on the region of interest to obtain coordinates of feature points in the region of interest; determining a plane equation of the region of interest based on the coordinates of the feature points; determining a target feature point according to a preset geometric constraint condition and the plane equation; and determining the pose of the region to be labeled in the image information according to the coordinates of the target feature point.

[0067] Specifically, the shooting mechanism continuously acquires image information of the object to be labeled in the tracking process, and by processing the image information, the pose information of the region to be labeled of the object to be labeled can be acquired.

[0068] For example, first, the first frame of image of the object to be labeled acquired by the shooting mechanism is preprocessed, for example, the image is standardized and normalized to eliminate interference factors and highlight feature information, thereby obtaining a region of interest of the object to be labeled. Then, feature extraction is performed on the region of interest, wherein the feature extraction includes extraction of feature points and extraction of feature line segments. For the extraction of feature points, SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF) can be used. For the extraction of feature line segments, LSD (Line Segment Decttor) can be used. Then, based on the coordinates of the feature points and the coordinates of the endpoints of the feature line segments, a plane equation of the region of interest can be calculated using a least squares fitting algorithm. The plane equation is as follows:

[0069] z = a0 * x + a1 * y + a2

[0070] wherein a0, a1, and a2 are plane equation coefficients, and the matrix form of the plane equation coefficients is as follows:

[0071]

[0072] wherein (x i , y i , z i ) is the coordinate of a feature point or the coordinate of an end point of a feature line segment; then, the distance d i of each point coordinate (x i , y i , z i ) to the plane equation is calculated, and the target feature point satisfying the geometric constraint condition is screened out in the distance d i . Finally, the target feature points screened out are clustered to obtain feature points of a line feature point, a graph or a character with good saliency, and the number of the target feature points is usually required to be greater than or equal to 4. Based on the target feature points obtained through clustering, the pose of the labeling area in the image information can be determined through the PNP (Perspective-N-Point, three-dimensional object pose estimation) algorithm.

[0073] In some embodiments, during dynamic tracking, the image information is matched and tracked based on a template matching method.

[0074] Specifically, during dynamic tracking, the shooting mechanism installed on the actuator continuously shoots to obtain multiple frames of images of the object to be labeled. In the above embodiments, the first frame of image information has been processed, and the region of interest of the object to be labeled and the pose information of the labeling area of the object to be labeled have been obtained. For subsequent frames of images, only the feature points and feature line segments in the region of interest of the object to be labeled need to be searched, and the subsequent frames of images are matched and tracked based on the template matching method, and the pose information of the labeling area of the object to be labeled is calculated through the PNP algorithm at a first preset servo frequency.

[0075] In this way, the subsequent frames of images are matched and tracked based on the template matching method, and the subsequent frames of images do not need to be preprocessed, which improves the calculation efficiency and further improves the tracking speed.

[0076] In some embodiments, the target feature points are determined according to the preset geometric constraint condition and the plane equation, including: in the case that the preset geometric constraint condition is a distance constraint, the distance of the feature points to the plane equation is obtained, and the feature points with a distance less than a preset distance are taken as the target feature points; in the case that the preset geometric constraint condition is a proportion constraint, the distances of the feature points to the plane equation are obtained, and all the distances are arranged in ascending order, and the feature points in the front preset proportion are taken as the target feature points.

[0077] Specifically, the preset geometric constraint condition can be a distance constraint and a proportion constraint. In the case of the distance constraint or the proportion constraint, the feature points (x i , y i , z i) are screened to obtain target feature points meeting distance constraints or proportion constraints.

[0078] For example, when the preset geometric constraint condition is distance constraints, the distance d i of the feature point (x i , y i , z i ) to the plane equation is obtained, and the feature point with a distance d i less than a preset distance d threshold is taken as a target feature point; when the preset geometric constraint condition is proportion constraints, the distance d i of the feature point (x i , y i , z i ) to the plane equation is obtained, all distances d i are arranged in ascending order, and the feature point with a proportion p threshold in the front is taken as a target feature point. In this way, some feature points or feature line segments with poor saliency or far from the plane are screened out, and the positioning accuracy is improved.

[0079] In some embodiments, when the error between the pose information of the labeling mechanism and the pose information of the region to be labeled is less than a preset error threshold, and the moving speed of the execution mechanism is the same as the moving speed of the object to be labeled, it is determined that the labeling mechanism is synchronized with the region to be labeled.

[0080] Specifically, the dynamic labeling process is divided into three stages, namely, a dynamic tracking stage, a synchronous tracking and labeling stage, and a reset and zero stage.

[0081] For example, in the dynamic tracking process, a motion control instruction is generated according to the error information between the pose information of the labeling mechanism and the pose information of the region to be labeled, the pose information of the execution mechanism, and the pose information of the conveying mechanism. The execution mechanism continuously adjusts in three degrees of freedom of the horizontal direction X, Y, and Rz based on the motion control instruction. When the error between the pose information of the labeling mechanism and the pose information of the region to be labeled is less than a preset error threshold, and the moving speed of the execution mechanism is the same as the moving speed of the object to be labeled, it is determined that the labeling mechanism is synchronized with the region to be labeled, i.e., the dynamic tracking stage ends, and the synchronous tracking and labeling stage is entered. At this time, the execution mechanism controls the labeling mechanism to quickly press down, accurately labels the label to the region to be labeled of the object to be labeled, and completes the labeling operation. Then, the execution mechanism returns to the starting position, enters the reset and zero stage, waits for the next object to be labeled to arrive, and circulates to label the object to be labeled on the conveying mechanism.

[0082] In summary, the motion control instruction is generated according to the error information between the pose information of the labeling mechanism and the pose information of the labeling area, the pose information of the execution mechanism and the pose information of the conveying mechanism, the execution mechanism dynamically tracks the labeling area of the object to be labeled based on the motion control instruction, and the labeling mechanism labels the labeling area when the labeling mechanism is synchronized with the labeling area, so that dynamic tracking and labeling are completed without stopping the conveying mechanism, and the efficiency and automation level of visual guided labeling are improved. In addition, when external disturbance occurs in the tracking stage, compensation for the external disturbance can be realized, and the robustness of the system is improved.

[0083] In some embodiments, referring to Figure 1 A machine vision labeling system is also provided, which includes an execution mechanism 11, a shooting mechanism 12, a conveying mechanism 13, a labeling mechanism 14 and a controller 16 (visual motion controller). The shooting mechanism 12 and the labeling mechanism 14 are detachably mounted on the execution mechanism 11, and the conveying mechanism 13 is configured to convey the object to be labeled. The shooting mechanism 12 is used to obtain image information of the object to be labeled during the movement of the execution mechanism 11 following the object to be labeled. The controller 16 is used to obtain the pose information of the labeling area of the object to be labeled according to the image information, and to control the execution mechanism 11 to dynamically track according to the pose information of the execution mechanism 11, the pose information of the conveying mechanism 13 and the pose information of the labeling area, and to control the labeling mechanism 14 to label the labeling area when the labeling mechanism 14 is synchronized with the labeling area.

[0084] Specifically, the conveying mechanism 13, such as a conveying belt, is configured to convey the object to be labeled and can move smoothly in the horizontal direction after being started. The conveying mechanism 13 is also provided with a photoelectric sensor 15, which can be one or more, for detecting whether there is an object to be labeled on the conveying mechanism 13. The execution mechanism 11, such as a gantry device or a SCARA robot, includes an execution mechanism body and an execution mechanism end effector, which is used to dynamically track the object to be labeled and has the same tracking direction as the movement direction of the conveying mechanism 13. The shooting mechanism 12 is used to obtain image information of the object to be labeled during the movement of the execution mechanism 11 following the object to be labeled, and to upload the image information to the controller 16. The labeling mechanism 14 can move up and down in the vertical direction and is controlled by the execution mechanism 11, which is used to perform labeling operation when the labeling mechanism 14 is synchronized with the labeling area. The shooting mechanism 12 and the labeling mechanism 14 are detachably mounted on the execution mechanism 11 (execution mechanism end effector), wherein there is a height difference between the shooting mechanism 12 and the labeling mechanism 14, the labeling mechanism 14 is installed closer to the conveying mechanism 13, and the vertical distance between the shooting mechanism 12 and the object to be labeled remains unchanged.

[0085] Specifically, the controller 16 integrates a vision processing function block and a motion control function block, and the two function blocks interact in real time through shared memory, wherein the vision processing function block includes a pose fusion module, and the motion control function block includes a dynamic tracking module. The controller 16 can also obtain real-time feedback of the pose information of the execution mechanism 11 and the pose information of the conveying mechanism 13 from the servo driver 17 through the EtherCAT bus at a second servo frequency.

[0086] The vision processing function block is used to obtain the pose information of the labeling region of the object to be labeled according to the image information, wherein the pose fusion module is used to fuse the pose information of the execution mechanism 11, the pose information of the conveying mechanism 13 and the pose information of the labeling region, and output the pose error information of the labeling region and the labeling mechanism 14.

[0087] The dynamic tracking module in the motion control function block is used to control the execution mechanism 11 to perform dynamic tracking. For example, the dynamic tracking module can use a PID (Proportional Integral Differential) control method. The pose error of the labeling region and the labeling mechanism 14, the proportional coefficient K p , the differential coefficient K d and the integral coefficient K i are input into the dynamic tracking module. Based on the coefficients and the pose error of the labeling region and the labeling mechanism 14, the dynamic tracking module calculates the control amount of the current control period through the following formula:

[0088]

[0089] Wherein, u(t) is the control amount of the current control period, and errDist t is the pose error curve of the labeling region and the labeling mechanism 14. The dynamic tracking module outputs a motion control instruction to the servo driver 17 according to the pose information of the execution mechanism 11, the pose information of the conveying mechanism 13 and the control amount of the current control period. The execution mechanism 11 performs dynamic tracking based on the motion control instruction issued by the servo driver 17, and controls the labeling mechanism 14 to label the labeling region when the labeling mechanism 14 and the labeling region are synchronized.

[0090] As a specific example, the actuator 11 is in a servo state at the starting position, waiting for the arrival of the object to be labeled, after the photoelectric sensor 15 detects the object to be labeled, outputs a trigger signal to the controller 16, the controller 16 generates a periodic hardware trigger signal and transmits it to the shooting mechanism 12, triggers the shooting mechanism 12 to shoot and obtain image information, and uploads the image information to the controller 16, while counting the number of objects to be labeled. After the controller 16 obtains the image information of the object to be labeled, it is processed, first, the first frame of image is preprocessed to obtain the region of interest of the object to be labeled, then the region of interest is feature extracted to obtain feature points and feature lines, then the feature point coordinates and feature line endpoint coordinates are screened with distance constraint or scale constraint to obtain target feature points that meet the constraint condition, and finally, the target feature points screened out are clustered to obtain line feature points, graphics or character feature points with good saliency, usually requiring the number of target feature points to be greater than or equal to 4, based on the target feature points obtained by clustering, the pose information of the labeling area in the image information is obtained by PNP algorithm. For subsequent frame images, only feature points and feature lines need to be searched in the region of interest of the object to be labeled, based on the template matching method, matching and tracking between adjacent frame images are performed, and the pose information of the labeling area of the moving object to be labeled is calculated at a first preset servo frequency. At the same time, the controller 16 obtains the real-time feedback pose information of the actuator 11 and the pose information of the conveying mechanism 13 from the servo driver 17 through the EtherCAT bus at a second servo frequency. The pose information of the labeling area of the object to be labeled calculated at the first preset servo frequency, the pose information of the actuator 11 and the pose information of the conveying mechanism 13 obtained at the second servo frequency are transmitted to the pose fusion module for fusion, and the pose error information of the labeling area and the labeling mechanism 14 is output. The dynamic tracking module outputs control commands to the servo driver 17 according to the pose information of the actuator 11, the pose information of the conveying mechanism 13 and the pose error information of the labeling area and the labeling mechanism 14, and the actuator 11 continuously adjusts the three degrees of freedom of horizontal direction X, Y and Rz based on the control commands issued by the servo driver 17 to dynamically track the object to be labeled. When the error between the pose information of the labeling mechanism 14 and the pose information of the labeling area is less than a preset error threshold, and the movement speed of the actuator 11 is the same as the movement speed of the object to be labeled, it is determined that the labeling mechanism 14 is synchronized with the labeling area, at this time, the labeling mechanism 14 is controlled to be quickly pressed to label the labeling area, and after the labeling operation is completed, the actuator 11 returns to the starting position to wait for the arrival of the next object to be labeled, and the cycle is repeated.

[0091] It should be noted that the above explanation and description of the embodiments and beneficial effects of the control method of the machine vision labeling system also apply to the machine vision labeling system of the embodiments of the present application, to avoid redundancy, which will not be described in detail here.

[0092] In some embodiments, there is also provided a computer readable storage medium having stored thereon a control program of a machine vision labeling system, which, when executed by a processor, implements the aforementioned control method of the machine vision labeling system.

[0093] It should be noted that the above explanations of the embodiments and advantages of the control method of the machine vision labeling system also apply to the computer readable storage medium of the embodiments of the present application, and to avoid redundancy, they will not be described in detail here.

[0094] In some embodiments, with reference to Figure 3 There is also provided a labeling apparatus 100 comprising the aforementioned machine vision labeling system 110.

[0095] It should be noted that the above explanations of the embodiments and advantages of the control method of the machine vision labeling system also apply to the labeling apparatus of the embodiments of the present application, and to avoid redundancy, they will not be described in detail here.

[0096] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of the above. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical apparatus), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, by optically scanning the paper or other suitable medium, then electronically converted into a form that is suitable for use by the instruction execution system, apparatus, or device, and for example, stored in the computer memory.

[0097] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0098] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0099] In addition, the terms "first", "second", are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0100] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A control method for a machine vision labeling system, characterized in that, The system includes an actuator, an imaging mechanism, a conveying mechanism, and a labeling mechanism. The imaging mechanism and the labeling mechanism are detachably mounted on the actuator. The imaging mechanism is configured to acquire image information, and the conveying mechanism is configured to convey the object to be labeled. The method includes: During the process of the actuator following the movement of the object to be labeled, image information of the object to be labeled is acquired; The pose information of the labeling area of ​​the object to be labeled is obtained based on the image information; The actuator is controlled to perform dynamic tracking based on the position information of the actuator, the position information of the conveying mechanism, and the position information of the area to be labeled. When the labeling mechanism is synchronized with the area to be labeled, the labeling mechanism is controlled to label the area to be labeled. The process of controlling the actuator to perform dynamic tracking based on the pose information of the actuator, the pose information of the conveying mechanism, and the pose information of the area to be labeled includes: The positional error information between the labeling area and the labeling mechanism is determined based on the positional information of the actuator, the positional information of the conveying mechanism, and the positional information of the labeling area. The positional error information between the labeling area and the labeling mechanism is determined using the following formula: in, This represents the pose error information between the area to be labeled and the labeling mechanism along the conveying direction at time k. This indicates the installation position deviation between the center of the shooting mechanism and the center of the labeling mechanism along the conveying direction. This represents the pose error information between the center of the imaging mechanism and the center of the area to be labeled at time t along the conveying direction. This represents the pose information of the actuator at time k. This represents the pose information of the actuator at time t. This represents the pose information of the transmission mechanism at time k. This represents the pose information of the transmission mechanism at time t.

2. The control method according to claim 1, characterized in that, Based on the pose information of the actuator, the pose information of the conveying mechanism, and the pose information of the area to be labeled, the actuator is controlled to perform dynamic tracking, including: Motion control commands are generated based on the pose error information, the pose information of the actuator, and the pose information of the transmission mechanism. The actuator is controlled to perform dynamic tracking according to the motion control command.

3. The control method according to claim 2, characterized in that, Controlling the actuator to perform dynamic tracking according to the motion control command includes: Dynamic tracking is achieved by adjusting the three degrees of freedom of the actuator in the horizontal direction (X, Y, and Rz).

4. The control method according to claim 1, characterized in that, In the presence of external interference, the pose error information between the labeling area and the labeling mechanism is determined using the following formula: in, This represents the pose error information between the labeling area and the labeling mechanism along the conveying direction at time k+1. This represents the pose error information between the center of the shooting mechanism and the center of the area to be labeled at time t+1 along the conveying direction. This represents the pose information of the actuator at time k+1. This represents the pose information of the actuator at time t+1. This represents the pose information of the transmission mechanism at time k+1. This represents the pose information of the transmission mechanism at time t+1.

5. The control method according to claim 1, characterized in that, Obtaining the pose information of the labeling region of the object to be labeled based on the image information includes: Feature extraction is performed on the image information to obtain the region of interest of the object to be labeled; Feature extraction is performed on the region of interest to obtain the coordinates of feature points in the region of interest; Determine the plane equation of the region of interest based on the coordinates of the feature points; The target feature points are determined based on the preset geometric constraints and the plane equation. The pose of the region to be labeled in the image information is determined based on the coordinates of the target feature points.

6. The control method according to claim 5, characterized in that, Also includes: During dynamic tracking, the image information is matched and tracked based on a template matching method.

7. The control method according to claim 5, characterized in that, Determining target feature points based on preset geometric constraints and the plane equation includes: When the preset geometric constraint is a distance constraint, the distance from the feature point to the plane equation is obtained, and the feature point whose distance is less than the preset distance is taken as the target feature point; When the preset geometric condition is a proportional constraint, the distance from the feature point to the plane equation is obtained, and all distances are arranged in ascending order. The feature point with the first preset ratio is taken as the target feature point.

8. The control method according to any one of claims 1-7, characterized in that, Also includes: The pose information of the area to be labeled is obtained at a first preset servo frequency, and the pose information of the actuator and the pose information of the conveying mechanism are obtained at a second preset servo frequency. Wherein, the first preset servo frequency is less than the second preset servo frequency.

9. The control method according to any one of claims 1-7, characterized in that, If the error between the pose information of the labeling mechanism and the pose information of the area to be labeled is less than a preset error threshold, and the moving speed of the actuator is the same as the moving speed of the object to be labeled, then the labeling mechanism is determined to be synchronized with the area to be labeled.

10. A machine vision labeling system, characterized in that, The system includes an actuator, a camera, a conveying mechanism, a labeling mechanism, and a controller. The camera and labeling mechanisms are detachably mounted on the actuator. The conveying mechanism is configured to convey the object to be labeled. The shooting mechanism is used to acquire image information of the object to be labeled during the movement of the execution mechanism following the object to be labeled. The controller is configured to obtain the pose information of the labeling area of ​​the object to be labeled based on the image information, and control the actuator to perform dynamic tracking based on the pose information of the actuator, the pose information of the conveying mechanism and the pose information of the labeling area, and control the labeling mechanism to label the labeling area when the labeling mechanism is synchronized with the labeling area. The process of controlling the actuator to perform dynamic tracking based on the pose information of the actuator, the pose information of the conveying mechanism, and the pose information of the area to be labeled includes: The positional error information between the labeling area and the labeling mechanism is determined based on the positional information of the actuator, the positional information of the conveying mechanism, and the positional information of the labeling area. The positional error information between the labeling area and the labeling mechanism is determined using the following formula: in, This represents the pose error information between the area to be labeled and the labeling mechanism along the conveying direction at time k. This indicates the installation position deviation between the center of the shooting mechanism and the center of the labeling mechanism along the conveying direction. This represents the pose error information between the center of the imaging mechanism and the center of the area to be labeled at time t along the conveying direction. This represents the pose information of the actuator at time k. This represents the pose information of the actuator at time t. This represents the pose information of the transmission mechanism at time k. This represents the pose information of the transmission mechanism at time t.

11. A computer-readable storage medium, characterized in that, It stores a control program for a machine vision labeling system, which, when executed by a processor, implements the control method for the machine vision labeling system according to any one of claims 1-9.

12. A labeling device, characterized in that, Including the machine vision labeling system as described in claim 10.

Images