Embedded active vision servo control method, system and defect recognition method

Abstract

This application provides an embedded active visual servo control method, system, and defect identification method, applied in the field of robotics. The steps are as follows: acquiring real-time image data of the manipulated object; obtaining the intensity values ​​of pixels in the real-time image based on the real-time image data; obtaining an error value based on the intensity values ​​of pixels in the real-time image and the intensity values ​​of pixels in a preset image; obtaining the input control quantities of the robot's joint actuators based on the error value, the pseudo-inverse matrix of the robotic arm's Jacobian matrix, and the gain factor. By extracting the intensity values ​​of pixels from the real-time image data of the manipulated object and using these intensity values ​​to obtain the input control quantities of the visual servo controller, the control of the robotic arm is achieved, reducing computation time and computational load. Furthermore, this process is implemented in an embedded system, further improving the real-time performance of the entire control. Simultaneously, based on the intensity values, defect identification of the manipulated object's surface is also achieved.

Description

Technical Field

[0001] This application relates to the technical field of robots, specifically to an embedded active vision servo control method, system, and defect identification method. Background Technology

[0002] Mobile manipulation robots consist of a mobile chassis and a multi-degree-of-freedom robotic arm. Typically, mobile manipulation robots are equipped with a vision processing system that uses photos taken by a camera to guide the robotic arm in operations such as object recognition, grasping, and transportation.

[0003] However, current vision processing systems need to extract feature information from the image data captured by the camera after the camera takes a picture. This involves extracting the outline, shape, or other feature information of the object being manipulated from the image data. However, this process requires a lot of computing resources and time, which causes a bottleneck problem that reduces the real-time control of the vision processing system and needs to be improved.

[0004] Therefore, a new vision processing system is needed that can reduce the consumption of computing resources and computing time, and improve the real-time performance of system control. Summary of the Invention

[0005] In view of this, the embodiments of this specification provide an active visual servo control method, system, and defect identification method, which reduces the consumption of computing resources and computing time, and improves the real-time performance of system control.

[0006] The embodiments in this specification provide the following technical solutions:

[0007] This specification provides an active vision servo control method, the steps of which are as follows:

[0008] Acquire real-time image data of the object being manipulated;

[0009] Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained;

[0010] The error value is obtained based on the intensity values ​​of all pixels in the real-time image and the intensity values ​​of all pixels in the preset image;

[0011] Based on the error value, the pseudo-inverse of the Jacobian matrix of the robotic arm, and the gain factor, the input control quantities of the actuators of each joint of the robot are obtained.

[0012] By using the above technical solution, after acquiring the real-time image data of the object being manipulated, instead of extracting complex image features, the intensity values ​​of all pixels in the real-time image are extracted. Compared to complex image features such as contours, shapes, and sizes, extracting only the intensity values ​​of pixels requires far less computation than extracting complex image features. Then, by comparing the intensity values ​​of the pixels in the real-time image with those in a preset image, an error value is obtained. Based on this error value, the input control quantities of the robot's joint actuators are calculated, and finally, the robotic arm is controlled. This process reduces both the consumption of computational resources and the computation time, significantly reducing the computational load of the entire control process and greatly improving the speed, thereby enhancing the real-time performance of the entire control process.

[0013] Preferably, the grayscale value of the real-time image is obtained based on the real-time image data of the object being operated on;

[0014] The smoothness of the object's surface is obtained based on the grayscale values ​​of the real-time image.

[0015] Adjust the illumination level and / or the camera angle for capturing real-time images based on the smoothness of the object's surface; wherein the illumination level has at least three settings.

[0016] By adjusting the shooting brightness and / or shooting angle of the target object, image data under different brightness and angles can be obtained, which is more conducive to image data processing and analysis.

[0017] This specification also provides an embedded vision processing system, including a PCB board, a camera, and a light source;

[0018] The camera is mounted on the wrist of the robotic arm to acquire real-time image data of the object being manipulated;

[0019] The PCB board is equipped with an embedded data processing system, a camera interface, a PCI interface, and a lighting control interface;

[0020] The PCB board is embedded inside the robot controller and is connected to the PCI interface slot inside the robot controller via a PCI interface.

[0021] The embedded data processing system connects to the camera via a camera interface. Based on the real-time image data of the object being operated acquired by the camera, it obtains the intensity values ​​of all pixels in the real-time image. Based on the intensity values ​​of all pixels in the real-time image and the intensity values ​​of all pixels in the preset image, it obtains the error value. Based on the error value, the pseudo-inverse matrix of the Jacobian matrix of the robotic arm, and the gain factor, it obtains the input control quantities of the actuators of each joint of the robot.

[0022] The light source is installed at the end of the robotic arm's wrist and is controlled by the vision processing program in the embedded data processing system through a light source control interface.

[0023] By integrating the embedded data processing system onto a PCB board and then directly embedding the PCB board into the robot controller, the real-time image data obtained by the camera can be directly processed by the embedded data processing system. The embedded data processing system directly generates feedback quantities for the vision servo controller, which are the input control quantities for the robot's joint actuators, thus improving the real-time performance of the data processing and consequently enhancing the real-time performance of the entire control process.

[0024] Preferably, the embedded data processing system obtains the grayscale value of the real-time image based on the real-time image data of the object being operated on, obtains the smoothness of the surface of the object being operated on based on the grayscale value of the real-time image, and adjusts the brightness of the light source and / or the shooting angle of the camera based on the smoothness of the surface of the object being operated on.

[0025] Preferably, the light source and the camera are respectively located on both sides of the wrist at the end of the robotic arm, and the light source and the camera are set at a fixed angle.

[0026] Preferably, the angle between the supplementary lighting end of the light source and the shooting end of the camera is between 0 degrees and 45 degrees.

[0027] This specification also provides a defect identification method based on visual servoing, the steps of which are as follows:

[0028] Acquire real-time image data of the object being manipulated;

[0029] Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained;

[0030] Based on the intensity values ​​of all pixels in the real-time image, a judgment value for the surface defects of the object being operated on is obtained.

[0031] With the above technical solution, the intensity values ​​of pixels on a flat surface do not fluctuate much in the acquired image information. However, once a defective area exists, the intensity values ​​of pixels in the defective area will fluctuate significantly with the intensity values ​​of pixels on the flat surface. Therefore, by using the intensity values ​​of all pixels in the real-time image, it is possible to quickly determine whether there are defects on the surface of the object being operated on.

[0032] Preferably, real-time image data of the object under different light intensity levels are acquired; wherein, the light intensity is provided at least three levels;

[0033] Based on the real-time image data of the object under different light intensity levels, the intensity values ​​of all pixels in each real-time image under different light intensity levels are obtained;

[0034] Based on the intensity values ​​of all pixels in each real-time image under different light intensity levels, several sets of pixel intensity differences are obtained; among them, the pixel intensity difference is the difference between the intensity values ​​of all pixels in real-time images under adjacent light intensity levels;

[0035] The image edge detection operator is executed on each of the several sets of pixel intensity differences to obtain several edge images; each edge image contains several pixels and their intensity values.

[0036] Several edge images are compared with a set threshold to obtain several groups of pixels whose intensity values ​​are higher than the set threshold.

[0037] The number of pixels is compared between several groups of pixels to obtain several judgment differences; where the judgment difference is the absolute value of the difference in the number of pixels between any two groups of pixels.

[0038] Based on the judgment difference and the preset difference, a judgment value for the surface defects of the operation object is obtained; if the judgment difference is less than or equal to the preset difference, then there is a defect on the surface of the operation object, otherwise there is no defect on the surface of the operation object.

[0039] By using the above technical solution, the surface defects of the object can be judged based on real-time image data of the object under different illumination levels, thereby improving the accuracy of the judgment of surface defects.

[0040] Preferably, the judgment values ​​of surface defects of the object being operated on are obtained from multiple shooting angles;

[0041] Based on the judgment values ​​of the surface defects of the object under multiple shooting angles, the final judgment on the surface defects of the object is obtained.

[0042] By using the above technical solution, judgment values ​​of surface defects of the operating object are obtained from multiple shooting angles, and the surface defects of the operating object are judged, thereby improving the accuracy and reliability of the judgment of surface defects.

[0043] Preferably, the object being operated on is located at the focal center of the camera under different shooting angles.

[0044] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least:

[0045] By extracting the intensity values ​​of all pixels in a real-time image, the computational load is far less than that of extracting complex image features such as contours, shapes, and sizes. The intensity values ​​of pixels in the real-time image are then compared with those in a preset image to obtain an error value. Based on this error value, the input control quantity for the visual servo controller is calculated. The input control quantity for each joint actuator of the robot is then calculated using the pseudo-inverse of the Jacobian matrix of the robotic arm and the gain factor. Finally, the robotic arm is controlled. This process reduces both computational resource consumption and computation time, significantly reducing the computational load of the entire control process and greatly improving speed, thereby enhancing the real-time performance of the entire control process. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the embedded active vision servo control method in this application;

[0048] Figure 2 This is a flowchart of the method for adjusting the brightness and shooting angle of an image in this application;

[0049] Figure 3 This is a system block diagram of the embedded active vision servo control method in this application;

[0050] Figure 4 This is a schematic diagram of the PCB board structure of the embedded vision processing system in this application;

[0051] Figure 5 This is a structural diagram of the mobile operating robot in its working state according to this application;

[0052] Figure 6 This is a schematic diagram of the structure of the wrist end of the robotic arm in this application;

[0053] Figure 7 This is a system architecture block diagram of the embedded vision processing system in this application;

[0054] Figure 8 This is a flowchart illustrating the process of identifying defects on the surface of an object under different levels of illumination in this application.

[0055] Figure 9This is another flowchart illustrating the process of identifying defects on the surface of the work object under different levels of illumination in this application.

[0056] Figure 10 This is a schematic diagram of the structure for defect identification under different shooting angles in this application.

[0057] Reference numerals: 1. Object to be operated; 2. End of robotic arm wrist; 3. PCB board; 4. Camera; 5. Light source; 6. DSP system; 7. Camera interface; 8. Light source control interface; 9. Power supply and I / O interface; 10. EtherCAT interface; 11. Flash memory; 12. RAM; 13. PCI interface. Detailed Implementation

[0058] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0059] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0061] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0062] In view of this, the inventors conducted in-depth research and improvement on the visual servoing system of mobile robots and image processing methods. They found that current visual servoing systems need to extract feature information such as the outline and shape of the object being operated from image data before performing the next operation. However, this process requires a lot of computing resources and time, resulting in a bottleneck problem that reduces the real-time control performance of the visual processing system.

[0063] Based on this, the embodiments of this specification propose an embedded active vision servo control method: such as Figure 1 As shown, real-time image data of the manipulated object is first acquired. The intensity values ​​of all pixels in the real-time image are obtained from the real-time image data and compared with the intensity values ​​of all pixels in the preset image to obtain the error value. Based on the error value, the pseudo-inverse matrix of the Jacobian matrix of the robotic arm and the gain factor are introduced to obtain the input control quantities of the actuators of each joint of the robot. Compared with extracting complex image features, such as contours, shapes, and sizes, the computational load and time required to extract intensity values ​​are much smaller than those required to extract complex image features. Therefore, obtaining the input control quantities of the vision servo controller based on the intensity values ​​of pixels can greatly reduce the computational load and time required for the entire control process and improve the real-time performance of the control.

[0064] The technical solutions provided by the various embodiments of this application are described below with reference to the accompanying drawings.

[0065] like Figures 1 to 3 As shown in the embodiments of this specification, an embedded active vision servo control method is provided, and the steps are as follows:

[0066] Obtain real-time image data of the object being manipulated.

[0067] Here, the manipulated object refers to the workpiece that the robotic arm of the mobile robot needs to manipulate. Real-time image data of the manipulated object is acquired by a camera mounted on the robotic arm. The real-time image data represents the state of the manipulated object relative to the camera mounted on the robotic arm, that is, the state of the manipulated object relative to the robotic arm.

[0068] Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained.

[0069] The real-time image data of the object being manipulated is processed by an embedded data processing system to obtain the intensity values ​​of all pixels in the real-time image.

[0070] The error value is obtained based on the intensity values ​​of all pixels in the real-time image and the intensity values ​​of all pixels in the preset image;

[0071] The error value is calculated by subtracting the intensity values ​​of all pixels in the real-time image from those in the preset image. This error value represents the difference between the real-time and preset images, and can be used to confirm the next operational command for the mobile robot's arm.

[0072] The input control quantity of the vision servo controller is obtained based on the error value, the pseudo-inverse of the Jacobian matrix of the robotic arm, and the gain factor.

[0073] The formula for calculating the input control quantity of the vision servo controller is as follows:

[0074]

[0075] in, γ is the angular velocity control input for the joint servo driver, and γ is the gain factor. This is the pseudo-inverse of the Jacobian matrix for the robot. This represents the intensity values ​​of all pixels in the real-time image. This is the intensity value of all pixels in the preset image.

[0076] By comparing the intensity values ​​of all pixels in the real-time image with those in the preset image, an error value is obtained. This error value reveals the difference between the real-time and preset images. Further data processing involves transforming the error value using a Jacobian inverse matrix and applying gain control, converting it into input control values ​​for the robot's joint actuators. This adjusts the relative state between the robotic arm and the manipulated object, ensuring the final state of the robotic arm relative to the object conforms to the preset state in the system, thus enabling the robotic arm to correctly manipulate the object. By extracting the intensity values ​​of all pixels in the real-time image, instead of extracting complex image features from the real-time image in existing technologies, the computational load and time are reduced, improving the real-time performance of the entire control process.

[0077] To ensure the clarity and quality of real-time images, you can adjust the image capture brightness and shooting angle. The specific steps are as follows:

[0078] Based on the real-time image data of the object being operated on, obtain the grayscale value of the real-time image;

[0079] The smoothness of the object's surface is obtained based on the grayscale values ​​of the real-time image.

[0080] Adjust the illumination level and / or the camera angle for capturing real-time images based on the smoothness of the object's surface; wherein the illumination level has at least three settings.

[0081] In this embodiment, the brightness has three levels: low light, medium light, and high light. Smoothness When =1, it indicates that the surface of the object being manipulated is smooth. In this case, the brightness can be adjusted to a low level using a command. Smoothness When the value is 2, it indicates that the surface of the object being manipulated is relatively smooth. In this case, the brightness can be adjusted to a higher level using a command. Smoothness When the value is 3, it indicates that the surface of the object being manipulated is rough. In this case, the brightness can be adjusted to bright using a command. The camera angle for capturing real-time images can be adjusted automatically or manually; no further restrictions are imposed here.

[0082] By adjusting the illumination and camera angle for capturing real-time images, better image quality can be achieved, resulting in clearer and higher-quality real-time images. This facilitates subsequent image data processing and improves the reliability and accuracy of the entire control process.

[0083] like Figures 4 to 7 As shown in the embodiments of this specification, an embedded vision processing system is also provided, which is installed on the wrist 2 at the end of a robotic arm and includes a PCB board 3, a camera 4, and a light source 5.

[0084] The PCB board 3 integrates an embedded data processing system, a camera interface 7, a lamp control interface 8, a power and I / O interface 9, an EtherCAT interface 10, flash memory 11, RAM 12, and a PCI interface 13. In this embodiment, the embedded data processing system is specifically a DSP system 6.

[0085] In other embodiments, the embedded data processing system may also be an FPGA system or an ASIC system.

[0086] PCB board 3 is directly embedded inside the robot controller via PCI interface 13 and PCI interface slot of the robot controller. EtherCAT interface 10 and power and I / O interface 9 can be used to connect the robot controller or other devices without using PCI interface 13.

[0087] Flash memory 11 is used to store data, such as the intensity values ​​of all pixels in a real-time image.

[0088] The light source control interface 8 is used to connect to the light source 5 to control the brightness of the light source 5.

[0089] Camera interface 7 is used to connect to camera 4 to control camera 4 to capture and transmit images.

[0090] EtherCAT interface 10 is used to connect to the robot controller, perform EtherCAT communication, and output image-processed data to the robot controller.

[0091] The DSP system 6 acquires real-time image data of the object 1 from the camera interface 7; based on the acquired real-time image data of the object, it obtains the intensity value of all pixels in the real-time image; based on the intensity value of all pixels in the real-time image and the intensity value of all pixels in the preset image, it obtains the error value; based on the error value, the pseudo-inverse matrix of the Jacobian matrix of the robotic arm, and the gain factor, it obtains the input control quantity of each joint actuator of the robot.

[0092] The DSP system 6 interacts with the camera 4, light source 5, flash memory 11, and running memory 12 to obtain real-time image data, intensity values ​​of all pixels in the preset image, and other data information from the camera 4, light source 5, flash memory 11, and running memory 12, and outputs the processed data to the visual servo controller for robot control.

[0093] Since the DSP system 6 is directly integrated onto the PCB board 3, and the PCB board 3 is embedded inside the robot controller via the PCI interface 13 and the robot controller's PCI interface slot, after acquiring real-time images of the manipulated object captured by the camera 4, the DSP system 6 on the PCB board 3 can directly process the real-time image data, converting it into input control quantities that the vision servo controller can read and use. During real-time image data processing, on the one hand, the method of calculating input control quantities using pixel intensity values ​​is employed, reducing computational load and time. On the other hand, the calculation of input control quantities for each joint actuator of the robot is directly executed within the DSP system 6 on the PCB board 3. This integrates the calculation steps of input control quantities for each joint actuator into the embedded real-time system on the PCB board, transforming the computationally resource-intensive and time-consuming image data processing stage of vision servo control into real-time data processing. This combination of two aspects overcomes bottlenecks and improves the real-time performance of the entire control process.

[0094] Furthermore, the DSP system 6 obtains the grayscale value of the real-time image based on the real-time image data of the object being operated on; obtains the smoothness of the surface of the object being operated on based on the grayscale value of the real-time image; and adjusts the illumination of the light source 5 and / or the shooting angle of the camera 4 based on the smoothness of the surface of the object being operated on.

[0095] Light source 5 has three brightness levels: low, medium, and high. (Smoothness) When the value is 1, it indicates that the surface of the object being manipulated is smooth. A command is transmitted via the lamp source control interface 8 to adjust the brightness of lamp source 5 to a low level. Smoothness When the value is 2, it indicates that the surface of the object being operated on is relatively smooth. A command is transmitted via the lamp source control interface 8 to adjust the brightness of lamp source 5 to a higher level. Smoothness When =3, it indicates that the surface of the object being operated on is rough. The command is transmitted through the lamp source control interface 8 to adjust the brightness of the lamp source 5 to bright.

[0096] The shooting angle of Camera 4 can be adjusted automatically or manually; we will not make any further restrictions here.

[0097] By adjusting the brightness of the light source 5 and the shooting angle of the camera 4, the imaging quality and clarity of the real-time image can be improved, thereby enhancing the reliability and accuracy of subsequent real-time image data processing.

[0098] It should be noted that, Figure 7 The modules within the dashed box are all set on PCB board 3.

[0099] Light source 5 and camera 4 are respectively mounted on both sides of the wrist 2 at the end of the robotic arm, with a fixed angle between them. The angle between the supplementary lighting end of light source 5 and the shooting end of camera 4 ranges from 0 degrees to 45 degrees.

[0100] In this embodiment, the angle between the supplementary lighting end of the light source 5 and the shooting end of the camera 4 is . , The specific values ​​are determined by factors such as the size of the parts being detected or operated, and the height of the conveyor belt. By reasonably setting the angle between the supplementary lighting end of the lamp source 5 and the shooting end of the camera 4, shadows in the real-time image can be effectively reduced, and the imaging quality of the real-time image can be improved.

[0101] This specification also provides a defect identification method based on visual servoing, such as... Figures 8 to 10 As shown, the steps are as follows:

[0102] Acquire real-time image data of the object being manipulated;

[0103] Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained;

[0104] Based on the intensity values ​​of all pixels in the real-time image, a judgment value for the surface defects of the object being operated on is obtained.

[0105] In the acquired image information, since the intensity values ​​of pixels on a flat surface do not fluctuate significantly, once a defective area exists, the intensity values ​​of pixels in the defective area will fluctuate significantly with the intensity values ​​of pixels on the flat surface. Therefore, by using the intensity values ​​of all pixels in the real-time image, it is possible to quickly determine whether there are defects on the surface of the object being operated on.

[0106] Furthermore, real-time image data of the object under different levels of illumination are acquired; wherein, the illumination level is set to at least three levels.

[0107] In this embodiment, the illumination level has three levels: weak light, moderate light, and bright light.

[0108] Based on the real-time image data of the object under different light intensity levels, the intensity values ​​of all pixels in each real-time image under different light intensity levels are obtained;

[0109] In this embodiment, the intensity set of pixels in a real-time image captured under low light illumination is: The intensity set of pixels in a real-time image captured under relatively bright illumination is The intensity set of pixels in a real-time image captured under bright illumination is The intensity set represents the set of intensity values ​​for all pixels in a real-time image.

[0110] Based on the intensity values ​​of all pixels in each real-time image under different light intensity levels, several sets of pixel intensity differences are obtained; where the pixel intensity difference is the difference between the intensity values ​​of all pixels in real-time images under adjacent light intensity levels.

[0111] In this embodiment, there are two pixel intensity differences, namely... and .in,

[0112]

[0113]

[0114] Pixel intensity difference This represents the difference between the intensity set of pixels in a real-time image captured under brighter lighting conditions and the intensity set of pixels in a real-time image captured under weaker lighting conditions.

[0115] Pixel intensity difference This represents the difference between the intensity set of pixels in a real-time image captured under bright illumination and the intensity set of pixels in a real-time image captured under relatively bright illumination.

[0116] The image edge detection operator is executed on each of the several sets of pixel intensity differences to obtain several edge images.

[0117] In this embodiment, the pixel intensity difference is determined by an image edge detection operator. and pixel intensity difference The image was converted into edge images Ea1 and Ea2. Through processing by the image edge detection operator, edge information of the defective region in the image was obtained, which is beneficial for the next step of defect judgment.

[0118] Image edge detection operators can be processed using the Sobel algorithm or the gradient algorithm; no further restrictions are imposed here.

[0119] The intensity value of each pixel in several edge images is compared with a set threshold to obtain several groups of pixels whose intensity value is higher than the set threshold.

[0120] In this embodiment, the threshold value is set to Ea_thes.

[0121] Compare the intensity values ​​of all pixels in the edge image Ea1 with the set threshold Ea_thes: if the intensity value of a pixel is greater than the set threshold Ea_thes, then retain the current pixel and count the number of pixels N1.

[0122] Compare the intensity values ​​of all pixels in the edge image Ea2 with the set threshold Ea_thes: if the intensity value of a pixel is greater than the set threshold Ea_thes, then retain the current pixel and count the number of pixels N2.

[0123] The number of pixels is compared between several groups of pixels to obtain several judgment differences; where the judgment difference is the absolute value of the difference in the number of pixels between any two groups of pixels.

[0124] Based on the judgment difference and the preset difference, a judgment value for the surface defects of the operation object is obtained; if the judgment difference is less than or equal to the preset difference, then there is a defect on the surface of the operation object, otherwise there is no defect on the surface of the operation object.

[0125] Furthermore, for objects with complex surface conditions, multiple shooting angles can be used to determine surface defects, as follows:

[0126] Obtain judgment values ​​regarding surface defects of the workpiece from multiple different shooting angles;

[0127] Based on the judgment values ​​of the surface defects of the object under multiple shooting angles, the final judgment on the surface defects of the object is obtained.

[0128] By incorporating shooting from multiple angles, the final judgment on the surface defects of the object being operated on becomes more accurate and reliable.

[0129] like Figure 10 As shown, in this embodiment, three shooting positions are set, namely a, b, and c. The included angle between any two adjacent poses in the three poses is 0. The focal center points a0, b0, and c0 of the camera in the three poses are all located on the same horizontal plane. The distance between this horizontal plane and the shooting surface of the object is h. h can be adjusted according to the relative distance between the camera and the object so that the object is within the range of the focal center points of the camera in the three poses.

[0130] Among them, the intensity sets of pixels in the real-time image corresponding to the three levels of illumination at point a are as follows: , , The intensity sets of pixels in the real-time image corresponding to the three levels of illumination at point b are as follows: , , The intensity sets of pixels in the real-time image corresponding to the three levels of illumination at point c are as follows: , , The methods for identifying surface defects of the workpiece under three levels of illumination at points b and c are consistent with those at point a. Multi-point identification is primarily used for complex surfaces; for defects on smooth surfaces, a single point is sufficient.

[0131] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the product embodiments described later are relatively simple since they correspond to the methods; relevant parts can be referred to the descriptions in the system embodiments.

[0132] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An embedded active visual servoing control method, characterized in that, The steps are as follows: Acquire real-time image data of the object being manipulated; Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained; The difference between the intensity values ​​of all pixels in the real-time image and the intensity values ​​of all pixels in the preset image is calculated to obtain the error value, so as to determine the error between the real-time image and the preset image. Based on the error value, the pseudo-inverse of the Jacobian matrix of the robotic arm, and the gain factor, the input control quantities of the actuators of each joint of the robot are obtained. The formula for calculating the input control quantity is as follows: in, γ is the angular velocity control input for the joint servo driver, and γ is the gain factor. This is the pseudo-inverse of the Jacobian matrix for the robot. This represents the intensity values ​​of all pixels in the real-time image. This is the intensity value of all pixels in the preset image; Based on the input control quantity, the relative state between the robotic arm and the object being operated is adjusted so that the final state of the robotic arm relative to the object being operated conforms to the state preset in the system. Based on the real-time image data of the object being operated on, obtain the grayscale value of the real-time image; The smoothness of the object's surface is obtained based on the grayscale values ​​of the real-time image. Adjust the illumination level and / or the camera angle for capturing real-time images based on the smoothness of the object's surface; wherein the illumination level has at least three settings.

2. An embedded vision processing system, characterized by: This includes the PCB board, camera, and light source; The camera is mounted on the wrist of the robotic arm and is used to acquire real-time image data of the object being operated on. The PCB board is equipped with an embedded data processing system, a camera interface, a PCI interface, and a lighting control interface. The PCB board is embedded inside the robot controller, and the PCB board is connected to the PCI interface slot inside the robot controller via a PCI interface. The embedded data processing system connects to the camera via a camera interface. Based on the real-time image data of the object being manipulated acquired by the camera, it obtains the intensity values ​​of all pixels in the real-time image. Based on the intensity values ​​of all pixels in the real-time image and the intensity values ​​of all pixels in a preset image, it obtains an error value. Based on the error value, the pseudo-inverse of the Jacobian matrix of the robotic arm, and the gain factor, it obtains the input control quantities of the actuators of each joint of the robot. Based on the input control quantities, it adjusts the relative state between the robotic arm and the object being manipulated so that the final state of the robotic arm relative to the object being manipulated conforms to the preset state in the system. Based on the real-time image data of the object being operated on, the grayscale value of the real-time image is obtained; based on the grayscale value of the real-time image, the smoothness of the surface of the object being operated on is obtained; and based on the smoothness of the surface of the object being operated on, the illumination of the real-time image and / or the angle of the camera for capturing the real-time image are adjusted. The brightness level is set to at least three levels; The light source is installed at the end of the robotic arm wrist and is controlled by a vision processing program in an embedded data processing system via a light source control interface.

3. The embedded vision processing system of claim 2, wherein, The embedded data processing system obtains the grayscale value of the real-time image based on the real-time image data of the object being operated on, obtains the smoothness of the surface of the object being operated on based on the grayscale value of the real-time image, and adjusts the brightness of the light source and / or the shooting angle of the camera based on the smoothness of the surface of the object being operated on.

4. The embedded vision processing system of claim 2, wherein, The light source and the camera are respectively located on both sides of the wrist at the end of the robotic arm, and the light source and the camera are set at a fixed angle.

5. The embedded vision processing system of claim 4, wherein, The angle between the supplementary lighting end of the light source and the shooting end of the camera ranges from 0 degrees to 45 degrees.

6. A defect identification method based on visual servoing, characterized in that, The steps are as follows: Acquire real-time image data of the object being manipulated; Based on the real-time image data of the object being operated on, the intensity values ​​of all pixels in the real-time image are obtained; Based on the intensity values ​​of all pixels in the real-time image, a judgment value regarding the surface defects of the object being manipulated is obtained, including: Based on the intensity values ​​of all pixels in each real-time image under different illumination levels, several sets of pixel intensity differences are obtained, and image edge detection is performed on these several sets of pixel intensity differences to obtain several edge images. The intensity value of each pixel in several edge images is compared with a set threshold to obtain several groups of pixels whose intensity values ​​are higher than the set threshold. The number of pixels is compared between several groups of pixels to obtain several judgment differences; Based on the difference between the judgment value and the preset difference, a judgment value for the surface defects of the object being operated on is obtained.

7. The defect identification method based on visual servoing according to claim 6, characterized in that, Acquire real-time image data of the object under different levels of illumination; wherein the illumination level has at least three levels; Based on the real-time image data of the object under different light intensity levels, the intensity values ​​of all pixels in each real-time image under different light intensity levels are obtained; Based on the intensity values ​​of all pixels in each real-time image under different light intensity levels, several sets of pixel intensity differences are obtained; wherein, the pixel intensity difference is the difference between the intensity values ​​of all pixels in real-time images under adjacent light intensity levels; The image edge detection operator is executed on each of the several sets of pixel intensity differences to obtain several edge images; each edge image contains several pixels and their intensity values. Several edge images are compared with a set threshold to obtain several groups of pixels whose intensity values ​​are higher than the set threshold. The number of pixels is compared between several groups of pixels to obtain several judgment differences; wherein, the judgment difference is the absolute value of the difference in the number of pixels between any two groups of pixels. Based on the judgment difference and the preset difference, a judgment value for the surface defects of the operation object is obtained; if the judgment difference is less than or equal to the preset difference, then there is a defect on the surface of the operation object, otherwise there is no defect on the surface of the operation object.

8. The defect identification method based on visual servoing according to claim 7, characterized in that, Obtain judgment values ​​regarding surface defects of the workpiece from multiple different shooting angles; Based on the judgment values ​​of the surface defects of the object under multiple shooting angles, the final judgment on the surface defects of the object is obtained. 9.The defect identification method based on visual servoing according to claim 8, wherein, The objects being manipulated at different shooting angles are all located within the focal length center of the camera.

Images