Element plugging test system based on visual perception, control method and device

Abstract

The invention discloses an element plugging test system based on visual perception and a control method and device.The system comprises a visual perception module, a grabbing device, a conveying device and an upper computer, and the visual perception module is used for conducting visual recognition on jacks of elements to be tested and comprises a 2D camera and a 3D camera; the 2D camera has a first field of view, the 3D camera has a second field of view, the first field of view comprises the second field of view, and the first field of view range is larger than the second field of view range; the grabbing device comprises a clamping end, and the clamping end is used for clamping the plug to plug and unplug the jack; the visual perception module is mounted at the clamping end so as to move along with the clamping end; the conveying device is used for conveying to-be-tested elements; the upper computer is in communication connection with the visual perception module, the grabbing device and the conveying device. The visual perception module can be compatible with large-deviation incoming materials, so that multi-degree-of-freedom position and posture adjustment is realized; therefore, the application range of the system is greatly expanded, and the system has excellent flexibility, compatibility and accuracy.

Description

Technical Field

[0001] The present application belongs to the field of automated control technology, and in particular relates to a component plugging and unplugging test system, control method and device based on visual perception. Background Art

[0002] In the field of high-end semiconductor automated production and manufacturing, the plug-in and unplugging test and assembly process of electronic components are its core key process flows. Flexible assembly of precision electronic components is one of the core tasks of modern high-end equipment manufacturing, and robots equipped with visual sensors are important carriers for flexible and stable plug-in and unplugging of plug-ins and interfaces. In order to achieve precise plug-in and unplugging tests of electronic components, the camera is first used to image the base of the plug-in socket (target object) and perform feature extraction. Then, the position and posture of the target object are transformed to the robot coordinate system through hand-eye calibration. Finally, the robot is driven to adjust the position and posture of the component to be plugged in at the end so that it is aligned with the base of the plug-in, and the component plug-in and unplugging test is performed.

[0003] However, in batch online testing, components beyond the field of view due to large incoming material deviations are an important factor causing the failure of vision-based robot plugging and unplugging operations in industrial applications. A single visual sensor perception solution is difficult to achieve high-precision and large-range visual perception and posture adjustment.

[0004] Under the condition of constant camera resolution, an increase in the field of view (FOV) will cause the physical size corresponding to the unit pixel to become larger, and the detection accuracy will decrease accordingly. In order to ensure high-precision visual perception and posture adjustment, cameras with smaller FOV are usually selected for posture perception in the plug-in and unplug test of electronic components. When the incoming material deviation is large and the component exceeds the perception range of the visual sensor, the system will report an error and prompt that the target capture failed, which greatly restricts the quality inspection efficiency and success rate. Increasing the FOV will lead to a decrease in visual perception accuracy, which will cause the male and female heads of the component to be inaccurately aligned, and the success rate of plugging and unplugging the two will be reduced. Summary of the invention

[0005] The present application provides a component plug-in and pull-out test system, control method and device based on visual perception, aiming to solve the problems of low operating efficiency and poor accuracy caused by the inability to accommodate high-precision plug-in and pull-out tests with a wide range of incoming material deviations.

[0006] In a first aspect, the present application provides a component plug-in and pull-out test system based on visual perception, the system being used to perform a plug-in and pull-out test on a socket of a component to be tested; the system comprising:

[0007] A visual perception module, used for visually identifying the socket of the component to be tested, comprising a 2D camera and a 3D camera; wherein the 2D camera has a first field of view, the 3D camera has a second field of view, the first field of view includes the second field of view, and the first field of view range is larger than the second field of view range;

[0008] The gripping device comprises a clamping end, the clamping end is used to clamp the test plug to perform plugging and unplugging operations on the jack; and the visual perception module is installed on the clamping end to move with the clamping end;

[0009] A transport device, used for transporting the component to be tested;

[0010] A host computer is communicatively connected with the visual perception module, the grasping device and the transport device.

[0011] In a second aspect, the present application provides a component plugging and unplugging test system control method based on visual perception, the method comprising:

[0012] Acquire system preset data; the preset data includes an initial detection position of a visual perception module; the detection area corresponds to the detection position of the visual perception module.

[0013] When the target socket of the target element is transported to the detection area by the transport device, controlling the 2D camera and the 3D camera to perform detection to determine whether the target socket exists within the first field of view and the second field of view;

[0014] When the target socket does not exist within the second field of view and the target socket exists within the first field of view, the visual perception module is transported to the next detection position based on the position of the target socket, and the target socket is re-detected to see whether it is within the second field of view;

[0015] When the target jack exists within the second field of view, determining, based on the current position of the target jack, the posture parameters of the plug-in test system when plugging in or out the target jack;

[0016] An insertion and extraction test is performed on the target socket based on the posture parameter, and a next target socket of the target element is transported to the inspection area.

[0017] In a third aspect, the present application provides a component plugging and unplugging test system control device based on visual perception, the device comprising:

[0018] An acquisition module, used to acquire system preset data; the preset data includes an initial detection position of a visual perception module;

[0019] A control module, for controlling a 2D camera and a 3D camera to perform detection when a target socket of the target element is transported to a detection area by the transport device; for determining whether the target socket exists within the first field of view and the second field of view; the detection area corresponds to a detection position of the visual perception module; when the target socket does not exist within the second field of view and exists within the first field of view, the visual perception module is transported to a next detection position based on the position of the target socket, and re-detects whether the target socket is located within the second field of view; and performs a plug-in test on the target socket based on the posture parameters, and transports the next target socket of the target element to the detection area;

[0020] A processing module is used to determine, when the target socket exists within the second field of view, the posture parameters of the plug-in test system when plugging and unplugging the target socket based on the current position of the target socket.

[0021] In a fourth aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the control method of the component plugging and unplugging test system based on visual perception is implemented.

[0022] In a fifth aspect, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the control method of the component plugging and unplugging test system based on visual perception as described above.

[0023] In a sixth aspect, a computer program product is provided, the computer program product comprising computer-readable instructions, the computer-readable instructions being stored in a storage medium, one or more processors of an electronic device reading the computer-readable instructions from the storage medium, loading and executing the computer-readable instructions, so that the electronic device implements the control method of the component plug-in and pull-out test system based on visual perception as described above.

[0024] Compared with the prior art, the component plugging and unplugging test system and control method based on visual perception of the present application has a wide range and high-precision visual perception by setting up a coarse-to-fine dual-camera fusion solution. It can be compatible with the multi-degree-of-freedom position and posture adjustment of incoming components with large deviations, greatly improving the scope of application of robot component plugging and unplugging testing based on visual perception guidance, with excellent flexibility, compatibility and accuracy; and achieving higher operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of a component plugging and unplugging test system based on visual perception provided in an embodiment of the present application;

[0027] Figure 2 for Figure 1 A schematic diagram of the decomposed structure of the component plugging and unplugging test system based on visual perception of the illustrated embodiment;

[0028] Figure 3 A schematic diagram of the main process of a component plugging and unplugging test system control method based on visual perception provided in an embodiment of the present application;

[0029] Figure 4 for Figure 3 A flow chart of an implementation method of a component plugging and unplugging test system control method based on visual perception according to the illustrated embodiment;

[0030] Figure 5 for Figure 1 A schematic diagram of the field of view of a 2D camera and a 3D camera in a component plugging and unplugging test system based on visual perception according to the illustrated embodiment;

[0031] Figure 6 A structural block diagram of a component plugging and unplugging test system control device based on visual perception provided in an embodiment of the present application;

[0032] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0035] In this embodiment, a component plugging and unplugging system based on visual perception is provided. Figure 1 and Figure 2 As shown, the system includes a visual perception module 22, a gripping device 20, a transportation device 40 and a host computer 10; the visual perception module 22 is used for visually identifying the socket of the component to be tested, and includes a 2D camera 51 and a 3D camera 52; wherein the 2D camera 51 has a first field of view, the 3D camera 52 has a second field of view, the first field of view includes the second field of view, and the first field of view range is larger than the second field of view range; the gripping device 20 includes a clamping end 21, the clamping end 21 is used to clamp the test plug 211 to perform plugging and unplugging operations on the socket 301; and the visual perception module 22 is installed on the clamping end 21 to move with the clamping end; the transportation device 40 is used to transport the component to be tested 30; the host computer 10 is communicatively connected with the visual perception module 22, the gripping device 20 and the transportation device 40.

[0036] In a specific embodiment, the gripping device 20 adopts a 6-DOF robot, wherein one end of the 6-DOF robot is fixed to the workbench, and the other end is an electric clamp, which is mounted on the end flange of the robot and is used to clamp the test plug 211. The visual perception module is composed of a 2D camera 51, an optical lens 511 and a 3D structured light camera 52, the optical lens 511 is mounted on the 2D camera 51, the 2D camera 51 and the 3D camera 52 are mounted on the camera mounting plate 23, and the camera mounting plate 23 is mounted on the electric clamp of the robot clamping end 21. The transport device 40 includes a transport platform, on which the component to be tested 30 is carried, and the component to be tested 30 moves with the transport platform. The host computer 10 can be an electronic device such as a computer or a mobile terminal, which is connected to the robot, the visual perception module 22 and the transportation device 40 through an information transmission cable or a wireless connection, and is used to control the transportation device 40 to move the component to be tested 30 to the corresponding detection position, and then control the robot to clamp the test plug 211 to perform plug and unplug test on the socket 301 of the component to be tested 30.

[0037] like Figure 5 As shown, the 2D camera 51 has a first field of view A, and the 3D camera 52 has a second field of view B. The first field of view A includes the second field of view B, and the range of the first field of view A is larger than the range of the second field of view B.

[0038] Optionally, the 2D camera 51 and the 3D camera 52 are arranged adjacent to each other, and the optical axes of the two have a certain angle, the field of view of the 3D camera 52 is within the field of view of the 2D camera 51, and the centers of the fields of view of the two cameras overlap, so that shooting can be performed at the same sampling position without driving the robot to move the camera to sample at multiple positions, which is beneficial to improving the operating efficiency of the system. Among them, the 2D camera 51 and the 3D camera 52 are installed on the camera mounting plate 23, and the camera mounting plate 23 has a slide groove, which can be used for the 2D camera 51 and the 3D camera 52 to be installed in different positions.

[0039] Understandably, since the multiple positions and postures of the surface distribution of the component to be tested 30 are different from the sockets 301, the 3D camera 52 is used to identify the specific position and posture of the socket 301 through methods such as image recognition, and control the clamping end of the robot to adopt the corresponding posture and move to the corresponding position to perform plug-in and pull-out tests on the socket 301. Since the recognition of the 3D camera 52 requires higher precision, its field of view will be limited, otherwise it will not be able to achieve a higher resolution. When the transported component to be tested 30 has a position deviation, there will be no target socket 301 within the field of view of the 3D camera, which will cause the work to be unable to continue; therefore, a 2D camera with a larger field of view is required for detection, and according to the detected position of the target socket 301, the position of the clamping end 21 is adjusted, thereby driving the 3D camera 52 and the 2D camera 51 to move to a new detection position, so that the target socket 301 can fall into the field of view of the 3D camera. Therefore, the 2D camera 51 is a large field of view, high frame rate camera, while the 3D camera 52 is a small field of view, high precision camera.

[0040] The component plugging and unplugging test system and control method based on visual perception of the present application can simultaneously meet the requirements of large-range and high-precision visual perception by setting up a coarse-to-fine dual-camera fusion solution, and can be compatible with the multi-degree-of-freedom position and posture adjustment of incoming components with large deviations, greatly improving the scope of application of robot component plugging and unplugging tests guided by visual perception, and has excellent flexibility, compatibility and accuracy.

[0041] The present application also provides an embodiment of a control method for a component extraction test system based on visual perception, which is used to control the component insertion and extraction system based on visual perception in the above embodiment, such as Figure 3 As shown, the method includes the following steps S100 to S500:

[0042] S100, obtaining system preset data;

[0043] The preset data includes an initial detection position of the visual perception module, and the detection position is used to indicate a detection area.

[0044] The initial position of the visual perception module is used as the detection position, and the area indicated by the lens line of sight of the 2D camera and the 3D camera is the detection area. The component to be tested can be transported to the detection area through a transportation device, and then the 2D camera and the 3D camera can be controlled for detection.

[0045] In an exemplary embodiment, obtaining system preset data may include the initial detection positions of the 2D camera and the 3D camera, the 2D camera pre-sampling template image, the 3D camera pre-sampling template image, the 3D camera pre-sampling point cloud data, the visual dynamic guidance control gain parameters, the robot's posture parameters when the teaching plug is inserted into the jack, etc. These preset system parameters can be used for subsequent specific control steps.

[0046] S200: When the target socket of the target component is transported to the detection area by the transport device, the 2D camera and the 3D camera are controlled to perform detection to determine whether the target socket exists within the first field of view and the second field of view.

[0047] In an exemplary embodiment, step S200 specifically includes the following steps:

[0048] S210: Control the 3D camera to perform detection to determine whether the target jack exists within the second field of view.

[0049] S220: When the target socket does not exist within the second field of view, control the 2D camera to perform detection to determine whether the target socket exists within the first field of view.

[0050] When the component to be tested is sent to the testing area, the 3D camera and the 2D camera are controlled to perform testing. If the target socket does not exist within the second field of view of the 3D camera, the 2D camera needs to be further controlled for testing. Since the 2D camera has a larger field of view, it is easier to capture the target socket for subsequent adjustment. It is also possible that the target socket exists within the second field of view of the 3D camera. In this case, it is not necessary to perform testing with the 2D camera, and step S400 can be directly executed.

[0051] In an exemplary embodiment, in this step, how to control the 3D camera and the 2D camera to perform detection to determine whether the target jack exists within the first field of view and the second field of view specifically includes the following steps:

[0052] S211 . Control the 2D camera and the 3D camera to perform sampling to obtain a current sampling graph of the 2D camera and a current sampling graph of the 3D camera.

[0053] S221: Compare the 3D camera pre-sampling template map with the 3D camera current sampling map, and the 2D camera pre-sampling template map with the 2D camera current sampling map to determine whether the target jack exists within the first field of view and the second field of view.

[0054] The 3D camera pre-sampling template map and the 2D camera pre-sampling template map are both system preset data. The 3D camera pre-sampling template map and the 2D camera pre-sampling template map are used as template reference maps under theoretical conditions. By comparing the theoretical template map with the actual sampling map, it is possible to identify whether there is a target jack within the field of view of the camera through methods such as image recognition.

[0055] In one embodiment, in order to ensure that the position feature points selected by the jack remain unchanged under the premise of the 2D camera image coordinates, the robot is driven to change different postures and different lighting conditions, and the 2D camera samples multiple pre-sampled template images Q12D, Q22D, Q32D, ..., Qm2D. If the current sampled image Q′2D and any pre-sampled template image Qi2D are successfully matched (i=1, 2, ..., m), it is determined that the jack is recognized, which greatly improves the recognition rate, reduces the missed detection rate, and improves the work efficiency. This multi-template recognition principle and method is also applicable to the template establishment and matching process of the 3D camera.

[0056] S300: When the target socket does not exist within the second field of view and the target socket exists within the first field of view, the visual perception module is transported to the next detection position based on the position of the target socket, and the target socket is re-detected to see whether it is within the second field of view.

[0057] In an exemplary embodiment, step S300 specifically includes steps S301 to S302:

[0058] S301, calculating the image position deviation between the 2D camera pre-sampling template image and the 2D camera current sampling image;

[0059] S302, determining the next detection position of the visual perception module according to the current detection position of the visual perception module, the image position deviation and the visual dynamic guidance control gain parameter, and transporting the visual perception module to the next detection position.

[0060] The purpose of this step is mainly to move the 2D camera and the 3D camera to reposition according to the position deviation of the image after comparison by the 2D camera, so as to be closer to the target socket. However, this positioning is a rough positioning based on the image deviation. After moving to the new detection position, it is necessary to repeat the above steps S211 and S221 to determine whether the target socket is within the field of view of the 3D camera.

[0061] Specifically, based on steps S211 and S221, the image position deviation obtained by template matching between the current sampling image and the pre-sampling template image is calculated and converted to the robot's coordinate system ΔP by hand-eye calibration. u , the robot reads the current position P c , the drive machine from the current detection position P c Move to the new detection position P′ c =P c +K u ΔP u , P c =P′ c .

[0062] Among them, Ku is the system preset data, used to represent the visual dynamic guidance control gain parameter; optionally, the visual dynamic guidance control gain K u It can be set so that in the initial detection position, the 2D camera observes half or less of the pixel equivalent value of the plane where the jack is located. For example, if the pixel equivalent value is 0.01 mm / pixels, then Ku≤0.005.

[0063] In one embodiment, the re-detection of whether the target socket is placed within the second field of view in step S300 determines whether the target socket enters the second field of view of the 3D camera. The determination can be made by calculating whether the image distance between the center point of the socket in the current sampling image and the center point of the socket in the pre-sampling template image is less than a set threshold. By setting a smaller allowable deviation, the position of the socket under the 3D camera system can be made as close to the template position as possible after the visually guided coarse positioning of the 2D camera, thereby reducing the calculation amount of 3D point cloud alignment and improving the calculation efficiency and solution accuracy.

[0064] In an exemplary embodiment, step S300 further includes step S310:

[0065] S310: If the target socket is not within the second field of view after multiple inspections, the next target socket of the target component is transported to the inspection area.

[0066] By repeatedly judging as no, such as more than 10 times of judging as no, the plug and unplug test of the next socket will be continued to ensure the normal operation of the assembly line and improve the operation efficiency.

[0067] S400: When the target socket exists within the second field of view, determine, based on the current position of the target socket, the posture parameters of the plugging and unplugging test system when plugging and unplugging the target socket.

[0068] In an exemplary embodiment, step S400 specifically includes steps S410 to S440:

[0069] S410, controlling the 3D camera to perform sampling to obtain current point cloud data of the 3D camera;

[0070] S420, converting the current point cloud data of the 3D camera and the pre-sampled point cloud data of the 3D camera into the coordinate system of the gripping device to perform point cloud registration to obtain a relative pose matrix;

[0071] S430, calculating calibration posture parameters according to the relative posture matrix and the set posture parameters;

[0072] S440, calculating the actual posture parameters according to the calibration posture parameters and the position difference between the current detection position and the initial detection position of the visual perception module.

[0073] Specifically, the current point cloud data of the 3D camera and the pre-sampled point cloud data of the 3D camera are converted to the robot coordinate system through hand-eye calibration, and point cloud registration is performed to obtain the relative pose matrix R 3D and T 3D , the insertion position after 3D correction is T′ bn =R 3D T b +T 3D , the insertion posture is R′ bn =R 3D R b ; The insertion position of the actual posture parameter after calculating the rough positioning deviation compensation is T bn =T′ bn +(P s -P c ), the insertion posture is R bn =R′ bn ; Among them, P c Indicates the current detection position, P s Indicates the initial detection position.

[0074] S500, performing a plug-in test on the target socket based on the posture parameters, and transporting the next target socket of the target element to the inspection area.

[0075] In an exemplary embodiment, step S500 further includes steps S510 to S520:

[0076] S510, recording the number of plug-in and plug-out tests performed to date;

[0077] S520: If the number of plug-in and pull-out tests reaches a set value, the plug-in and pull-out tests on the target component are stopped.

[0078] In one implementation of the present embodiment, when there is no target socket within the first field of view and the second field of view, it is determined that the incoming target component is beyond the field of view of the 2D camera and the 3D camera, and the number of plug-in tests to the current time is recorded. If the number of plug-in tests reaches the set value, the plug-in test of the target component is stopped.

[0079] In the actual operation process, setting the corresponding tolerance upper limit can better realize the closed-loop assembly line operation of the system and improve the operation efficiency.

[0080] The control method of the component plugging and unplugging test system based on visual perception in the present application is based on a continuous dynamic closed-loop control mode of visual feedback of a large field of view, high frame rate 2D camera, and a continuous dynamic closed-loop control mode of visual feedback of a small field of view, high precision 3D camera. The two together constitute a dual closed-loop control strategy based on 2D / 3D visual feedback information, further improving the accuracy and operating efficiency of the system.

[0081] In a specific application scenario, such as Figure 4 As shown, based on the above-mentioned component plugging and unplugging test system and control method based on visual perception, the specific operation steps are as follows:

[0082] Step 1: Initialize system parameters, set the initial counter to zero, set the total number of cycles to N, and record the initial detection position P of the 2D / 3D camera s , 2D camera pre-sampling template image Q 2D , 3D camera pre-sampling template image Q 3D , 3D camera pre-sampled template point cloud data G 3D , visual dynamics guided control gain K u , the robot position when the teaching plug is inserted into the jack is T b , posture is R b ;

[0083] Step 2: Determine whether the counter value reaches the total number of loops N. If so, jump to step 12; if not, continue to step 3;

[0084] Step 3: Control the transport device to transport the next socket of the component to be tested to below the initial detection position, and reset the current detection position P c is zero;

[0085] Step 4: Control 3D camera sampling and sample the current sampling graph Q′ 3D and the pre-sampled template graph Q 3D Perform template matching. If a jack is identified, jump to step eight; otherwise, continue to step five;

[0086] Step 5: Control 2D camera sampling and sample the current sampling graph Q′ 2D and the pre-sampled template graph Q 2D Perform template matching. If the jack is recognized, continue to step 6; otherwise, the incoming material exceeds the field of view of the 2D / 3D camera, and jump to step 11;

[0087] Step 6: Calculate the current sampling graph Q′ 2D and the pre-sampled template graph Q 2D The image position deviation obtained by template matching is converted to the robot coordinate system ΔP through hand-eye calibration u , the robot controller reads the current position P c, the controller machine detects the current position P c Move to the new detection position P′ c =P c +K u ΔP u , P c =P′ c ;

[0088] Step 7: Determine whether the current jack is within the field of view of the 3D camera. If so, return to step 4; if not, return to step 5.

[0089] Step 8: Sample the current 3D point cloud data G′ 3D and pre-sampled template point cloud data G 3D Through hand-eye calibration, the coordinate system is converted to the robot coordinate system, and point cloud registration is performed to obtain the relative pose matrix R 3D and T 3D , the insertion position after 3D correction is T′ bn =R 3D T b +T 3D , the insertion posture is R′ bn =R 3D R b ;

[0090] Step 9: Calculate the new insertion position after coarse positioning deviation compensation as T bn =T′ bn +(P s -P c ), the new insertion posture is R bn =R′ bn ;

[0091] Step 10: Perform component plug-in and pull-out test;

[0092] Step 11: counter +1, jump to step 2;

[0093] Step 12: The loop ends.

[0094] like Figure 6As shown, the embodiment of the present application also provides a component plug-in test control device 500 based on visual perception, which includes an acquisition module 501, a control module 502 and a processing module 503; wherein the acquisition module 501 is used to acquire system preset data; the preset data includes the initial detection position of the visual perception module; the control module 502 is used to control the 2D camera and the 3D camera to detect when the target socket of the target component is transported to the detection area by the transportation device; to determine whether the target socket exists within the first field of view and the second field of view; the detection area corresponds to the detection position of the visual perception module Positioning; when the target socket does not exist within the second field of view and the target socket exists within the first field of view, the visual perception module is transported to the next detection position based on the position of the target socket, and the target socket is re-detected whether it is placed within the second field of view; and plug-in and unplug test is performed on the target socket based on the posture parameters, and the next target socket of the target element is transported to the detection area; the processing module 503 is used to determine the posture parameters of the plug-in and unplug test system when plugging and unplugging the target socket based on the current position of the target socket when the target socket exists within the second field of view.

[0095] In an exemplary embodiment, the acquisition module 501 is used to acquire system preset data; the preset data also includes the initial detection positions of the 2D camera and the 3D camera, the 2D camera pre-sampling template image, the 3D camera pre-sampling template image, the 3D camera pre-sampling point cloud data, the visual dynamic guidance control gain parameters, and the posture parameters of the robot when the teaching plug is inserted into the jack.

[0096] In an exemplary embodiment, the control module 502 includes a first sampling unit, and the processing module 503 also includes a first comparison unit, the sampling unit is used to control the 2D camera and the 3D camera to perform sampling to obtain the current sampling map of the 2D camera and the current sampling map of the 3D camera; the comparison unit is used to compare the 3D camera pre-sampling template map with the 3D camera current sampling map, and the 2D camera pre-sampling template map with the 2D camera current sampling map to determine whether the target jack exists within the first field of view and the second field of view.

[0097] In an exemplary embodiment, the first comparison unit includes a comparison subunit and a determination subunit, the comparison subunit is used to compare the 3D camera pre-sampling template image with the 3D camera current sampling image; the determination unit is used to determine that the target socket is placed within the second field of view when the image distance between the socket center point of the 3D camera current sampling image and the socket center point of the 3D camera pre-sampling template image is less than a set threshold.

[0098] In an exemplary embodiment, the control module 502 also includes a second sampling unit, and the processing module also includes a second comparison unit, a first calculation unit and a second calculation unit, the second sampling unit is used to control the 3D camera to perform sampling to obtain the current point cloud data of the 3D camera; the second comparison unit is used to convert the current point cloud data of the 3D camera and the pre-sampled point cloud data of the 3D camera to the coordinate system of the grasping device for point cloud registration to obtain a relative pose matrix; the first calculation unit is used to calculate the calibration pose parameters according to the relative pose matrix and the set pose parameters; the second calculation unit is used to calculate the actual pose parameters according to the calibration pose parameters and the position difference between the current detection position and the next detection position of the visual perception module.

[0099] In an exemplary embodiment, the control module 502 further includes a transport unit, and the transport unit is used to transport the next target socket of the target element to the inspection area when the target socket is not within the second field of view after multiple inspections.

[0100] In an exemplary embodiment, the processing module 503 also includes a third calculation unit and a fourth calculation unit, the third calculation unit is used to calculate the image position deviation between the 2D camera pre-sampling template image and the 2D camera current sampling image; the fourth calculation unit is used to determine the next detection position of the visual perception module based on the current detection position of the visual perception module, the image position deviation and the visual dynamic guidance control gain parameter.

[0101] See also Figure 7 In an embodiment of the present application, an electronic device is provided, and the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0102] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0104] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0105] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0106] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0107] The computer-readable instructions are executed by one or more processors 4001 to implement the control method of the component plugging and unplugging test system based on visual perception in the above-mentioned embodiments.

[0108] In addition, a storage medium is provided in an embodiment of the present application, on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the above-mentioned control method of the component plug-in and pull-out test system based on visual perception.

[0109] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the above-mentioned control method of a component plug-in and pull-out test system based on visual perception.

[0110] Compared with the related art, the component plug-in and pull-out test system and control method based on visual perception of the present application has a wide range and high-precision visual perception by setting a coarse-to-fine dual-camera fusion solution, which can be compatible with the multi-degree-of-freedom position and posture adjustment of large-deviation incoming components, greatly improving the scope of application of robot component plug-in and pull-out testing guided by visual perception, and has excellent flexibility, compatibility and accuracy. Through the coarse positioning of the 2D camera with the image space features as the tracking target, through real-time visual perception and adaptive closed-loop dynamic deviation adjustment, the robot can be accurately positioned in the image space, with strong stability and robustness. The centers of the dual-camera field of view coincide, and shooting can be performed at the same sampling position without having to drive the robot to move the camera to sample at multiple positions, which is conducive to improving the operating efficiency of the hand-eye system. In addition, the overall operating efficiency can be improved by further controlling the component plug-in and pull-out test system based on visual perception.

[0111] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0112] The present application is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the teachings of the present application, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope protected by the present application.

Claims

1. A component plug-in and pull-out test system based on visual perception, used for plug-in and pull-out testing of the socket of the component to be tested; characterized in that: The system comprises: A visual perception module, used for visually identifying the socket of the component to be tested, comprising a 2D camera and a 3D camera; wherein the 2D camera has a first field of view, the 3D camera has a second field of view, the first field of view includes the second field of view, and the first field of view range is larger than the second field of view range; The gripping device comprises a clamping end, the clamping end is used to clamp the test plug to perform plugging and unplugging operations on the jack; and the visual perception module is installed on the clamping end to move with the clamping end; A transport device, used for transporting the component to be tested; A host computer is communicatively connected with the visual perception module, the grasping device and the transport device.

2. A component extraction test system control method based on visual perception, characterized in that: The method is used to control the component plugging and unplugging test system based on visual perception as claimed in claim 1, and the method comprises: Acquire system preset data; the preset data includes an initial detection position of the visual perception module; the detection area corresponds to the detection position of the visual perception module; When the target socket of the target element is transported to the detection area by the transport device, controlling the 2D camera and the 3D camera to perform detection to determine whether the target socket exists within the first field of view and the second field of view; When the target socket does not exist within the second field of view and the target socket exists within the first field of view, the visual perception module is transported to the next detection position based on the position of the target socket, and the target socket is re-detected to see whether it is within the second field of view; When the target jack exists within the second field of view, determining, based on the current position of the target jack, the posture parameters of the plug-in test system when plugging in or out the target jack; An insertion and extraction test is performed on the target socket based on the posture parameter, and a next target socket of the target element is transported to the inspection area.

3. The component extraction test system control method based on visual perception according to claim 2 is characterized in that: The controlling of the 2D camera and the 3D camera for detection; Determining whether the target jack exists within the first field of view and within the second field of view includes: Controlling the 3D camera to perform detection to determine whether the target jack exists within the second field of view; When the target socket does not exist within the second field of view, the 2D camera is controlled to perform detection to determine whether the target socket exists within the first field of view.

4. The component extraction test system control method based on visual perception according to claim 2 is characterized in that: The method further comprises: transporting the visual perception module to a next detection position based on the location of the target socket, and re-detecting whether the target socket is located within the second field of view. If the target socket is not within the second field of view after multiple inspections, the next target socket of the target component is transported to the inspection area.

5. The component extraction test system control method based on visual perception according to claim 2 or 3, characterized in that: The preset data includes a 3D camera pre-sampling template image and a 2D camera pre-sampling template image; The controlling of the 2D camera and the 3D camera for detection; Determining whether the target jack exists within the first field of view and within the second field of view includes: Control the 2D camera and the 3D camera to perform sampling to obtain the current sampling map of the 2D camera and the current sampling map of the 3D camera; The 3D camera pre-sampling template map is compared with the 3D camera current sampling map, and the 2D camera pre-sampling template map is compared with the 2D camera current sampling map to determine whether the target jack exists within the first field of view and the second field of view.

6. The component extraction test system control method based on visual perception according to claim 5 is characterized in that: The preset data includes 3D camera pre-sampled point cloud data and set posture parameters of the gripping device when the teaching plug is inserted into the target socket; The determining, based on the current position of the target jack, actual posture parameters of the clamping end when the clamping end is inserted into the plug, comprises: Control the 3D camera to sample to obtain the current point cloud data of the 3D camera; Convert the current point cloud data of the 3D camera and the pre-sampled point cloud data of the 3D camera into the coordinate system of the gripping device for point cloud registration to obtain a relative pose matrix; Calculating calibration pose parameters according to the relative pose matrix and the set pose parameters; The actual posture parameters are calculated based on the calibration posture parameters and the position difference between the current detection position and the next detection position of the visual perception module.

7. The component extraction test system control method based on visual perception according to claim 5 is characterized in that: The re-detecting whether the target jack is located within the second field of view includes: Compare the 3D camera pre-sampling template image with the 3D camera current sampling image; When the image distance between the center point of the socket of the current sampling image of the 3D camera and the center point of the socket of the pre-sampling template image of the 3D camera is less than the set threshold, it is determined that the target socket is located within the second field of view.

8. The component extraction test system control method based on visual perception according to claim 5 is characterized in that: The preset data also includes visual dynamic guidance control gain parameters; The method of transporting the visual perception module to a next detection position based on the position of the target jack includes: Calculating the image position deviation between the 2D camera pre-sampling template image and the 2D camera current sampling image; According to the current detection position of the visual perception module, the image position deviation and the visual dynamic guidance control gain parameter, the next detection position of the visual perception module is determined, and the visual perception module is transported to the next detection position.

9. The component extraction test system control method based on visual perception according to claim 2, characterized in that: After performing the plug-in and pull-out test on the target jack based on the posture parameter, the method further includes: Record the number of plug-in and plug-out tests to the current time; If the number of plug-in and pull-out tests reaches the set value, the plug-in and pull-out tests on the target component are stopped.

10. A component extraction test control device based on visual perception, characterized in that: The device comprises: An acquisition module is used to acquire system preset data; the preset data includes an initial detection position of a visual perception module; A control module is used to control a 2D camera and a 3D camera to perform detection when a target socket of a target element is transported to a detection area by a transport device; to determine whether a target socket exists within a first field of view and a second field of view; the detection area corresponds to a detection position of a visual perception module; when a target socket does not exist within the second field of view and a target socket exists within the first field of view, the visual perception module is transported to a next detection position based on the position of the target socket, and re-detect whether the target socket is located within the second field of view; and to perform a plug-in test on the target socket based on the posture parameter, and transport the next target socket of the target element to the detection area; The processing module is used to determine the posture parameters of the plug-in and pull-out test system when plugging and unplugging the target jack based on the current position of the target jack when there is a target jack within the second field of view.

Images