A method and device for dynamic table-mounted device automatic identification, positioning and following grabbing

Abstract

The application provides a method and device for automatic identification, positioning and following grabbing of a dynamic surface-mounted device, and realizes grabbing of the surface-mounted device. The method comprises the following steps: receiving an image collected by an industrial camera, recording an image collection time, and processing the image to obtain a standardized image; inputting the standardized image into a target detection model to obtain all surface-mounted devices in the standardized image, and position and type information of each surface-mounted device, and then determining a target device to be grabbed; extracting an edge contour of the target device by using an image segmentation algorithm to obtain a center coordinate of the target device and a minimum circumscribed rectangle of the target device, and then obtaining an inclination angle of the target device; based on the above information, a mapping relationship between an image coordinate system and a global coordinate system of a robot is used to obtain a space-time coordinate of the target device; the above steps are repeated until the space-time coordinates of the target device at N time points are obtained, so as to fit a space-time motion model of the target device, and the robot can grab the target device.

Description

Technical Field

[0001] This application relates to the field of automatic testing technology for electronic components, specifically to a method and apparatus for automatic identification, positioning, and tracking of dynamic surface-mount devices. Background Technology

[0002] With the rapid iteration of electronic information technology, electronic products are developing towards miniaturization and integration. Surface mount components (SMTs) are widely used due to their small size, long lifespan, and high quality factor. However, the testing of SMTs has always been a major challenge. Traditional testing methods rely heavily on manual operation, which is not only time-consuming and labor-intensive but also susceptible to human factors. In modern large-scale production, this method cannot meet the demands for efficient, accurate, and low-cost production. Therefore, automated testing technology has gradually become a research hotspot in this field.

[0003] Current automated testing systems for electronic components mostly handle static components from fixed trays, and mechanical positioning largely relies on preset fixed trajectories and offline calibration, failing to meet the automated feeding requirements of moving components on production lines. In dynamic environments, the accuracy of traditional image recognition algorithms (such as template matching) drops significantly. Furthermore, surface-mount device features often rely on manual extraction, leading to large fluctuations in the classification accuracy of traditional machine learning models (such as SVM and HOG) under dynamic random disturbances on production lines, making rapid deployment across different device types difficult. Current mainstream solutions struggle to achieve a good balance between cost and robustness in dynamic scenarios.

[0004] Therefore, there is an urgent need to develop a solution that integrates dynamic perception and follow-up grasping control to overcome the limitations of existing technologies in dynamic surface mount device grasping scenarios and meet the development needs of intelligent automatic component testing technology. Summary of the Invention

[0005] This application provides a method and apparatus for automatic identification, positioning, and tracking of dynamic surface-mount devices. Based on deep neural networks for visual recognition and positioning, it can identify information of surface-mount devices moving on a conveyor belt and quickly output their pose. At the same time, it can accurately grasp the dynamic target device by following the trajectory of a robotic arm for subsequent electrical parameter testing.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides a method for automatic identification, positioning, and tracking of dynamic surface-mount devices, the method comprising:

[0008] S1. Receive images captured by an industrial camera, record the image acquisition time, and perform standardization processing on the images to obtain standardized images;

[0009] S2. Input the standardized image into the YOLOv8 target detection model to obtain all surface-mount devices in the standardized image, as well as the location and type information of each surface-mount device;

[0010] S3. Based on the location and type information of each surface-mount device, determine the target device to be captured;

[0011] S4. Using an image segmentation algorithm, extract the edge contour of the target device to obtain the center coordinates of the target device and the minimum bounding rectangle of the target device, and then obtain the tilt angle of the target device.

[0012] S5. Based on the target device's position information, tilt angle, and image acquisition time, the pose information of the target device in the robot's global coordinate system is obtained using the mapping relationship between the image coordinate system and the robot's global coordinate system. This leads to the determination of the target device's spatiotemporal coordinates v.

[0013] S6. Repeat steps S1-S5 above until the spatiotemporal coordinates of the target device at N time points are obtained, where N is a preset positive number greater than zero.

[0014] S7. Based on the spatiotemporal coordinates of the target device at N time points, fit the spatiotemporal motion model of the target device;

[0015] S8. Based on the spatiotemporal motion model of the target device, the robotic arm grasps the target device.

[0016] In a second aspect, an apparatus for automatically identifying, locating, and following dynamic surface-mount devices is provided, the apparatus comprising a module for performing the method of the first aspect described above.

[0017] In one possible design, the device for automatically identifying, locating, and following dynamic surface-mount devices in the third aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the device for automatically identifying, locating, and following dynamic surface-mount devices and other devices.

[0018] In one possible design, the device for automatic identification, positioning, and tracking of the dynamic surface-mount device in the third aspect may further include a memory. This memory may be integrated with the processor or disposed separately. The memory may be used to store the instructions involved in the method of the first aspect.

[0019] Thirdly, an apparatus for automatically identifying, locating, and following a dynamically surface-mount device is provided. The apparatus includes a processor coupled to a memory, the processor executing instructions stored in the memory to cause the apparatus to perform the method of the first aspect.

[0020] In one possible design, the device for automatic identification, positioning, and tracking of dynamically surface-mount devices in the fourth aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the device for automatic identification, positioning, and tracking of dynamically surface-mount devices in the fourth aspect and other devices.

[0021] Fourthly, an apparatus for automatically identifying, locating, and following a dynamically surface-mount device is provided, comprising: a processor and a memory; the memory is used to store instructions, which, when executed by the processor, cause the apparatus for automatically identifying, locating, and following a dynamically surface-mount device to perform the method of the first aspect.

[0022] In one possible design, the device for automatic identification, positioning, and tracking of dynamically surface-mount devices in the fifth aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used to communicate between the device for automatic identification, positioning, and tracking of dynamically surface-mount devices in the fifth aspect and other devices.

[0023] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program or instructions that, when the computer program or instructions are executed, cause the method of automatically identifying, locating, and following the dynamic surface-mount device of the first aspect to be executed.

[0024] The main advantages of this application are as follows: The visual recognition algorithm based on the YOLOv8 target detection model can quickly identify and accurately locate surface-mount electronic components moving on a conveyor belt without the need for manual feature design. It has a stronger adaptability to the complexity of the background and the deformation of the target device in the motion environment. The end-to-end learning method simplifies the visual recognition process and improves the recognition accuracy. The dynamic surface-mount device motion model construction based on the pure vision solution has better scalability in speed compared to the traditional method that requires calibration of the conveyor belt speed or encoder. It can adapt to more complex planar motion and direction, while reducing the impact of hardware system latency and time synchronization problems on the following and grasping accuracy. The use of segmented fixed-point following tasks to replace the solution of time-varying motion curves, thereby inversely calculating the grasping speed of the robot arm, helps to simplify the dynamic trajectory calculation process, improve the system response speed, and ensure the stability and safety of the grasping process.

[0025] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating the method for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in this application embodiment;

[0028] Figure 2 A schematic diagram illustrating the type information of each surface-mount device provided in the embodiments of this application;

[0029] Figure 3 A schematic diagram of the minimum circumscribed rectangle of the target device provided in the embodiments of this application;

[0030] Figure 4 A schematic diagram of a robotic arm grasping a target device provided in an embodiment of this application. Figure 1 ;

[0031] Figure 5 A schematic diagram of a robotic arm grasping a target device provided in an embodiment of this application. Figure 2 ;

[0032] Figure 6 A schematic diagram of the structure of the device for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in this application embodiment. Figure 1 ;

[0033] Figure 7 A schematic diagram of the structure of the device for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in this application embodiment. Figure 2 . Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0035] Figure 1 This is a flowchart illustrating the method for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in this application embodiment.

[0036] The process of automatic identification, positioning, and tracking of dynamic surface-mount devices is as follows:

[0037] Step S1: Receive images acquired by an industrial camera, record the acquisition time of the images, and perform standardization processing on the images to obtain standardized images.

[0038] Images of surface-mount devices being transported on a conveyor belt are captured using an industrial camera, and the capture time, such as t0, is recorded.

[0039] The image is standardized by performing the standardization process, and the specific steps are as follows:

[0040] Step S101: Use bilinear interpolation algorithm to adjust the image size to the size required by the YOLOv8 object detection model to obtain the first image. For example, if the subsequent image processing model requires the image to be 640×480, then the bilinear interpolation algorithm is used to calculate the new pixel value by considering the values ​​of neighboring pixels, thereby achieving smooth image scaling.

[0041] Step S102: Adjust the three-channel pixel values ​​of the first image to a preset range to obtain the second image; it can be understood that the preset range is the value obtained after training the YOLOv8 object detection model in the early stage. In this application, it is only necessary to adjust the three-channel pixel values ​​of the first image to the preset range.

[0042] Step S103: Adjust the format of the second image to the format required by the YOLOv8 object detection model to obtain a standardized image.

[0043] Step S2: Input the standardized image into the YOLOv8 target detection model to obtain all surface-mount devices in the standardized image, as well as the location and type information of each surface-mount device.

[0044] Specifically, this can be understood as:

[0045] Step S201: Input the standardized image into the YOLOv8 target detection model. The YOLOv8 target detection model outputs multiple predicted surface-mount devices and the confidence scores of the multiple predicted surface-mount devices.

[0046] Step S202: Based on the confidence levels of multiple predicted surface-mount devices and the preset confidence levels, obtain all surface-mount devices in the standardized image.

[0047] That is, the YOLOv8 target detection model outputs multiple predictions that could be surface-mount devices, along with their corresponding confidence levels. However, based on previous verification of the YOLOv8 target detection model, for example, when the confidence level is below 0.5, the predicted surface-mount device may be another device or other possible false detections, which will not be specifically considered in this application.

[0048] Furthermore, the aforementioned preset reliability of 0.5 is merely an example; the specific setting should be based on actual circumstances, and no restrictions are imposed here.

[0049] In step S203, the YOLOv8 object detection model also identifies the location information of each surface-mount device, which is the coordinates of the surface-mount device in the standardized image.

[0050] In step S204, the YOLOv8 target detection model also identifies the type information of each surface-mount device. The type information includes front, back, side, and overlapping. For details, please refer to [reference needed]. Figure 2 understand.

[0051] Step S3: Based on the location and type information of each surface-mount device, determine the target device to be captured.

[0052] Specifically, this can be understood as:

[0053] Step S301: Calculate the center point of each surface-mount device based on its position information.

[0054] It is understandable that, assuming each surface-mount device is rectangular or polygonal in shape, then the center point is the coordinate of the geometric center of that device.

[0055] Step S302: Calculate the outlier value of each surface-mount device based on the center point of each surface-mount device.

[0056] Calculating outliers for each surface-mount device typically employs statistical methods, most commonly based on the mean and standard deviation, the interquartile range (IQR), or distance metrics such as Euclidean distance. For details, please refer to existing technologies; they will not be elaborated upon here.

[0057] Step S303: Based on the outlier value of each surface-mount device, identify at least one isolated device.

[0058] Step S304: Based on the type information of each surface-mount device, determine the target device to be grabbed from at least one isolated device.

[0059] By analyzing the type information of each surface-mount device, we can determine whether it is front-facing, back-facing, side-facing, or overlapping, and then prioritize picking up the corresponding device. Figure 2 As shown, the device is positioned on its side, representing a non-grasping target, while the reverse side faces upwards, representing a graspable target.

[0060] Step S4: Extract the edge contour of the target device using an image segmentation algorithm to obtain the center coordinates of the target device and the minimum bounding rectangle of the target device, and then obtain the tilt angle of the target device.

[0061] For details, please refer to... Figure 3 The steps to understand are as follows:

[0062] Step S401: Extract the edge contour of the target device using an image segmentation algorithm to obtain the coordinate set of multiple points in the edge contour.

[0063] Suppose we have an image of a circuit board displaying multiple surface-mount components. Image segmentation algorithms (such as Canny edge detection) will extract the outer contour of each component, obtaining a set of edge contour coordinates for these components. The coordinates of each contour point may be a set of points on a two-dimensional plane, such as (x1, y1), (x2, y2), (x3, y3)...

[0064] Step S402: Calculate the center coordinates of the target device based on the coordinate set of multiple points.

[0065] The center coordinates of the device can be obtained by calculating the average position of (x1, y1), (x2, y2), (x3, y3), ...

[0066] Step S403: Based on the edge contour of the target device, obtain the minimum bounding rectangle of the target device, wherein the minimum bounding rectangle satisfies the condition of completely enclosing the target device and having the smallest area, and the minimum bounding matrix has a rotation angle.

[0067] The method for determining the minimum bounding rectangle of a specific target device is as follows: First, find a rectangle that includes all edge points. Then, try rotating the rectangle, gradually changing the rotation angle, until the minimum rectangle area is found.

[0068] Step S404: Based on the rotation angle of the minimum circumscribed matrix, obtain the rotation angle of the target device.

[0069] The rotation angle of the target device can be understood as the tilt angle between the minimum circumscribed matrix and the horizontal line.

[0070] Step S5: Based on the position information of the target device, the tilt angle of the target device and the image acquisition time, the pose information of the target device in the global coordinate system of the robot is obtained by using the mapping relationship between the image coordinate system and the robot's global coordinate system, and then the spatiotemporal coordinates of the target device are obtained.

[0071] Before describing step S5 in detail, it should be noted that before the execution of the method of this application, that is, before step S1, camera calibration is first performed, that is, the mapping relationship between the image coordinate system and the robot's global coordinate system is determined.

[0072] Because the camera is separate from the robotic arm in this application and fixed at a certain position outside the robotic arm, the camera's field of view includes the robotic arm's workspace. Therefore, an eye-to-hand calibration method is adopted. The specific steps are as follows:

[0073] Step 1: Use an industrial camera to take multiple images of the calibration board printed with a black and white checkerboard pattern;

[0074] Step 2: Using Harris corner detection and subpixel-level optimization operators, extract m corner points and record the image coordinates A = [P1, P2, ..., P...] corresponding to each of the m corner points. m Correspondingly, record the coordinates of each corner point in the robot's global coordinate system: B = [R1, R2, ..., R]. m ], where m is a positive integer greater than or equal to zero;

[0075] Step 3: Calculate the transformation matrix M using the transformation equation A = MB, and determine the mapping relationship between the image coordinate system and the robot's global coordinate system.

[0076] In the above scenario, the camera's field of view is within the robot's travel range. When there is no overlap between the field of view of the industrial camera and the travel range of the robot, it is also necessary to calculate the motion vector v of the conveyor belt and the actual distance l of the calibration plate moving from the field of view of the industrial camera to the travel range of the robot.

[0077] Let the coordinates of any point on the robotic arm be R. i (X iY i The coordinates of the same point within the field of view of the industrial camera are R′. i (X′ i ,Y′ i ), coordinates R i and coordinates R′ i The following modification relation R′ is satisfied. i (X′ i ,Y′ i ) = R i (X i Y i -vl;

[0078] Based on the correction relationship, the corrected coordinates of the robot in the global coordinate system are obtained, which facilitates the subsequent grasping operation of the robot.

[0079] In step S5, based on the position information (i.e., coordinate information (X0, Y0)) of the target device and the tilt angle R0 of the target device, the pose information (X′0, Y′0, R′0) of the target device in the global coordinate system of the robot is obtained by using the mapping relationship between the image coordinate system and the robot's global coordinate system.

[0080] Based on the image acquisition time, the spatiotemporal coordinates (X′0, Y′0, R′0, t0) of the target device are obtained.

[0081] Step S6: Repeat steps S1-S5 until the spatiotemporal coordinates of the target device at N time points are obtained, where N is a preset positive number greater than zero.

[0082] By setting an appropriate frame rate and acquiring multiple frames of images within the field of view of the industrial camera, the frame rate setting needs to balance the algorithm processing efficiency and the conveyor belt running speed. Repeating steps S1-S5 yields the spatiotemporal coordinates of the target device at N time points: (X′0, Y′0, R′0, t0), (X′1, Y′1, R′1, t1), (X′2, Y′2, R′2, t2), ..., (X′...). N ,Y′ N , R′ N , t N ).

[0083] Step S7: Based on the spatiotemporal coordinates of the target device at N time points, fit the spatiotemporal motion model of the target device.

[0084] Based on the spatiotemporal coordinates of the target device at N time points, a discrete trajectory is established, and the spatiotemporal motion model of the target device is fitted by first-order linear regression.

[0085] The spatiotemporal motion model of the target device is represented as f(x, y) = D(t), where D(t) represents the position of the target device at time t.

[0086] Step S8: Based on the spatiotemporal motion model of the target device, the robotic arm grasps the target device.

[0087] Based on the spatiotemporal motion model of the target device, the motion trajectory of the target device is predicted. Before the robot arm grasps the target device, in the first time period, the robot arm and the target device are relatively stationary in the horizontal direction, and the robot arm moves diagonally downward until it contacts the target device. In the second time period, the robot arm contacts the surface of the target device and follows the target device to complete a stable grasp. In the third time period, the robot arm grasps the target device and moves diagonally upward for a safe distance before rising steadily to a preset height. The first, second, and third time periods are preset.

[0088] For details, please refer to [link / reference]. Figure 4 and Figure 5 The steps to understand are as follows:

[0089] Step 1: After the model fitting calculation is completed, the robotic arm moves to the global trigger point and waits for the device to move into the grasping range;

[0090] Step 2: Align the robot controller with the visual algorithm service time axis. At time t0, execute the follow command. The robot follows the target device diagonally downwards at time t1, during which time it remains relatively stationary with respect to the target device in the horizontal direction.

[0091] Step 3: The robotic arm's end effector contacts the surface of the target device and executes the complete follow command, horizontally following the target device for time t2.

[0092] Step 4: The robotic arm's end-effector has completed a stable gripping action and follows the target device diagonally upwards for time t3. During this time, it remains relatively stationary with respect to the target device in the horizontal direction. Once the robotic arm reaches a safe height, the following action ends, and subsequent test loading instructions are executed.

[0093] It can be seen that the above steps conform to two principles: (1) the robotic gripping mechanism and the target device have an overlapping spatiotemporal trajectory to ensure stable gripping; (2) avoid dangerous interference, especially the collision between the gripping mechanism and the edge of the device tray. Figure 5 This is an incorrect scraping step, which does not comply with the two principles mentioned above.

[0094] In addition, based on the motion trajectory of the target device, the first time period, the second time period, and the third time period, the speed at which the robot arm grasps the target device is determined by inverse calculation, ensuring that the robot arm can adjust its motion state in a timely manner.

[0095] In addition, the speed at which the robotic arm grasps the target device can be determined by taking the median value of multiple target devices after grasping them multiple times, and no specific limit is imposed based on the actual situation.

[0096] It should also be noted that software communication latency and robotic arm response latency must be taken into account, and the following timestamp should be adjusted synchronously to obtain a stable dynamic following and grasping trajectory.

[0097] In summary, in the embodiments of this application, the visual recognition algorithm based on the YOLOv8 target detection model can quickly identify and accurately locate surface-mount electronic components moving on a conveyor belt. It eliminates the need for manual feature design, exhibits stronger adaptability to background complexity and target device deformation in motion environments, and simplifies the visual recognition process through end-to-end learning, thereby improving recognition accuracy. The dynamic surface-mount device motion model construction based on a pure vision solution offers better scalability in speed compared to traditional methods that require conveyor belt speed or encoder calibration. It can adapt to more complex planar motions and directions, while reducing the impact of hardware system latency and time synchronization issues on the tracking and grasping accuracy. The use of a segmented fixed-point following task to replace the solution of time-varying motion curves, thereby inversely calculating the robotic arm's grasping speed, simplifies the dynamic trajectory calculation process, improves system response speed, and ensures the stability and safety of the grasping process.

[0098] The above describes the method for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in the embodiments of this application. The following, in conjunction with... Figures 6-7 This document describes in detail the apparatus used to perform automatic identification, positioning, and tracking of dynamic surface-mount devices provided in the embodiments of this application.

[0099] Figure 6 This is a schematic diagram of the structure of the device for automatic identification, positioning, and tracking of dynamic surface-mount devices provided in this application embodiment. Figure 1 For example, such as Figure 6 As shown, the device 600 for automatic identification, positioning, and tracking of dynamic surface-mount devices includes a transceiver module 601 and a processing module 602. For ease of explanation, Figure 6 Only the main components of the device for automatic identification, positioning, and follow-up grasping of the dynamic surface-mount device are shown.

[0100] The transceiver module 601 is used to perform the transceiver function of the above-mentioned method for automatic identification, positioning and tracking of dynamic surface-mount devices, and the processing module 602 is used to perform other functions of the above-mentioned method for automatic identification, positioning and tracking of dynamic surface-mount devices besides the transceiver function.

[0101] Optionally, the transceiver module 601 may include a transmitting module ( Figure 6 (not shown in the image) and receiving module ( Figure 6 (Not shown in the image). The transmitting module is used to implement the transmitting function of the device 600 for automatic identification, positioning, and following of dynamically surface-mount devices, while the receiving module is used to implement the receiving function of the device 600 for automatic identification, positioning, and following of dynamically surface-mount devices.

[0102] Optionally, the device 600 for automatic identification, positioning, and tracking of dynamically surface-mount devices may further include a storage module. Figure 6 (Not shown in the image), the storage module stores programs or instructions. When the processing module 602 executes the program or instructions, the device 600 for automatically identifying, locating, and following the dynamic surface-mount device can perform the method for automatically identifying, locating, and following the dynamic surface-mount device in the embodiments of this application.

[0103] The following is combined with Figure 7 Each component of the device 700 for automatic identification, positioning, and tracking of dynamic surface-mount devices is described in detail:

[0104] The processor 701 is the control center of the device 700 for automatic identification, positioning, and tracking of dynamically surface-mount devices. It can be a single processor or a collective term for multiple processing elements. For example, the processor 701 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0105] Optionally, the processor 701 can execute various functions of the device 700 for automatic identification, positioning, and following of dynamic surface-mount devices by running or executing software programs stored in the memory 702 and calling data stored in the memory 702, such as executing the method for automatic identification, positioning, and following of dynamic surface-mount devices in the embodiments of this application.

[0106] In a specific implementation, as one example, the processor 701 may include one or more CPUs, for example... Figure 7 CPU0 and CPU1 are shown in the diagram.

[0107] In a specific implementation, as one example, the device 700 for automatic identification, positioning, and tracking of dynamically surface-mount devices may also include multiple processors, for example... Figure 7The processors 701 and 704 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The memory 702 is used to store the software program executing the scheme of this application, and its execution is controlled by the processor 701. Specific implementation methods can be found in the above method embodiments, and will not be repeated here.

[0108] Optionally, the memory 702 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 702 may be integrated with the processor 701 or exist independently, and its interface circuit for automatic identification, positioning, and following of the gripping device 700 via a dynamic surface-mount device (SMT) may also be used. Figure 7 (Not shown in the image) is coupled to the processor 701, and this embodiment does not specifically limit this.

[0109] Transceiver 703 is used for communication with other communication devices. For example, if the device 700 for automatic identification, positioning, and tracking of dynamic surface-mount devices is the first device, transceiver 703 can be used to communicate with a second device or a third device.

[0110] Optionally, transceiver 703 may include a receiver and a transmitter. Figure 7 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0111] Optionally, the transceiver 703 can be integrated with the processor 701 or exist independently, and its interface circuit for automatically identifying, locating, and following the grasping device 700 via a dynamic surface-mount device (SSD). Figure 7 (Not shown in the image) is coupled to the processor 701, and this embodiment does not specifically limit this.

[0112] Understandable, Figure 7 The structure of the device 700 for automatic identification, positioning and following of dynamic surface mount devices shown in the figure does not constitute a limitation on the device. The actual device for automatic identification, positioning and following of dynamic surface mount devices may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0113] Furthermore, the technical effects of the device 700 for automatic identification, positioning, and tracking of dynamic surface-mount devices can be referred to the technical effects of the method described in the above-described method embodiments, and will not be repeated here.

[0114] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0115] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0116] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

Claims

1. A method for automatic identification, positioning, and tracking of dynamic surface-mount devices, characterized in that, The method includes: S1. Receive images captured by an industrial camera, record the acquisition time of the images, and perform standardization processing on the images to obtain standardized images; S2. Input the standardized image into the YOLOv8 target detection model to obtain all surface-mount devices in the standardized image, as well as the position and type information of each surface-mount device; S3. Based on the location and type information of each surface-mount device, determine the target device to be captured; S4. Using an image segmentation algorithm, extract the edge contour of the target device to obtain the center coordinates of the target device and the minimum bounding rectangle of the target device, and then obtain the tilt angle of the target device. S5. Based on the position information of the target device, the tilt angle of the target device and the acquisition time of the image, the pose information of the target device in the global coordinate system of the robot is obtained by using the mapping relationship between the image coordinate system and the global coordinate system of the robot, and then the spatiotemporal coordinates of the target device are obtained. S6. Repeat steps S1-S5 above until the spatiotemporal coordinates of the target device at N time points are obtained, where N is a preset positive number greater than zero. S7. Based on the spatiotemporal coordinates of the target device at N time points, fit the spatiotemporal motion model of the target device; S8. Based on the spatiotemporal motion model of the target device, the robotic arm grasps the target device.

2. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, The standardization process for the image to obtain a standardized image includes: The image size is adjusted to the size required by the YOLOv8 object detection model using bilinear interpolation algorithm to obtain the first image; The three-channel pixel values ​​of the first image are adjusted to a preset range to obtain the second image; The format of the second image is adjusted to the format required by the YOLOv8 object detection model to obtain a standardized image.

3. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, The process involves inputting the standardized image into the YOLOv8 target detection model to obtain all surface-mount devices in the standardized image, as well as the location and type information of each surface-mount device, including: The standardized image is input into the YOLOv8 object detection model, which outputs multiple predicted surface-mount devices and the confidence scores of the multiple predicted surface-mount devices. Based on the confidence levels of the multiple predicted surface-mount devices and the preset confidence levels, all surface-mount devices in the standardized image are obtained. The YOLOv8 target detection model also identifies the location information of each surface-mount device, which is the coordinate of the surface-mount device in the standardized image; The YOLOv8 target detection model also identifies the type information of each surface-mount device, including front, back, side, and overlapping.

4. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1 or 3, characterized in that, The step of determining the target device to be captured based on the location and type information of each surface-mount device includes: Based on the position information of each surface-mount device, the center point of each surface-mount device is calculated; Based on the center point of each surface-mount device, the outlier value of each surface-mount device is calculated; Based on the outlier value of each surface-mount device, at least one isolated device is identified; Based on the type information of each surface-mount device, the target device to be captured is determined from the at least one isolated device.

5. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, The process involves extracting the edge contour of the target device using an image segmentation algorithm to obtain the center coordinates and minimum bounding rectangle of the target device, and then determining the tilt angle of the target device. This includes: The edge contour of the target device is extracted using an image segmentation algorithm to obtain a set of coordinates for multiple points in the edge contour; Based on the coordinate set of the multiple points, the center coordinates of the target device are calculated; Based on the edge contour of the target device, the minimum bounding rectangle of the target device is obtained, wherein the minimum bounding rectangle satisfies the condition of completely enclosing the target device and having the smallest area, and the minimum bounding rectangle has a rotation angle; The rotation angle of the target device is obtained based on the rotation angle of the minimum circumscribed matrix.

6. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, In step S5, the mapping relationship between the image coordinate system and the robot's global coordinate system is derived in advance. The derivation steps include: Step 1: Use an industrial camera to take multiple images of the calibration board printed with a black and white checkerboard pattern; Step 2: Using Harris corner detection and subpixel-level optimization operators, extract m corner points and record the image coordinates A = [P1, P2, ..., P] corresponding to the m corner points. m Correspondingly, record the coordinates of each corner point in the robot's global coordinate system: B = [R1, R2, ..., R]. m ], where m is a positive integer greater than or equal to zero; Step 3: Calculate the transformation matrix M using the transformation equation A = MB, and determine the mapping relationship between the image coordinate system and the robot's global coordinate system.

7. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 6, characterized in that, In step S5, the mapping relationship between the image coordinate system and the robot's global coordinate system is derived in advance. The derivation step further includes: When the field of view of the industrial camera and the travel range of the robot do not overlap, it is also necessary to calculate the motion vector v of the conveyor belt and the actual distance l of the calibration plate moving from the field of view of the industrial camera to the travel range of the robot. Let the coordinates of any point on the robotic arm be R. i (X i Y i The coordinates of the same point within the field of view of the industrial camera are R′. i (X′ i Y i ′), the coordinates R i and the coordinates R′ i The following modification relation R′ is satisfied. i (X′ i ,Y′ i ) = R i (X i Y i -vl; Based on the aforementioned correction relationship, the corrected coordinates of the robot in the global coordinate system are obtained.

8. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, The step of fitting a spatiotemporal motion model of the target device based on its spatiotemporal coordinates at N time points includes: Based on the spatiotemporal coordinates of the target device at N time points, a discrete trajectory is established, and the spatiotemporal motion model of the target device is fitted by first-order linear regression. The spatiotemporal motion model of the target device is represented as f(x, y) = D(t), where D(t) represents the position of the target device at time t.

9. The method for automatic identification, positioning, and tracking of dynamic surface-mount devices according to claim 1, characterized in that, The robotic arm grasping the target device, based on the spatiotemporal motion model of the target device, includes: Based on the spatiotemporal motion model of the target device, the motion trajectory of the target device is predicted, such that before the robotic arm grasps it, in the first time period, the robotic arm and the target device are relatively stationary in the horizontal direction, and the robotic arm moves diagonally downward until it contacts the target device; in the second time period, the robotic arm contacts the surface of the target device, and the robotic arm follows the target device to complete a stable grasp; in the third time period, the robotic arm grasps the target device and moves diagonally upward for a safe distance, and then rises stably to a preset height, wherein the first time period, the second time period, and the third time period are preset; Based on the motion trajectory of the target device, the first time period, the second time period, and the third time period, the speed at which the robotic arm grasps the target device is determined.

10. A device for automatic identification, positioning, and tracking of dynamic surface-mount devices, characterized in that, The apparatus includes a module for performing the method as described in any one of claims 1-9.

Images