Intelligent detection and classification method and system for optocoupler packaging defects

Abstract

The invention discloses an optical coupler packaging defect intelligent detection and classification method and system, and relates to the technical field of optical coupler packaging. The method comprises the steps that S1, a camera array collects a first image; s2, preliminarily identifying the devices based on the second image to output a preliminary identification detection result, directly sorting the preliminarily abnormal devices, packaging the first image of the preliminarily normal device and the corresponding device identification code into a depth identification instruction, and sending the depth identification instruction to the step S3; s3, performing depth identification based on the first image, outputting a depth identification detection result, and controlling a manipulator to sort the corresponding devices to a preliminary abnormal sorting area, a depth abnormal sorting area, a qualified product sorting area or a manual reinspection area; in the process, the length of the conveying belt from the inlet to the sorting position guarantees real-time buffering between the detection result and the sorting action of the manipulator; a one-stop automatic process from visual identification to mechanical sorting is formed, and the automation degree and the overall efficiency of a production line are improved.

Description

Technical Field

[0001] The present application relates to the field of optocoupler packaging technology, and more specifically, to an optocoupler packaging defect intelligent detection and classification method and system. Background Art

[0002] An optocoupler (or optoisolator) is an electronic component that converts electrical signals into optical signals and then back into electrical signals. Its core function is to achieve electrical isolation. Packaging is a key link in protecting the fragile internal structure of the optocoupler. Packaging defects can cause the isolation layer to break down, causing high voltage to flow into the low-voltage terminal, resulting in fatal accidents or equipment burns. Poor packaging can accelerate the internal LED light decay (brightness drop) and shorten the device life. From home appliances to spacecraft, all scenarios involving high and low voltage interaction rely on optocouplers. Therefore, optocoupler packaging defect detection is very necessary.

[0003] During the defect inspection process for optocoupler packages, the resolution is increased to improve accuracy. However, this significantly increases image transmission, preprocessing, and inference latency (the pixel count increases 16-fold from 1K to 4K, and the image transmission bandwidth requirement increases exponentially). This makes it difficult to meet the real-time inspection requirements of current high-speed production lines.

[0004] In order to solve the above defects, an intelligent detection and classification method for optocoupler package defects is provided. Summary of the Invention

[0005] According to one aspect of the present application, a method for intelligent detection and classification of optical coupler package defects is provided, the method comprising the following steps:

[0006] S1: At the entrance of the defect inspection line, a camera array is used to capture first images of the same device on different designated surfaces and store them. The first images are downsampled to generate second images, which are then sent to step S2. Defect identification for a device includes several subtasks, with the first or second image of a designated surface being a subtask.

[0007] S2, based on the second image, performs preliminary identification on the device to output a preliminary identification detection result, the preliminary identification detection result including preliminary abnormality and preliminary normality, sends the preliminary identification detection result of the preliminary abnormality to S4, and retrieves the first image of the preliminary normal device and its corresponding device identification code, encapsulates it into a deep identification instruction, and sends it to step S3;

[0008] S3: Perform deep recognition on the device based on the second image and output a deep recognition detection result. The deep recognition detection result includes deep abnormality, manual re-inspection, and qualified products. The execution process adopts a thread parallel approach; the thread pool load status is determined. If the thread pool is in a low load state, the dynamic and hard association random allocation principle is implemented; otherwise, the scheduling allocation principle is implemented;

[0009] S4, based on the received preliminary identification test results and deep identification test results, the device identification code is parsed and the robot is controlled to sort the corresponding devices to the "preliminary abnormality sorting area", "deep abnormality sorting area" or "qualified product sorting area" or "manual re-inspection area"; in this process, the length of the conveyor belt from the entrance to the sorting point ensures real-time buffering between the detection results and the robot's sorting action.

[0010] Optionally, the initial identification process is:

[0011] Step 1: pre-process the second image to obtain a binary edge map;

[0012] Step 2: Identify all connected domains in the binary edge map, calculate the contour area and corresponding convex hull area of ​​each connected domain, and obtain the Solidity value of each connected domain. From this, we can obtain the Solidity value of each connected domain in each second image of the same device. If there is any connected domain with a Solidity value less than the solid threshold, the device is judged to be "preliminary abnormal"; otherwise, proceed to the next step;

[0013] Step 3: Extract the average grayscale value of the minimum enclosing rectangular area of ​​each connected domain and the average grayscale value of the annular background area outside it from the binary edge map, and calculate the contrast value. This can be used to obtain the solidity value of each connected domain in the second image of the same device. If any connected domain has a contrast value greater than the contrast threshold, it is determined to be a "preliminary anomaly"; otherwise, proceed to the next step.

[0014] Step 4: Calculate the ratio of the area of ​​each connected domain in the binary edge map to the area of ​​the entire image, and compare it with the preset area threshold; thereby, the Area value of each connected domain in each second image of the same device can be obtained. If there is any connected domain with an Area value greater than the area threshold, it is judged as "preliminary abnormality"; otherwise, the device is judged as "preliminary normal".

[0015] Optional, deep recognition process:

[0016] Step 1: parse the device identification code and several first images from the depth recognition instruction, and perform hard association on the second images with the same identification code;

[0017] Step 2: Extract the CPU occupancy of each thread. When the CPU occupancy is ≤0.3, it indicates that the thread is in a low-load state. If the low-load threads occupy more than half of all threads in the thread pool, the thread pool is judged to be in a low-load state, and the hard-association random allocation principle is activated. Specifically, the first image with a hard association is randomly assigned to the threads in the thread pool. The first image with a hard association relationship cannot be assigned to the same thread at the same time; otherwise, the scheduling allocation principle is activated.

[0018] Step 3: The depth recognition performed by each thread is:

[0019] The first image is converted and normalized, and each pixel channel is standardized to complete the input tensor construction. The first image is detected for defects using a trained deep convolutional neural network. Each type of defect has a corresponding characteristic pattern and confidence level. A list of defect detection results is output for each first image, including the defect type and the confidence level corresponding to each type of defect. The maximum confidence level is recorded as softmax. The inference time of each subtask completed by the thread is recorded.

[0020] Optionally, the scheduling allocation principle is:

[0021] 1-6. Collect thread performance indicators, including CPU usage, number of currently queued tasks, and average inference time within the past scheduled time. These indicators are recorded as A, Q, and T respectively. Traverse the "tasks to be assigned" and calculate a scheduling priority score for each thread. Sort the threads in the thread pool by the scheduling priority score to obtain a thread list, and update the thread list in real time. The calculation formula is: Where Amax is the maximum acceptable CPU usage; Qmax is the maximum acceptable threshold for queue length; Tmax is the worst acceptable average delay threshold;

[0022] 1-7: From all the subtasks to be assigned, the time they enter the thread pool is recorded as the entry time. Each subtask is classified according to its corresponding hard association. If all subtasks of the same device are performing deep recognition, the subtask with the earliest entry time is selected and given the first priority, and the others are given the second priority.

[0023] 1-8, the first-priority subtasks are sorted by entry time, the first subtask is selected and assigned to the first thread in the thread list, and this cycle repeats until all the first-priority subtasks are assigned;

[0024] 1-9, if the sofmax of a subtask ∈ (0.5, 0.7), all other subtasks hard-associated with the subtask are promoted to the first priority; if the sofmax of a subtask is ≤ 0.5, the subtask with the earliest entry time of the subtask hard-associated with the subtask is extracted from the second priority and promoted to the first priority; if the sofmax of a subtask is ≥ 0.7, the depth recognition of other subtasks hard-associated with the subtask is directly interrupted and the depth recognition detection result of "depth abnormality" is output; each time the thread completes the reasoning of a subtask, its reasoning time is recorded;

[0025] If the sofmax of each subtask in the same identification number is ∈ (0.5, 0.7), the deep recognition detection result of "manual re-inspection" is output, and the defect list corresponding to each subtask is output as auxiliary information; if the sofmax of each subtask in the same identification number is ≤ 0.5, the deep recognition detection result of "qualified product" is output, and the stored first image is cleaned.

[0026] Optionally, an 8-connected domain algorithm is used to determine the connected domain. The convex hull calculation is performed by forming a minimum convex polygon by constructing a convex hull for the connected domain boundary point set to obtain the convex hull area of ​​the connected domain. The minimum enclosing rectangle is an axisymmetric horizontal-vertical rectangle, and angle rotation is not considered.

[0027] In the first step of step S2, if there is a reflection or shadow area on the plastic package surface in the second image, a local adaptive histogram equalization operation is further performed on the image to avoid excessive stretching.

[0028] Optionally, the camera array captures images in the following process:

[0029] There are several cameras with adjustable resolutions on the defect detection line. Each camera is responsible for photographing the specified surface of the device to obtain the first image of the same device on different specified surfaces. Each device corresponds to each identification code. The robot controls the posture of the optocoupler packaging device to the specified posture by grabbing the device. The identification code of the device is in a fixed position and direction, and the specified surface is relative to the identification code. Therefore, the position of the identification code is fixed, and the position of each specified surface of the device is also uniformly fixed.

[0030] According to one aspect of the present application, there is provided an intelligent detection and classification system for optical coupler package defects, the system comprising: a camera array, an entry control terminal, a background processing terminal, and a screening control terminal;

[0031] There are several cameras with adjustable resolutions on the defect detection line. Each camera is responsible for photographing a specified surface of the device to obtain the first image of the same device on different specified surfaces and send it to the entry control terminal;

[0032] The entry control terminal has a built-in preliminary identification module and a node storage module. The first image is stored in the node storage module. The preliminary identification module performs preliminary identification on the device based on the second image to output a preliminary identification detection result. The preliminary identification detection result includes preliminary abnormality and preliminary normality. The preliminary identification detection result of the preliminary abnormality is sent to the screening control terminal, and the first image of the preliminary normal device and its corresponding device identification code are retrieved from the node storage module, encapsulated as a deep identification instruction, and sent to the background processing terminal;

[0033] The backend processing end has several built-in threads forming a thread pool. Each thread executes deep recognition in parallel and outputs deep recognition detection results. The deep recognition detection results include deep anomalies, manual re-inspections, and qualified products. The execution process adopts a thread parallel approach. The thread pool load status is judged. If the thread pool is in a low load state, the dynamic and hard association random allocation principle is implemented; otherwise, the scheduling allocation principle is implemented.

[0034] The screening control end parses the device identification code based on the received preliminary identification test results and deep identification test results, and controls the robot to sort the corresponding devices to the "preliminary abnormality sorting area", "deep abnormality sorting area" or "qualified product sorting area" or "manual re-inspection area"; in this process, the length of the conveyor belt from the entrance to the sorting point ensures real-time buffering between the test results and the robot's sorting action.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) This application uses a high-resolution camera array to accurately capture each designated surface of the optocoupler package device, and under the fixed identification code positioning of each device, the consistency and reliability of high-resolution image acquisition are guaranteed: the robot puts the device into a predefined posture by flipping it, and all surfaces can be completely covered by the corresponding 4K camera, whether it is fine cracks, reflections in bubbles or foreign matter on the surface, they can all be clearly presented; at the same time, the original 4K image is downsampled to a 1K low-resolution image and transmitted to the preliminary recognition module, which not only retains the geometric and grayscale information sufficient for coarse screening, but also greatly reduces the data transmission and computing overhead during the initial screening; more importantly, all 4K images are uploaded to the node storage end according to the unique identification code, providing a complete image archive for subsequent deep recognition, manual review or quality inspection backtracking;

[0037] (2) This application generates a high-quality binary edge map through pre-processing methods such as grayscale conversion, histogram equalization, filtering noise reduction and edge enhancement, which provides a solid foundation for the subsequent calculation of geometric and grayscale features based on connected domains; Solidity, Contrast and Area are used for hierarchical screening in sequence, and "obvious defects" are judged as preliminary abnormalities at the millisecond level and immediately fed back to the screening control end, ensuring that these most critical and most likely to cause failure defective components are eliminated as soon as possible; and for the remaining "preliminary normal" samples, only the corresponding high-resolution 4K images need to be uploaded for more accurate deep recognition, thereby truly achieving the perfect combination of "coarse first, fine later, fast first, precise later", which not only ensures the high throughput of the pipeline, but also greatly reduces the total sample size of deep reasoning, allowing the overall system to filter out most obvious defects at the gateway link, reducing the waste of back-end resources;

[0038] (3) This application conducts parallel reasoning on the same "deep recognition batch" (4K images of each surface of the same device): once the thread pool is in a low-load state, the "hard association + random allocation" principle is adopted. This pure parallel method maximizes the use of hardware resources and reduces the single-piece deep recognition latency; if the thread pool is busy, the startup scheduling allocation principle is executed. While taking into account absolute accuracy, the worst-case latency is strictly controlled within an acceptable range through parallelism and early stopping mechanisms. In addition, when resources are tight, tasks can be dynamically allocated to achieve load balancing, achieving both "high efficiency" and "high precision";

[0039] To sum up, the entire intelligent detection and classification system of this application forms a one-stop automated process from visual recognition to mechanical sorting, realizing seamless connection of "detection results → physical sorting actions", greatly improving the automation level, classification accuracy and overall efficiency of the production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0041] Figure 1 It is a flowchart of the method of the present invention;

[0042] Figure 2 It is a system connection block diagram of the present invention;

[0043] Figure 3 Schematic diagram of the preliminary identification and detection process of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0046] It should be noted that the terms "first," "second," and the like in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate for the embodiments of the present application described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatus.

[0047] like Figure 1-2 As shown, an embodiment of the present application provides an intelligent detection and classification method for optical coupler package defects, which is implemented based on an intelligent detection and classification system for optical coupler package defects, the system including a high-resolution camera array, an entry control terminal, a background processing terminal, and a screening control terminal;

[0048] The following steps are involved:

[0049] S1: A high-resolution camera array captures a comprehensive image of the optocoupler package, specifically:

[0050] There are several adjustable high-resolution cameras on the defect detection line. Each high-resolution camera is responsible for photographing the specified surface of the optocoupler package device. Each optocoupler package device corresponds to each identification code. The robot controls the specified surface of the optocoupler package device by grabbing the optocoupler package device, and aligns the specified surface with the corresponding high-resolution camera to obtain the images of each specified surface of the optocoupler package device. The images of each specified surface directly obtained by the high-resolution camera here are high-resolution images, and they are down-sampled to generate low-resolution images of each specified surface, which are sent to the preliminary recognition module; at the same time, the high-resolution images of each specified surface are coded according to the identification code of the optocoupler package device. The code is uploaded to the node storage end for storage one by one; for example, the optocoupler packaging device can be approximately regarded as a cuboid and divided into six faces. Each camera is set to correspond to a specified face by default. The identification code of the optocoupler packaging device is in a fixed position and direction. The specified face is relative to the identification code. Therefore, the identification code position is fixed, and the position of each specified face of each optocoupler packaging device is also unified; the robot flips the optocoupler packaging device to place the optocoupler packaging device in a specified posture, so that high-resolution cameras at different positions on the defect detection line can collect images of the specified face of the optocoupler packaging device. The specified face image here is high-resolution (the high resolution is set to 4k in this application scenario);

[0051] S2: The preliminary recognition module performs preliminary recognition based on the low-resolution images of each designated surface of the optocoupler dispenser. If the preliminary detection result is preliminary abnormal, the identification code of the optocoupler package device is extracted and sent to the screening control end. It should be noted that according to the actual application statistics of technicians on the production line, the defects of the optocoupler package devices in the preliminary abnormal category are more obvious. If the preliminary detection result is preliminary normal, the high-resolution images of the optocoupler package device with this identification code on each designated surface are extracted and packaged as a deep recognition instruction, and sent to the background processing end for deep recognition.

[0052] like Figure 3 As shown, the steps for preliminary identification are:

[0053] Step 1: Preprocessing of low-resolution images of each specified surface:

[0054] Convert a low-resolution image of a specified surface into a grayscale image, then perform histogram equalization to improve global contrast. Use Gaussian filtering or median filtering for denoising. Finally, use the Sobel or Canny algorithm for edge enhancement to highlight edges with potential defects, resulting in a binary edge map. It should be noted that the goal is to make the grayscale difference between "light bubbles" and "dark cracks" and the surrounding plastic packaging background more obvious, improving the robustness of the contrast calculation. If the plastic packaging surface is severely reflective or shadowed, local adaptive histogram equalization can also be performed here to avoid excessive local brightness stretching.

[0055] Step 2: Solidity value calculation and judgment:

[0056] After edge enhancement (Canny or Sobel), only "edges with pixel values ​​of 255" and "backgrounds with pixel values ​​of 0" remain in the binary edge map. We want to "aggregate" these scattered edge pixels into connected boundary curves (contours), that is, extract continuously connected 255 pixel points and organize them into a set of points (x, y) in a certain order. Each such point set is the boundary of a "closed or approximately closed" connected area in the map. This set of points can be used to calculate geometric features such as contour area and convex hull. In the binary edge map, all foreground pixels (value 255) that are connected to each other "up, down, left, right (4-connected) or diagonally (8-connected)" will be regarded as a whole area, called a "connected domain", from which each connected domain in the binary edge map and its corresponding connected domain boundary (a set of ordered point sets) can be obtained; calculate the contour area of ​​each connected domain in the binary edge map, and connect this connected domain to the boundary of the connected domain. The connected boundary of the domain (all points of the contour) is constructed as a convex hull to obtain a convex polygon. Its area is then calculated to obtain the convex hull area of ​​this connected domain. The contour area is divided by the convex hull area to obtain the Solidity value. All connected domains in the binary edge map are traversed and their corresponding Solidity values ​​are obtained. A solid threshold is set (the technician sets it to 0.82, which is the technician's experience and can be manually fine-tuned by the technician). The Solidity value of each connected domain is compared with the solid threshold one by one. If any Solidity value is less than the solid threshold, it means that the connected domain is "very concave and not full in shape" and has typical defect characteristics such as cracks / fractures. The identification code of the optocoupler package device is extracted and packaged as a preliminary identification test result of "preliminary abnormality". If the Solidity value is greater than or equal to the solid threshold, it is considered that the shape is relatively full and needs to continue to the lower layer judgment, and step three is executed.

[0057] Step 3: Calculate and judge the contrast value:

[0058] Local grayscale contrast is used to describe the absolute value of the grayscale difference between a connected domain and its surrounding background. If the difference is too large, it means that the area is very "abrupt" in appearance, which usually corresponds to phenomena such as "internal reflection of bubbles" or "plastic debonding". At a low resolution of 1K, "bubbles" may appear to be brighter inside than around, or darker after local debonding of the plastic. These phenomena can be quickly distinguished by calculating the Contrast value; the minimum enclosing rectangle of each connected area is extracted. The minimum enclosing rectangle refers to the minimum area axisymmetric rectangular frame that just encloses the connected area. Its boundary is horizontal or vertical, and the angle rotation and angle tilt are not considered. The grayscale average value is calculated for all pixels in the minimum enclosing rectangle area to obtain the grayscale average value, which is used to represent the brightness characteristics of the connected area corresponding to the minimum enclosing rectangle; a larger rectangle is constructed by expanding a certain pixel width in four directions (up, down, left, and right) along the minimum enclosing rectangle area, and Eliminate the part about the minimum enclosing rectangular area to obtain the annular background area. This annular area contains background pixels adjacent to the candidate area, which usually represents a normal area. The grayscale values ​​of all pixels in the annular background area are averaged to obtain the background brightness value. Then, the grayscale average value of the minimum enclosing rectangular area minus the corresponding background brightness value is subtracted and the absolute value is taken to obtain the contrast value of the minimum enclosing rectangular area corresponding to the connected area. Traverse each connected area in the binary edge image to obtain each contrast value, set a contrast threshold (technicians set it to 30, this value is an empirical value and can be manually fine-tuned by technicians). If any contrast value is greater than the contrast threshold, it is considered that the connected area has an obvious grayscale mutation and may be a defective area. Then, the identification code of the optocoupler package device is extracted and encapsulated as a preliminary recognition detection result of "preliminary abnormality". Otherwise, continue with the lower layer judgment and execute step 4.

[0059] Step 4: Extract the binary edge image and divide the area of ​​each connected region therein by the area of ​​the entire binary edge image to obtain the area value of each connected region. Set an area threshold (0.3, this is an empirical value and can be manually fine-tuned by technicians). Traverse each connected region in the binary edge image to obtain the area value of each connected region. If any area value is greater than the area threshold, it is very likely to be a "large-area plastic seal crack" or "residual glue / debonding area". In this case, the identification code of the optocoupler package device is extracted and packaged as a preliminary identification test result of "preliminary abnormality". Otherwise, the preliminary identification test result of "preliminary normality" is output.

[0060] By using a high-resolution camera to capture 4K images and downsampling them to 1K for preliminary identification, the algorithm retains rough defect information while significantly reducing image processing and model inference overhead. Three simple and efficient geometric and grayscale features (Solidity, Contrast, and Area) are used to quickly remove "obvious large-area defects" or "defects with concave shapes or sudden grayscale changes." The algorithm's contour extraction, convex hull, ROI averaging, and ring averaging steps are computationally low and can be completed in milliseconds.

[0061] S3: There is a thread pool consisting of several parallel threads. Each thread executes the depth recognition process to quickly output the depth recognition detection results. Specifically:

[0062] Step 1: parse the identification code of the optocoupler package device (the product identification code used on this production line is a barcode, one object and one code, and the identification code of each product is unique and certain) and the high-resolution image of each specified surface from the depth recognition instruction, and associate each high resolution with the identification code of the corresponding optocoupler package device. It should be noted that a complete "depth recognition batch" contains high-resolution images of specified surfaces of the optocoupler package device, and the hard association between them is the same identification code. Each high-resolution image of the specified surface is a depth recognition task; thus, the depth recognition of an optocoupler package device is divided into several depth recognition tasks, the number of which is consistent with the number of optocoupler package devices and the set specified surfaces, and the total number of threads in the thread pool is ≥ the number of specified surfaces;

[0063] Step 2: Extract the CPU occupancy of each thread. When the CPU occupancy is ≤0.3, it indicates that the thread is in a low-load state. If the low-load threads occupy more than half of all threads in the thread pool, the thread pool is judged to be in a low-load state, and the hard-association random allocation principle is activated. Specifically, each hard-association deep recognition task is randomly assigned to the threads in the thread pool. Hard-association deep recognition tasks cannot be assigned to the same thread at the same time to avoid the same thread sequentially processing multiple images of the same device, which wastes parallel resources.

[0064] Deep recognition process: Scale the pixel values ​​of the high-resolution image from [0,255] to [0.0,1.0] to complete the data type conversion and normalization, and then standardize each pixel channel (using ImageNet specifications) to complete the input tensor construction. After completing the preprocessing of each specified surface of the optocoupler package, the model inference can be performed; use the trained deep convolutional neural network to detect defects in high-resolution images. The detection targets include cracks, bubbles, dents, dirt, package deviation and surface foreign matter. Each type of defect has a corresponding feature pattern and confidence level. The high-resolution image outputs a defect detection result list, which includes the defect type and the confidence level corresponding to each type of defect, and the maximum confidence level is recorded as sofmax; if any sofmax of each specified surface in a complete "depth recognition batch" is ≥ 0.7, the depth recognition of other specified surfaces will be directly canceled, and the depth recognition detection result of "depth abnormality" will be output; if the sofmax of each specified surface in a complete "depth recognition batch" is less than or equal to 0.5, the depth recognition detection result of "qualified product" will be output; in other cases, the depth recognition detection result of "manual re-inspection" will be output;

[0065] Otherwise, the scheduling allocation principle is activated, specifically:

[0066] 1-1: Collect the following indicators of each thread in the thread pool in real time: CPU usage, number of currently queued tasks, average inference time within the past scheduled time (technicians usually set it to 30 seconds initially), and record them as A, Q, and T respectively; traverse the "tasks to be assigned" and calculate a scheduling priority score for each thread. Sort the threads in the thread pool according to the scheduling priority score to obtain a thread list, and update the thread list in real time; the calculation formula is: Where Amax is the maximum acceptable CPU usage; Qmax is the maximum acceptable threshold for queue length; Tmax is the worst acceptable average delay threshold;

[0067] 1-2, among all the tasks to be assigned, the time when they enter the thread pool is recorded as the entry time, and all the tasks to be assigned are regarded as an element, and each element contains a triplet recorded as (τk,i,Rk ,i ,Bk), where τk,i represents the depth task of the i-th surface of the k-th device, Rk ,i It is the time when the task enters the queue, B k This is the "deep identification batch" to which the task belongs. All tasks to be assigned are sorted according to the "deep identification batch" to which they belong. k Grouping, in batch k (i.e. device k), find the one with the smallest timestamp when entering the queue, and set the earliest arriving subtask in all batches as the first priority, and the others as the second priority;

[0068] 1-3, sort the first-priority subtasks by their entry time, select the first subtask and assign it to the first thread in the thread list, and repeat this cycle to assign all the first-priority subtasks;

[0069] 1-4: Result fusion judgment output:

[0070] If the sofmax of a subtask is ∈ (0.5, 0.7), all other subtasks that are hard-associated with the subtask are promoted to the first priority level. If the sofmax of a subtask is ≤ 0.5, the subtask with the earliest entry time that is hard-associated with the subtask is extracted from the second priority level and promoted to the first priority level. If the sofmax of a subtask is ≥ 0.7, the depth recognition of other subtasks that are hard-associated with the subtask is directly canceled, and a depth recognition detection result of "depth anomaly" is output.

[0071] If the sofmax of each subtask in the same deep recognition batch is ∈ (0.5, 0.7), the deep recognition detection result of "manual re-inspection" is output, and the defect list corresponding to each subtask is output as auxiliary information to help manual rapid re-inspection; if the sofmax of each subtask in the same deep recognition batch is ≤ 0.5, the deep recognition detection result of "qualified products" is output, and an image cleaning instruction is generated and fed back to the storage module at the node entry control end to guide it to clean the high-resolution image of the qualified products;

[0072] By performing full 4K model inference on only a small number of devices in the "preliminary normal" sample set (which often have more subtle and smaller defects), combined with a parallel thread pool and early stopping mechanism, we achieve both high accuracy and acceptable worst-case latency.

[0073] S4: The conveyor belt between the entrance of the defect detection line of the high-resolution camera and the screening area of ​​the optocoupler packaging device is long enough to receive the defect detection results of the optocoupler packaging device and classify it accordingly; there is a robot at the screening area, and the robot classifies the optocoupler packaging devices with corresponding identification codes according to the received control instructions; when the screening control end receives the preliminary identification detection result of the preliminary abnormality, it parses out the identification code of the optocoupler packaging device from it, and controls the robot to sort the optocoupler packaging device to the sorting area of ​​the preliminary abnormality category; when the screening control end receives the depth identification detection result of the deep abnormality, it parses out the identification code of the optocoupler packaging device from it, and controls the robot to sort the optocoupler packaging device to the sorting area of ​​the deep abnormality category; when the screening control end receives the depth identification detection result of the qualified product, it parses out the identification code of the optocoupler packaging device from it, and controls the robot to sort the optocoupler packaging device to the sorting area of ​​the qualified product category;

[0074] Through the robotic sorting steps, the results of different levels (preliminary abnormality / deep abnormality / qualified products) are classified accordingly to ensure the closed loop of the entire production line.

[0075] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0076] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An intelligent detection and classification method for optocoupler package defects, comprising: S1： At the entrance of the defect inspection line, a camera array is used to obtain the first image of the same device on different specified surfaces and store it; Downsampling the first image to generate a second image, and sending the second image to step S2; defect recognition of a device includes several subtasks, wherein the first image or the second image of a specified surface is a subtask; It is characterized by further comprising: S2, based on the second image, performs preliminary identification on the device to output a preliminary identification detection result, the preliminary identification detection result including preliminary abnormality and preliminary normality; the preliminary identification detection result of the preliminary abnormality is sent to S4, and the first image of the preliminary normal device and its corresponding device identification code are retrieved and encapsulated into a deep identification instruction and sent to step S3; S3: Perform deep recognition on the device based on the second image and output a deep recognition test result. The deep recognition test result includes deep abnormality, manual re-inspection, and qualified products. The execution process adopts a thread parallel approach; the thread pool load status is determined. If the thread pool is in a low load state, the dynamic and hard association random allocation principle is implemented; otherwise, the scheduling allocation principle is implemented; S4: Based on the received preliminary identification test results and the deep identification test results, the device identification code is parsed and the robot is controlled to sort the corresponding devices to the "preliminary abnormality sorting area", "deep abnormality sorting area", "qualified product sorting area" or "manual re-inspection area". During this process, the length of the conveyor belt from the entrance to the sorting area ensures real-time buffering between the detection results and the robot's sorting action.

2. The method for intelligent detection and classification of optical coupler package defects according to claim 1, characterized in that: The initial identification process is: Step 1: pre-process the second image to obtain a binary edge map; Step 2: Identify all connected domains in the binary edge map, calculate the contour area and corresponding convex hull area of ​​each connected domain, and obtain the Solidity value of each connected domain. This can be used to obtain the Solidity value of each connected domain in each second image of the same device. If any connected domain has a Solidity value less than the solid threshold, the device is judged to be "preliminarily abnormal"; otherwise, proceed to the next step. Step 3: Extract the average grayscale value of the minimum enclosing rectangular area of ​​each connected domain and the average grayscale value of the annular background area outside it from the binary edge map, and calculate the contrast value. This yields the solidity value of each connected domain in the second image of the same device. If any connected domain has a contrast value greater than the contrast threshold, it is determined to be a "preliminary anomaly"; otherwise, proceed to the next step. Step 4: Calculate the ratio of the area of ​​each connected domain in the binary edge map to the area of ​​the entire image and compare it with a preset area threshold. This yields the Area value of each connected domain in each second image of the same device. If any connected domain has an Area value greater than the area threshold, the device is judged to be "preliminarily abnormal." Otherwise, the device is judged to be "preliminarily normal." 3. The method for intelligent detection and classification of optical coupler package defects according to claim 2, characterized in that: Depth recognition process: Step 1: parse the device identification code and several first images from the depth recognition instruction, and perform hard association on the second images with the same identification code; Step 2: Extract the CPU occupancy of each thread. When the CPU occupancy is ≤0.3, it indicates that the thread is in a low-load state. If the low-load threads occupy more than half of all threads in the thread pool, the thread pool is judged to be in a low-load state, and the hard-association random allocation principle is activated. Specifically, the first image with a hard association is randomly assigned to the threads in the thread pool. The first image with a hard association relationship cannot be assigned to the same thread at the same time; otherwise, the scheduling allocation principle is activated. Step 3: The depth recognition performed by each thread is: The first image is converted to a different data type and normalized, and each pixel channel is then normalized to complete the input tensor construction. The first image is then subjected to defect detection using a trained deep convolutional neural network. Each type of defect has a corresponding characteristic pattern and confidence level. A defect detection result list is output for each first image. The list includes the defect type and the corresponding confidence level for each defect type, and the maximum confidence level is recorded as softmax. Each time a thread completes the reasoning of a subtask, its reasoning time is recorded.

4. The method for intelligent detection and classification of optical coupler package defects according to claim 1, characterized in that: The scheduling allocation principles are: 1-1, collect thread performance indicators, including: CPU usage, number of currently queued tasks, and average inference time within the past scheduled time. The performance indicators are normalized and integrated to obtain the scheduling priority score (Score), and the score of each thread is updated in real time. 1-2. From all the subtasks to be assigned, the time they enter the thread pool is recorded as the entry time. Each subtask is classified according to its corresponding hard association. If all subtasks of the same device are performing deep recognition, the subtask with the earliest entry time is selected and given the first priority, and the others are given the second priority. 1-3, the first-priority subtasks are sorted by entry time, the first subtask is selected and assigned to the first thread in the thread list, and this cycle repeats until all the first-priority subtasks are assigned; 1-4, if the sofmax of a subtask ∈ (0.5, 0.7), all other subtasks hard-associated with the subtask are promoted to the first priority; if the sofmax of a subtask is ≤ 0.5, the subtask with the earliest entry time of the subtask hard-associated with the subtask is extracted from the second priority and promoted to the first priority; if the sofmax of a subtask is ≥ 0.7, the depth recognition of other subtasks hard-associated with the subtask is directly interrupted and the depth recognition detection result of "depth abnormality" is output; each time the thread completes the reasoning of a subtask, its reasoning time is recorded; 1-5. If the sofmax of each subtask in the same identification number is ∈ (0.5, 0.7), the depth recognition detection result of "manual re-inspection" is output, and the defect list corresponding to each subtask is output as auxiliary information; if the sofmax of each subtask in the same identification number is ≤ 0.5, the depth recognition detection result of "qualified product" is output, and the stored first image is cleaned.

5. The method for intelligent detection and classification of optical coupler package defects according to claim 4, characterized in that: The 8-connected domain algorithm is used to determine the connected domain. The convex hull calculation is performed by forming a minimum convex polygon by constructing the convex hull of the connected domain boundary point set to obtain the convex hull area of ​​the connected domain. The minimum enclosing rectangle is an axisymmetric horizontal-vertical rectangle, and angle rotation is not considered. In the first step of step S2, if there is a reflection or shadow area on the plastic package surface in the second image, a local adaptive histogram equalization operation is further performed on the image to avoid excessive stretching.

6. The method for intelligent detection and classification of optical coupler package defects according to claim 5, characterized in that: The process of camera array collecting images is as follows: There are several cameras with adjustable resolutions on the defect detection line. Each camera is responsible for photographing the specified surface of the device to obtain the first image of the same device on different specified surfaces. Each device corresponds to each identification code. The robot controls the posture of the optocoupler packaging device to the specified posture by grabbing the device. The identification code of the device is in a fixed position and direction, and the specified surface is relative to the identification code. Therefore, the position of the identification code is fixed, and the position of each specified surface of the device is also uniformly fixed.

7. An intelligent detection and classification system for optical coupler package defects, characterized in that An intelligent detection and classification method for optical coupler package defects according to any one of claims 1 to 6, the system comprising a camera array, an entry control terminal, a background processing terminal, and a screening control terminal; There are several cameras with adjustable resolutions on the defect detection line. Each camera is responsible for photographing a specified surface of the device to obtain the first image of the same device on different specified surfaces and send it to the entry control terminal; The entry control terminal has a built-in preliminary identification module and a node storage module. The first image is stored in the node storage module. The preliminary identification module performs preliminary identification on the device based on the second image to output a preliminary identification detection result. The preliminary identification detection result includes preliminary abnormality and preliminary normality. The preliminary identification detection result of the preliminary abnormality is sent to the screening control terminal, and the first image of the preliminary normal device and its corresponding device identification code are retrieved from the node storage module, encapsulated as a deep identification instruction, and sent to the background processing terminal; The backend processing end has several built-in threads forming a thread pool. Each thread executes deep recognition in parallel and outputs deep recognition detection results. The deep recognition detection results include deep anomalies, manual re-inspections, and qualified products. The execution process adopts a thread parallel approach. The thread pool load status is judged. If the thread pool is in a low load state, the dynamic and hard association random allocation principle is implemented; otherwise, the scheduling allocation principle is implemented. The screening control terminal parses the received preliminary identification test results and deep identification test results to obtain the device identification code, and controls the robot to sort the corresponding devices to the "preliminary abnormality sorting area", "deep abnormality sorting area", "qualified product sorting area", or "manual re-inspection area". During this process, the length of the conveyor belt from the entrance to the sorting area ensures real-time buffering between the test results and the robot sorting action. The identification code of the "qualified product" device is extracted and sent to the entrance control terminal as a storage guide. The control node storage module cleans up the first image of this identification code.

Images