A chip glue point area identification method, terminal equipment and computer readable storage medium

Abstract

The present invention provides a chip glue spot area recognition method, terminal equipment and computer-readable storage medium, belonging to the chip glue spot detection technology field; including: establishing a LGDT data set based on laser chip glue points; designing a SFEDNet network based on the LGDT data set, and applying the SFEDNet network to perform pixel-by-pixel segmentation and labeling of the glue spot area of ​​the image; the SFEDNet network includes a spatial channel fusion module based on a U-net module, two weak edge detection modules and a jump connection. The SFEDNet network used in the present invention accurately segments the glue spot area, and the SFEDNet network includes a spatial channel fusion module based on a U-net module, two weak edge detection modules and a jump connection; firstly, a spatial fusion module is used in the backbone network of the U-net module to achieve an increase in network calculation speed and robust recognition of the glue spot area position; then, based on the weak edge detection module, the extraction of edge details of the glue spot area and semantic interaction are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of chip glue spot detection, and in particular relates to a chip glue spot area recognition method, terminal equipment and computer-readable storage medium. Background Art

[0002] In recent years, the detection of glue dots has received widespread attention in the fields of optoelectronic packaging, semiconductor chips, automobile manufacturing, and image recognition. In the field of automated coupling packaging of small laser devices, the glue dot area (usually with a radius of 0.4mm) poses a great challenge to identification due to its extremely small area. On the one hand, the tiny size of the glue dot area causes the identification of its position to take up a considerable amount of time during the coupling process. On the other hand, the limitations of the detection method lead to misjudgments when detecting the area and range of the glue dot area. In fact, glue dot detection or other quality detection based on glue dots has become one of the hot spots in image recognition in recent years, and there is a huge demand in automation equipment.

[0003] For glue spots in laser devices, the following work needs to be done before identification: (1) determine the placement direction of the laser chip (2) determine the position and area of ​​the glue spot in the laser chip. In the prior art, the glue spot area is seriously interfered by the background of the laser chip, which makes identification difficult. Summary of the invention

[0004] In order to achieve accurate recognition of laser chip glue spots, inspired by the application of deep learning in image recognition, the present invention provides a chip glue spot area recognition method that can achieve extraction of edge details of the glue spot area and semantic interaction.

[0005] The present invention provides a method for identifying a chip glue point area, comprising the following steps:

[0006] Step 1: Establish a LGDT dataset based on laser chip glue points;

[0007] Step 2: Design the SFEDNet network based on the LGDT dataset, and use the SFEDNet network to segment and mark the glue spot area of ​​the image pixel by pixel;

[0008] The SFEDNet network includes a spatial channel fusion module based on a U-net module, two weak edge detection modules and a skip connection;

[0009] The upsampling stage 1 and the downsampling stage 1 of the U-net module are connected via one of the weak edge detection modules, and the upsampling stage 2 and the downsampling stage 2 of the U-net module are connected via another weak edge detection module, for transmitting key channel features and edge detail information in the glue spot image to the output feature map;

[0010] Skip connections are set in both the upsampling and downsampling stages of the U-net module;

[0011] The spatial channel fusion module is applied in the downsampling stages 3 to 5 of the U-net module to obtain the multi-scale semantic feature map B5 of the glue spot image;

[0012] Taking the multi-scale semantic feature map B5 as the initial value, the spatial channel fusion module is applied in the upsampling 3rd and 4th stages of the U-net module.

[0013] Optionally, the spatial channel fusion module is used to calculate and identify the glue point area of ​​the image in the LGDT data set, and the spatial channel fusion module includes an efficient channel module and a position channel module that interact with each other.

[0014] Optionally, the process of using the spatial channel fusion module to process the original image and the predicted image is as follows:

[0015] S2.1 takes the input image F in Transmitted to the efficient channel module and the position channel module, the input image F in The operations are performed in the efficient channel module and the position channel module respectively to obtain the output feature map F of the efficient channel module e And the position channel module outputs the feature map F P ;

[0016] S2.2. Output feature map F of efficient channel module e And the position channel module outputs the feature map F P Perform matrix addition operation to obtain the output image F out .

[0017] Optionally, the position channel module outputs a feature map F P The process of obtaining is as follows:

[0018] Input image F in Transmitted to the first convolution layer, the second convolution layer, the third convolution layer and the average pooling layer respectively;

[0019] Input image F in Through the first convolutional layer K s = 1 convolution kernel to reconstruct and obtain the first feature map F P 0 ; Input image F in Through the second convolutional layer K s = 1 convolution kernel to reconstruct and obtain the second feature map F P 1 ; The first feature map F P0 and the second feature map F P 1 Perform matrix multiplication to obtain the third feature map F P 2 ;

[0020] Input image F in K that pass through the third convolutional layer in sequence s =1 convolution kernel, K s = 2 convolution kernels and K s =3 convolution kernels to reconstruct and obtain the feature map F with position information interaction P 3 , and the third feature map F P 2 And the fourth characteristic graph F P 3 Perform matrix multiplication to obtain the fifth feature map F P 4 ; The fifth feature map F P 4 Reconstruct and obtain the position channel module output feature map F P .

[0021] Optionally, the efficient channel module outputs the feature map F e The process of obtaining is as follows:

[0022] The input image F in Perform global average pooling processing;

[0023] Through 3×3 convolution and Sigmoid activation, and with the input image F in Recombined to obtain the efficient channel module output feature map F e .

[0024] Optionally, the weak edge detection module is used to extract and semantically interact with the edge details of the glue point area of ​​the image in the LGDT dataset, and the weak edge detection module includes edge feature fusion for extracting the edge details of the glue point area and edge feature refinement for semantically interacting with the edge details of the glue point area.

[0025] Optionally, the specific process of edge feature fusion is as follows:

[0026] Input image F in After edge extraction, the first edge feature map A is obtained e 0 ; Input image F in After the inversion operation, the second edge feature map is obtained Input image F in After high-frequency operation, the third edge feature map is obtained

[0027] For the first edge feature map With the input image F in Add the elements to get the first edge feature fusion map For the second edge feature map With the input image F in Add the elements to get the second edge feature fusion map For the third edge feature map With the input image F in Add the elements to get the third edge feature fusion map

[0028] The first edge feature fusion map, the second edge feature fusion map and the third edge feature fusion map are tensor-joined to obtain an edge fusion feature map F with complete edge interaction. ∈B，3C ,H ,W ;

[0029] Optionally, the specific process of edge feature refinement is as follows:

[0030] The edge fusion feature map F ∈B，3C,H,W After 3×3 convolution, the number of feature map channels is reduced to obtain the output feature map and

[0031] Output feature map and The number of channels of the feature map is the same as the number of channels of each feature map in the first step of feature refinement. After splicing and recombination, the edge refinement output map F is obtained. ∈B，2C,H,W ;

[0032] Output image F for edge refinement ∈B，2C,H,W Through 3×3 convolution, we get the edge refinement feature map

[0033] The present invention also provides a terminal device, which includes a memory and a processor, wherein the memory stores a computer program; the processor executes the computer program to implement the identification method as described above.

[0034] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement the identification method as described above.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The SFEDNet network used in the present invention accurately segments the glue spot area. The SFEDNet network includes a spatial channel fusion module based on the U-net module, two weak edge detection modules and jump connections; first, the spatial fusion module is used in the backbone network of the U-net module to achieve the improvement of network calculation speed and the robust recognition of the glue spot area position; then based on the weak edge detection module, the extraction of edge details and semantic interaction of the glue spot area are realized. The present invention improves the calculation speed by embedding the spatial channel fusion module, and at the same time improves the accuracy of glue spot position recognition; in addition, by using the weak edge detection module, more weights are allocated to the edge position and channel, thereby improving the accuracy of glue spot area edge segmentation. Compared with other network model algorithms, the recognition method provided by the present invention has more advantages in detection performance. In the actual laser chip glue spot detection experiment, SFEDNet also showed excellent performance.

[0037] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0039] FIG. 1( a ) is a microlens image of a qualified glue spot area after glue dispensing in the LGDT data set in an embodiment of the present invention;

[0040] FIG1( b ) is a transparent microlens image of the glue spot area captured after glue dispensing in the LGDT data set representing the optical laser coupling process in an embodiment of the present invention;

[0041] FIG1( c ) is a microlens image of the undispensed right glue point after dispensing in the LGDT data set in an embodiment of the present invention representing the optical laser coupling process;

[0042] FIG1( d ) is a microlens image in the LGDT data set representing an excessively large glue spot area after glue dispensing during the optical laser coupling process in an embodiment of the present invention;

[0043] FIG. 1( e ) is a microlens image in the LGDT data set representing a large difference in the area of ​​the left and right glue spots after glue dispensing in an optical laser coupling process in an embodiment of the present invention;

[0044] FIG1( f ) is a microlens image of a small left glue spot area after glue dispensing in the LGDT data set in an embodiment of the present invention, representing the optical laser coupling process;

[0045] Figure 2 Schematic diagram of the structure of the SFEDNet network in an embodiment of the present invention;

[0046] Figure 3 for Figure 2 Schematic diagram of the structure of the mid-space channel fusion module;

[0047] Figure 4 for Figure 2 Schematic diagram of the structure of the weak edge detection module;

[0048] Figure 5 This is a schematic diagram of the comparison of the output images of the SFEDNet ablation experiment in the experimental example of the present invention;

[0049] Figure 6 Schematic diagram of the comparison of output image effects of five networks in the experimental example of the present invention;

[0050] Figure 7 Schematic diagram of ROC curves of five segmentation networks based on the LGDT data set in the experimental example of the present invention;

[0051] Figure 8 Schematic diagram for comparing the detection of glue spot area and position in the experimental example of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more clear and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be noted that the drawings of the present invention are all simplified and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the implementation of the present invention; the "number" mentioned in the present invention is not limited to the specific number in the examples in the accompanying drawings; the directions or positional relationships indicated by "front", "middle", "back", "left", "right", "up", "down", "top", "bottom", "middle", etc. mentioned in the present invention are based on the directions or positional relationships shown in the drawings of the present invention, and do not indicate or imply that the devices or components referred to must have a specific direction, nor can they be understood as limitations on the present invention.

[0053] Embodiment 1:

[0054] The present invention provides a method for identifying a chip glue point area, comprising the following steps:

[0055] Step 1: Establish a LGDT dataset based on laser chip glue points.

[0056] The method for obtaining the data set is as follows:

[0057] The camera set in the semiconductor laser automatic coupling device is used to collect chip glue spot images to obtain the LGDT data set;

[0058] And 60% of the data in the LGDT dataset is used for training, 20% of the data is used for testing, and 20% of the data is used for validation.

[0059] Furthermore, in order to stably capture the glue spot image and realize high-precision recognition of the glue spot area, the camera used to collect the chip glue spot image is a HIKROBOT MV-CU050-60UM camera, and the lens used for image acquisition is a telecentric lens. The specific working parameters of the camera and lens are shown in Table 1.

[0060] Table 1 Camera and lens parameters

[0061]

[0062] See Figure 1 and Table 2 for representative glue spot images and their labels.

[0063] Table 2 Characteristics of laser chip glue points

[0064]

[0065] As shown in Figure 1 and Table 1, the image size in the LGDT dataset is 256×192, and it mainly contains glue dot images of different glue dispensing situations. Since the glue dispensing position required by the laser chip is too close to the grayscale value of the glue image, it is difficult to distinguish the exact range of the actual glue dot area (see Figure 1(a)). However, the laser chip has a much higher reflectivity to light, so its surface has a very high brightness. At the same time, the semi-transparent glue and the extremely small glue dot area make it almost impossible for the human eye to distinguish the edge and area range of the glue dot area under the interference of the laser chip background (see Figure 1(b) to Figure 1(f) shown).

[0066] Step 2: Segment and mark the glue spot area of ​​the image in the LGDT dataset pixel by pixel to distinguish the actual glue spot area and the laser chip background. The specific process is as follows:

[0067] S2.1. Design SFEDNet (Spatially Fused Faint Edge Detection Network) based on LGDT dataset.

[0068] S2.2. Apply the SFEDNet network to accurately segment glue spots.

[0069] Furthermore, when segmenting the glue spot area in the laser device coupling, due to the diversity of the glue area shape and background interference, strict network design is required to achieve its accurate identification. This application uses a binary classification method to segment the image of the LGDT dataset (specifically, only the image in the LGDT dataset is segmented from the background of its laser chip), and combines a neural network to improve the efficiency and accuracy of glue spot area segmentation.

[0070] See also Figure 2 As shown, the SFEDNet network includes a spatial channel fusion module (SCFM) based on a U-net module, two weak edge detection modules (WEEMs) and skip connections;

[0071] A U-net module is provided in the spatial channel fusion module; the first upsampling stage and the first downsampling stage of the U-net module are connected via one of the weak edge detection modules, and the second upsampling stage and the second downsampling stage of the U-net module are connected via another weak edge detection module, so as to transmit key channel features and edge detail information in the glue point image to the output feature map, thereby increasing the weights of the corresponding channels and edges;

[0072] Skip connections are added in the upsampling and downsampling stages of the U-net module to enrich the semantic features and position information of the output feature map;

[0073] The spatial channel fusion module is applied in the downsampling stages 3 to 5 of the U-net module to obtain the multi-scale semantic feature map B5 of the glue spot image. Specifically, the multi-scale semantic feature map B5 serves as the initial guiding region feature map for subsequent upsampling and contains important global information features.

[0074] Furthermore, taking the multi-scale semantic feature map B5 as the initial value, the spatial channel fusion module is applied in the upsampling third and fourth stages of the U-net module. Through the operation of the spatial channel fusion module, the extraction efficiency of the target features is improved.

[0075] See also Figure 3 As shown, the spatial channel fusion module includes an efficient channel module and a position channel module, and the efficient channel module and the position channel module interact with each other.

[0076] Furthermore, the process of using the spatial channel fusion module to process the original images in the LGDT dataset is as follows:

[0077] S2.1 takes the input image F in Transmitted to the efficient channel module and the position channel module, the input image F inThe operations are performed in the efficient channel module and the position channel module respectively to obtain the output feature map F of the efficient channel module e And the position channel module outputs the feature map F P ;

[0078] S2.2. Output feature map F of efficient channel module e And the position channel module outputs the feature map F P Perform matrix addition operation to obtain the output image F out

[0079]

[0080] Furthermore, the position channel module outputs a feature map F P The process of obtaining is as follows:

[0081] Input image F in Transmitted to the first convolution layer, the second convolution layer, the third convolution layer and the average pooling layer respectively;

[0082] Input image F in Through the first convolutional layer K s = 1 convolution kernel to reconstruct and obtain the first feature map Input image F in Through the second convolutional layer K s = 1 convolution kernel to reconstruct and obtain the second feature map The first feature map And the second feature map Perform matrix multiplication to obtain the third feature map

[0083]

[0084] Among them, Re is the feature map reconstruction, conv1 and conv3 represent the operation with convolution kernel 1×1 and 3×3 respectively, and Softmax is represented by the normalized exponential function;

[0085] Input image F in K that pass through the third convolutional layer in sequence s =1 convolution kernel, K s = 2 convolution kernels and K s =3 convolution kernels to reconstruct and obtain a feature map with position information interaction And the third feature map And the fourth characteristic diagram Perform matrix multiplication to obtain the fifth feature map The fifth feature map Reconstruct and obtain the position channel module output feature map F P ;

[0086]

[0087] Among them, conv5 represents the operation using convolution kernel 5×5.

[0088] Furthermore, the efficient channel module outputs the feature map F e The process of obtaining is as follows:

[0089] The input image F in Perform global average pooling processing;

[0090] Through 3×3 convolution and Sigmoid activation, and with the input image F in Recombined to obtain the efficient channel module output feature map F e ;

[0091]

[0092] Among them, Sigmoid represents the refined features after activation of convolution operation, and Aug_pool represents the average pooling operation.

[0093] See also Figure 4 As shown, the faint edge detection module includes edge feature fusion and edge feature refinement.

[0094] Furthermore, the specific process of edge feature fusion is as follows:

[0095] Input image F in After edge extraction, the first edge feature map is obtained Input image F in After the inversion operation, the second edge feature map is obtained Input image F in After high-frequency operation, the third edge feature map is obtained

[0096] For the first edge feature map With the input image F in Add the elements to get the first edge feature fusion map For the second edge feature map With the input image F in Add the elements to get the second edge feature fusion map For the third edge feature map With the input image F in Add the elements to get the third edge feature fusion map

[0097] The first edge feature fusion map, the second edge feature fusion map and the third edge feature fusion map are tensor-joined to obtain an edge fusion feature map F with complete edge interaction. ∈B，3C ,H ,W ;

[0098]

[0099] in, Represents matrix element-wise addition. Represents tensor concatenation.

[0100] Furthermore, the specific process of edge feature refinement is as follows:

[0101] The edge fusion feature map F ∈B，3C ,H ,W After 3×3 convolution, the number of feature map channels is reduced to obtain the output feature map and

[0102] Output feature map and The number of channels of the feature map is the same as the number of channels of each feature map in the first step of feature refinement. After splicing and recombination, the edge refinement output map F is obtained. ∈B，2C ,H ,W ;

[0103] Output image F for edge refinement ∈B，2C ,H ,W Through 3×3 convolution, we get the edge refinement feature map

[0104] F ∈B , 2C,H ,W =conv3(F ∈B ,3C,H ,W )

[0105]

[0106] Among them, ReLU represents nonlinear mapping.

[0107] The weak edge segmentation mechanism is achieved by merging edge feature maps and performing convolution operations and fusion refinement on the feature maps. The specific details of the further fusion refinement calculation are as follows;

[0108]

[0109]

[0110] Among them, A represents the edge feature refinement output feature map; Represents matrix multiplication, that is, merging the further refined and fused feature map with the input feature map to output the edge feature map.

[0111] WEEM can not only use the interaction of edge information to assign weights to different positions, but also focus attention weights on edge key channels by fusing and refining channel features. Laser chip glue point segmentation usually has complex chip background light reflection and ambient light interference, and the feature fusion of the edge of the glue point area can be achieved through WEEM, that is, eliminating the non-critical feature interference of the background and assigning weights to more important edge pixel positions and channels, thereby achieving semantic enhancement of the glue point edge feature map.

[0112] Embodiment 2:

[0113] The present invention also provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0114] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the first embodiment above.

[0115] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0116] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0117] Embodiment three:

[0118] The present invention further provides a computer-readable storage medium corresponding to the above embodiment, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.

[0119] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0120] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language python and etc.

[0121] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0123] Experimental example:

[0124] 1. Experimental setup

[0125] The experiments were conducted using the PyTorch framework on a computer with a 12GB NVIDIA RTX 3060 GPU. Adam was used as the regularized optimizer with a learning rate of 10-4. In addition, in order to expand the number of glue dot images, the dataset was processed using image augmentation during training. To accelerate the convergence of the network, we use a combination of binary cross entropy (BCE) loss and dice loss function to form a new loss function L CE ;

[0126]

[0127] Among them, α is a hyperparameter, which is taken as 0.5 in the experiment; and O represent the predicted value and the true value of the mask respectively.

[0128] For numerical evaluation, we applied commonly used evaluation indicators in image recognition to evaluate the network we designed, such as F1 score (F1), normalized intersection over Union (nIoU), and Dice. Dice reflects the similarity between the predicted result and the mask, and the closer it is to 1, the better the network performance. Specifically, the calculation details of the numerical evaluation are as follows:

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] Among them, Precision represents the accuracy of the prediction, Recall represents the recall rate of the prediction, Mask represents the target area in the label image, pred represents the number of pixels in the predicted image, true represents the number of pixels in the mask, Detectedpixels of targets represents the predicted number of correct target pixels, Total pixels of detection represents the total number of pixels in the area to be detected, Total pixels of targets represents the total number of pixels in the target area, and Precisionregion represents the predicted image area.

[0135] In addition, the receiver operating characteristic (ROC) curve is used to reflect the relationship between the true positive rate (TPR) and the false alarm rate (FAR). In the curve between the true positive rate (TPR) and the false alarm rate (FAR), the larger the enclosed area in the lower right corner, the better the performance of the network.

[0136]

[0137]

[0138] Among them, True pixels of small targets represents the number of pixels correctly identified as small targets by the model, that is, the part of pixels that are actually small targets that are correctly detected by the model; Total pixels of small targets represents the number of all pixels that actually belong to small targets, which is the sum of all small target pixels, including those that are misjudged or not detected; False pixels besides small targets represent pixels that are incorrectly marked as targets by the model, but these pixels do not actually belong to the small target area; Total pixels of infrared image represents the total number of all pixels in the entire image. As the denominator, it is used to measure the proportion of incorrectly marked pixels in the overall image.

[0139] (II) SFEDNet ablation experiment

[0140] Ablation experiments can clearly show the effect of adding each module to the network. In this experiment, we add each module in turn based on U-net and evaluate the results numerically and visually. The visualization results are shown in Table 3 and Table 4. Figure 6 shown.

[0141] Table 3 SFEDNet ablation experiment

[0142]

[0143]

[0144] Through the analysis of Table 3, it can be concluded that the evaluation indicators "F1", "nIoU" and "Dice" based only on U-net reached 87.31%, 87.24% and 93.10% respectively. This may be because most of the images in the glue dot dataset are of qualified quality in the glue dot area, resulting in relatively good segmentation results. However, in the actual glue dot and recognition process of the laser chip, considering the diverse glue dot areas, higher evaluation indicators are needed to ensure the success rate of recognition. After the introduction of WEEM, "F1", "nIoU" and "Dice" increased by 0.69%, 1.05% and 0.60% respectively, but the improvement effect of U-net+WEEM is limited compared to U-net. Using the weak edge detection module, sufficient weight allocation and feature fusion of the glue dot edge area are completed in the first two stacked layers. Thereby improving the overall performance of the network.

[0145] When the U-net+SCFM network structure is adopted, compared with U-net, the performance of "F1", "nIoU" and "Dice" is improved by 1.25%, 2.50% and 1.43%, respectively. However, compared with U-net+WEEM, SFEDNet shows the same limited performance improvement. Compared with U-net+WEEM, SFEDNet based on the spatial fusion attention module improves "F1", "nIoU" and "Dice" by 0.56%, 1.45% and 0.83%, respectively. Using WEEM and SCFM alone has limited improvement on the overall performance of the network, but when the two modules are added and fused to form SFEDNet, the performance of "F1", "nIoU" and "Dice" is improved by 1.84%, 3.91% and 2.21%, respectively. The performance of the network can be improved more effectively. Based on the spatial channel fusion and weak edge detection mechanism, SFEDNet realizes the interaction of global channel and position information and the extraction of key edge features, which effectively improves the segmentation ability of the neural network for the glue point area.

[0146] Figure 5 The visual display shows the individual contribution of each module to the glue dot segmentation performance.

[0147] for Figure 5 The outlines of the glue dot areas shown in columns (1) and (2) are different from the qualified glue dot areas, and there are glue shortages and glue overflows. When only using U-net to detect the outlines of the glue dot areas, the network cannot distinguish the actual glue dot areas due to global feature fuzziness and background interference in the chip glue dot area; while using WEEM can focus on the edge positions and channels, thereby achieving accurate segmentation of the glue dot range. Figure 5 As shown in columns (1) and (2) of , after adding the WEEM module, the contours of the glue-missing and glue-overflowing parts can be better distinguished from the background compared to U-net.

[0148] for Figure 5 As shown in columns (3) and (4) of , there are no glue spots in the laser chip. However, due to the high reflectivity of the chip to light, there are complex background light reflections and ambient light interference, which leads to a certain false alarm probability when using the U-net network for detection, and other areas in the chip are mistakenly predicted as glue spot areas. Similarly, after adding WEEM alone, the weak edge attention mechanism does not eliminate the interference of the glue-free area in the background, and the combination of U-net+WEEM still produces some false alarms. However, after adding SCFM, the wrong prediction of the glue spot position can be effectively avoided, the background interference is eliminated to a large extent, and the accuracy of the glue spot position prediction is improved.

[0149] In general, Figure 5The (5th) column in shows the image of the glue spot area with qualified shape. The results of the ablation experiment show that the single module fusion exhibits a higher detection recall rate. The possible reason is that the edge features of the glue spot area image with qualified shape are obvious under the normal background, the target pixels are large, and a complex network for edge extraction and position positioning is not required. However, after detailed comparison, we found that the network of SFEDNet with SCFM added is more accurate in position prediction and has a higher detection recall rate.

[0150] The ablation experiment results show that the combination of the weak edge detection module WEEM and the spatial channel fusion module SCFM not only fully extracts the edge detail information and key channel features of the glue spots, but also enhances the accurate prediction of the glue spot position and improves the detection accuracy.

[0151] (III) Network comparison experiment

[0152] The research on the segmentation network for laser chip glue spots has not been widely reported. After comprehensive selection, we used the network for infrared small target segmentation to train LGDT, and compared the evaluation indicators and output results of the verification results with SFEDNet. The comparison methods mainly include UIU, LSPM, DNA, MTU, etc. UIU embeds a small U-Net into the U-Net backbone network to achieve multi-level and multi-scale representation learning of the target. LSPM uses a powerful pyramid structure and attention mechanism to segment the target. DNA achieves target segmentation by combining densely nested interaction modules and channel space attention modules. The MTU network constructs a multi-level feature extraction module based on the visual transformer (ViT) convolutional neural network to achieve detection of distant targets. The results of the comparative experiment are as follows:

[0153] Table 4 Comparison of five latest LGDT segmentation networks

[0154] network F1 iGO Dice Time (s / 100 pictures) UIU 0.7892 0.7594 0.8621 7.92 LSPM 0.8300 0.7774 0.8734 64.06 DNA 0.8548 0.8433 0.9130 11.13 MTU 0.8816 0.9008 0.9473 3.72 SFEDNet 0.8915 0.9115 0.9531 3.55

[0155] Table 4 shows the evaluation indicators and network performance results of the five detection methods. The evaluation indicators show that the performance of the five comparison networks in "F1", "nIoU" and "Dice" increases one by one, and SFEDNet has the best performance compared with the first four networks. Compared with the optimal values ​​of the other four networks, SFEDNet improves by 0.99%, 1.07% and 0.58% in "F1", "nIoU" and "Dice", respectively. The segmentation effect of MTU is second only to SFEDNet. Since MTU is a convolutional neural network embedded with visual transformer (ViT) and multi-level feature extraction modules, it has achieved good results in glue point area segmentation. However, since UIU only uses a simple U-Net network nesting, its segmentation performance is poor. In the image detection speed comparison that automation equipment pays more attention to, the processing speed of SFEDNet is only 3.55s per 100 images, which is better than the processing speed of other algorithms. The improvement of image segmentation speed can quickly improve the efficiency of glue point area recognition, which is very important for improving the speed of laser automation production. MTU is 0.17 seconds slower than SFEDNet, while LSPM is the slowest, requiring 64.06 seconds. It is proved that the spatial fusion module embedded with the efficient channel module can effectively reduce the prediction time of the network.

[0156] Figure 6 The output comparison of five networks is intuitively shown. Figure 6 The columns (1) to (6) shown in the figure respectively contain semiconductor laser chips of different models and different glue dispensing conditions. Figure 6 In the (3rd) and (6th) columns shown in , when faced with complex backgrounds and glue spots of varying sizes, the UIU, LSPM, DNA, and MTU networks cannot accurately segment the glue spots, while SFEDNet can accurately detect the location and shape of the glue spots, proving the accuracy of SFEDNet in position and edge detection. When there are no glue spots in the chip image, see Figure 6 As shown in column (2) of , the output image of MTU produces artifacts, which may be due to its lack of position information interaction and spatial channel fusion to determine and exclude similar background and target information. Figure 6 As shown in columns (2), (4) and (5), except for the DNA network, from Figure 7 It can be seen from the output image that the other four networks can accurately locate the position of the glue spot area, but there are still large differences when comparing the details of the regional contour. Compared with the other four networks, SFEDNet based on the weak edge detection module is more accurate in the edge segmentation of the glue spot area, which is of great significance for the identification of the edge contour of the glue spot area in subsequent preprocessing.

[0157] The ROC curve can help us get a more intuitive evaluation from the perspective of visual and numerical results. The ROC curves of the five comparison networks based on LGDT are as follows: Figure 7 As shown. The MTU method ranks second in performance, indicating that the global self-attention mechanism and multi-level feature extraction module of the visual transformer (ViT) can effectively improve the accuracy of region segmentation. Compared with other networks, SFEDNet has the largest closed area, proving that the spatial channel fusion and weak edge extraction mechanisms are helpful for accurate segmentation of the target area. The reason why the UIU effect is not ideal may be that the network is not suitable for the segmentation of glue point area targets with diverse features. The comprehensive comparison results of the evaluation indicators and the output images show that SFEDNet has a positive effect on improving the accuracy of glue point area segmentation. And the combination of the spatial channel fusion module and the weak edge extraction mechanism can accurately segment the edge details of the glue point area and improve the detection efficiency.

[0158] Figure 8 The mask is compared with the area and position of the glue spot after segmentation. The present invention uses the Hough ellipse fitting algorithm to identify the elliptical contour of the glue spot area edge. Figure 8 The glue spots and different sizes of glue spot areas in different chip models shown in columns (1) to (6) are all interfered by the bright background of the chip or the grayscale value of the glue spot area is similar to that of the glue spot. Comparison with the mask shows that the maximum errors of the glue spot area position and area recognition after SFEDNet segmentation and the true value are 21.83μm and 0.003mm2 respectively. The above experimental results show that SFEDNet based on the weak edge detection module and the spatial channel fusion module can be used to achieve accurate segmentation and position detection of the glue spot area. Finally, based on the test results of SFEDNet, by adjusting the glue needle to the chip glue spot position and controlling the glue spot amount, hundreds of glue spotting experiments show that the recognition success rate reaches more than 95%.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying a chip glue point area, characterized in that: The following steps are involved: Step 1: Establish a LGDT dataset based on laser chip glue points; Step 2: Design the SFEDNet network based on the LGDT dataset, and use the SFEDNet network to segment and mark the glue spot area of ​​the image pixel by pixel; The SFEDNet network includes a spatial channel fusion module based on a U-net module, two weak edge detection modules and a skip connection; The upsampling stage 1 and the downsampling stage 1 of the U-net module are connected via one of the weak edge detection modules, and the upsampling stage 2 and the downsampling stage 2 of the U-net module are connected via another weak edge detection module, for transmitting key channel features and edge detail information in the glue spot image to the output feature map; Skip connections are set in both the upsampling and downsampling stages of the U-net module; The spatial channel fusion module is applied in the downsampling stages 3 to 5 of the U-net module to obtain the multi-scale semantic feature map B5 of the glue spot image; Taking the multi-scale semantic feature map B5 as the initial value, the spatial channel fusion module is applied in the upsampling 3rd and 4th stages of the U-net module; The spatial channel fusion module is used to calculate and identify the glue point area of ​​the image in the LGDT data set, and the spatial channel fusion module includes an efficient channel module and a position channel module that interact with each other; The weak edge detection module is used to extract and semantically interact with the edge details of the glue point area of ​​the image in the LGDT data set, and the weak edge detection module includes edge feature fusion for extracting the edge details of the glue point area and edge feature refinement for semantically interacting with the edge details of the glue point area; The specific process of edge feature fusion is as follows: Input image F in After edge extraction, the first edge feature map is obtained Input image F in After the inversion operation, the second edge feature map is obtained Input image F in After high-frequency operation, the third edge feature map is obtained For the first edge feature map With the input image F in Add the elements to get the first edge feature fusion map For the second edge feature map With the input image F in Add the elements to get the second edge feature fusion map For the third edge feature map With the input image F in Add the elements to get the third edge feature fusion map The first edge feature fusion map, the second edge feature fusion map and the third edge feature fusion map are tensor-joined to obtain an edge fusion feature map F with complete edge interaction. ∈B，3C ,H ,W .

2. The identification method according to claim 1, characterized in that: The process of processing the original image and the predicted image using the spatial channel fusion module is as follows: S2.1 takes the input image F in Transmitted to the efficient channel module and the position channel module, the input image F in The operations are performed in the efficient channel module and the position channel module respectively to obtain the output feature map F of the efficient channel module e And the position channel module outputs the feature map F P ; S2.

2. Output feature map F of efficient channel module e And the position channel module outputs the feature map F P Perform matrix addition operation to obtain the output image F out .

3. The identification method according to claim 2, characterized in that: The position channel module outputs a feature map F P The process of obtaining is as follows: Input image F in Transmitted to the first convolution layer, the second convolution layer, the third convolution layer and the average pooling layer respectively; Input image F in Through the first convolutional layer K s = 1 convolution kernel to reconstruct and obtain the first feature map Input image F in Through the second convolutional layer K s = 1 convolution kernel to reconstruct and obtain the second feature map The first feature map And the second feature map Perform matrix multiplication to obtain the third feature map Input image F in K that pass through the third convolutional layer in sequence s =1 convolution kernel, K s = 2 convolution kernels and K s =3 convolution kernels to reconstruct and obtain a feature map with position information interaction And the third feature map And the fourth characteristic diagram Perform matrix multiplication to obtain the fifth feature map The fifth feature map Reconstruct and obtain the position channel module output feature map F P .

4. The identification method according to claim 2, characterized in that: Efficient channel module output feature map F e The process of obtaining is as follows: The input image F in Perform global average pooling processing; Through 3×3 convolution and Sigmoid activation, and with the input image F in Recombined to obtain the efficient channel module output feature map F e .

5. The identification method according to claim 1, characterized in that: The specific process of edge feature refinement is as follows: The edge fusion feature map F ∈B，3C ,H ,W After 3×3 convolution, the number of feature map channels is reduced to obtain the output feature map and Output feature map and The number of channels of the feature map is the same as the number of channels of each feature map in the first step of feature refinement. After splicing and recombination, the edge refinement output map F is obtained. ∈B，2C ,H ,W ; Output image F for edge refinement ∈B，2C ,H ,W Through 3×3 convolution, we get the edge refinement feature map 6. A terminal device, characterized in that: The terminal device includes a memory and a processor, wherein the memory stores a computer program; the processor executes the computer program to implement the identification method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the identification method according to any one of claims 1 to 5.

Images