Target defect detection method and detection device, computer equipment, and storage medium

Abstract

The embodiment of the present application provides a target defect detection method and detection device, computer equipment, and storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a BGA image to be detected, the BGA image includes at least one candidate detection sub-image; using a Mosaic function to perform data enhancement operations on each candidate detection sub-image to obtain at least one target defect image in the BGA image; performing defect detection on each target defect image according to a pre-trained FPN architecture model, and outputting the target defect category and target defect probability value corresponding to each target defect image. The embodiment of the present application performs target defect detection on candidate detection sub-images in a BGA image according to a pre-trained FPN architecture model, which can improve the accuracy and efficiency of small target defect detection of solder balls.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target defect detection method and detection device, computer equipment, and storage medium. Background Art

[0002] There are currently two main methods for solder ball defect detection: one is manual detection. The traditional manual detection method is for workers to inspect solder balls one by one by wearing microscopes. This method has high work intensity and is easily affected by subjective factors such as workers' working status and work experience, thus affecting the accuracy of solder ball defect judgment. In addition, the production line speed is subject to certain restrictions during manual inspection, which affects production efficiency. The second is 2D image processing based on machine vision, which detects solder ball defects through optical and image processing technology, and uses traditional visual Blob analysis, template matching and other technologies. Although the recognition speed meets laboratory requirements, such speed is definitely far from enough in actual production. At the same time, due to the small size of solder ball damage defects, there is a high missed detection rate. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a target defect detection method and detection device, computer equipment, and storage medium, which can improve the accuracy and efficiency of small target defect detection of solder balls.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a target defect detection method, the method comprising:

[0005] Acquire a BGA image to be detected, wherein the BGA image includes at least one candidate detection sub-image;

[0006] Using a Mosaic function to perform a data enhancement operation on each candidate detection sub-image to obtain at least one target defect image in the BGA image;

[0007] Defect detection is performed on each target defect image according to the pre-trained FPN architecture model, and the target defect category and target defect probability value corresponding to each target defect image are output.

[0008] In some embodiments, the FPN architecture model is trained by the following method:

[0009] Constructing a training sample set, wherein the training sample set includes a plurality of defect sample images;

[0010] Performing a downsampling operation on each defect sample image to obtain a second defect feature map having a plurality of different feature sizes corresponding to the defect sample image;

[0011] Constructing a multi-layer downsampling structure according to the plurality of second defect feature maps with different feature sizes;

[0012] Performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes;

[0013] Constructing a multi-layer upsampling structure according to the plurality of third defect feature maps of different sizes;

[0014] Performing a downsampling operation on each third defect feature map according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes;

[0015] The FPN architecture model is trained according to each fourth defect feature map and a preset defect category until the precise value of the model training meets a preset training end condition to obtain the FPN architecture model.

[0016] In some embodiments, performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes includes:

[0017] Obtaining a second defect feature map in the deepest layer of the multi-layer downsampling structure, wherein the second defect feature map in the deepest layer is obtained by performing a convolution operation on a second defect feature map in a previous layer corresponding to the second defect feature map in the deepest layer;

[0018] Performing feature fusion on the second defect feature map of the previous layer and the second defect feature map of the deepest layer to obtain a third defect feature map having a feature size equal to that of the second defect feature map of the previous layer;

[0019] The obtained third defect feature map is replaced with the second defect feature map of the deepest layer, and feature fusion is performed with the second defect feature map of the previous layer corresponding to the current second defect feature map of the deepest layer to obtain multiple third defect feature maps of different sizes.

[0020] In some embodiments, the defect detection is performed on each target defect image according to the pre-trained FPN architecture model, and the target defect category and target defect probability value corresponding to each target defect image are output, including:

[0021] Perform defect detection on each target defect image according to the pre-trained FPN architecture model to obtain a candidate defect probability value corresponding to each target defect image under a candidate defect category;

[0022] Numerical comparison is performed on the plurality of candidate defect probability values ​​to obtain a target defect category and a target defect probability value corresponding to each target defect image.

[0023] In some embodiments, the method further comprises:

[0024] The target defect category and the target defect probability value are marked on the corresponding target defect image in the BGA image.

[0025] A second aspect of an embodiment of the present application provides a target defect detection device, the device comprising:

[0026] An image acquisition module, used to acquire a BGA image to be detected, wherein the BGA image includes at least one candidate detection sub-image;

[0027] A data enhancement module, used for performing a data enhancement operation on each candidate detection sub-image using a Mosaic function to obtain at least one target defect image in the BGA image;

[0028] The defect detection module is used to perform defect detection on each target defect image according to the pre-trained FPN architecture model, and output the target defect category and target defect probability value corresponding to each target defect image.

[0029] In some embodiments, the apparatus further comprises:

[0030] FPN architecture model module, the FPN architecture model corresponding to the FPN architecture model module is trained by the following method:

[0031] Constructing a training sample set, wherein the training sample set includes a plurality of defect sample images;

[0032] Performing a downsampling operation on each defect sample image to obtain a second defect feature map having a plurality of different feature sizes corresponding to the defect sample image;

[0033] Constructing a multi-layer downsampling structure according to the plurality of second defect feature maps with different feature sizes;

[0034] Performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes;

[0035] Constructing a multi-layer upsampling structure according to the plurality of third defect feature maps of different sizes;

[0036] Performing a downsampling operation on each third defect feature map according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes;

[0037] The FPN architecture model is trained according to each fourth defect feature map and a preset defect category until the precise value of the model training meets a preset training end condition to obtain the FPN architecture model.

[0038] In some embodiments, the defect detection module includes:

[0039] A defect detection submodule is used to perform defect detection on each target defect image according to the pre-trained FPN architecture model to obtain a candidate defect probability value corresponding to each target defect image under a candidate defect category;

[0040] The numerical comparison module is used to perform numerical comparison on the plurality of candidate defect probability values ​​to obtain the target defect category and the target defect probability value corresponding to each target defect image.

[0041] A third aspect of an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute a method as described in any one of the embodiments of the first aspect of the present application.

[0042] The fourth aspect of the embodiments of the present application proposes a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the method described in any one of the embodiments of the first aspect of the present application.

[0043] The target defect detection method and detection device, computer equipment, and storage medium proposed in the embodiments of the present application obtain at least one candidate detection sub-image in the BGA image by acquiring a BGA image to be detected. In order to improve the accuracy of defect detection on the image, a Mosaic function is used to perform data enhancement operations on each candidate detection sub-image to obtain at least one target defect image in the BGA image. Defect detection is performed on each target defect image according to a pre-trained FPN architecture model, and the target defect category and target defect probability value corresponding to each target defect image are output. The embodiments of the present application perform target defect detection on candidate detection sub-images in the BGA image according to the pre-trained FPN architecture model, which can improve the accuracy and efficiency of small target defect detection of solder balls. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a target defect detection method provided by an embodiment of the present application;

[0045] Figure 2 It is a first flow chart of the FPN architecture model provided in an embodiment of the present application;

[0046] Figure 3 It is a second flow chart of the FPN architecture model provided in an embodiment of the present application;

[0047] Figure 4 yes Figure 1 Flow chart of step S130 in FIG.

[0048] Figure 5 It is a structural schematic diagram of a target defect detection method provided in an embodiment of the present application;

[0049] Figure 6 It is a module structure block diagram of the target defect detection device provided in an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0055] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0057] First, some nouns involved in this application are analyzed:

[0058] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] Blob analysis: refers to a connected area in an image. Blob analysis is to extract and mark the connected domain of a binary image after foreground / background separation. Each marked blob represents a foreground target, and then some related features of the blob can be calculated. Its advantage is that through blob extraction, information about related areas can be obtained, but the speed is slow and the analysis is difficult.

[0060] Ball Grid Array (BGA) technology: a surface mount packaging technology used on integrated circuits, often used to permanently fix devices such as microprocessors. BGA packaging can provide more pins than other packages such as dual in-line packages or quad flat packages, and the entire bottom surface of the device can be used as pins. It also has a shorter average wire length than the surrounding limited package types and has better high-speed performance.

[0061] Feature Pyramid Network (FPN): It uses both the high resolution of low-level features and the high semantic information of high-level features to achieve prediction results by fusing features from different layers.

[0062] The three primary colors (Red-Green-Blue, RGB) are used to represent any color that can be seen by the naked eye in nature, that is, it can be mixed and superimposed by these three colors, so it is also called additive color mode.

[0063] Down-sampling operation: refers to reducing the size of an image. Its main purpose is to make the image fit the size of the display area and generate a thumbnail of the corresponding image.

[0064] Pooling: It is a form of downsampling that compresses the input feature image. On the one hand, it reduces the image features, resulting in a reduction in parameters, thereby simplifying the complexity of convolutional network calculations; on the other hand, it maintains a certain invariance of the features (rotation, translation, scaling, etc.).

[0065] Unsampling operation: that is, expanding the height and width of the feature image. When extracting features from an image, it is necessary to downsample the target feature image, so during the encoding process, a corresponding upsampling operation must be performed. Common upsampling operation methods include bilinear interpolation, transposed convolution, upsampling, and up-pooling.

[0066] There are currently two main methods for solder ball defect detection: one is manual detection. The traditional manual detection method is for workers to inspect solder balls one by one by wearing microscopes. This method has high work intensity and is affected by subjective factors such as workers' working status and work experience, which affects the accuracy of solder ball defect judgment. In addition, the production line speed is subject to certain restrictions during manual inspection, which affects production efficiency. The second is 2D image processing based on machine vision, which detects solder ball defects through optical and image processing technology, and uses traditional visual Blob analysis, template matching and other technologies. Although the recognition speed meets laboratory requirements, such speed is definitely far from enough in actual production. At the same time, due to the small size of solder ball damage defects, there is a high missed detection rate.

[0067] Based on this, the embodiments of the present application propose a target defect detection method and detection device, computer equipment, and storage medium, aiming to improve the accuracy and efficiency of small target defect detection of solder balls.

[0068] The target defect detection method and detection device, computer equipment, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the target defect detection method in the embodiments of the present application is described.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The target defect detection method provided in the embodiment of the present application relates to the field of artificial intelligence. The target defect detection method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services; the software can be an application that implements the target defect detection method, etc., but is not limited to the above forms.

[0071] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] Please refer to Figure 1 , Figure 1 is an optional flow chart of the target defect detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S130.

[0073] Step S110, obtaining a BGA image to be inspected, wherein the BGA image includes at least one candidate inspection sub-image;

[0074] Step S120, using a Mosaic function to perform a data enhancement operation on each candidate detection sub-image to obtain at least one target defect image in the BGA image;

[0075] Step S130 , performing defect detection on each target defect image according to the pre-trained FPN architecture model, and outputting a target defect category and a target defect probability value corresponding to each target defect image.

[0076] In step S110 of some embodiments, when defect detection is performed on a target object with relatively small defects, since the corresponding defects are also relatively small, it causes a lot of difficulties in target defect detection. In order to improve the accuracy of small target defect detection, first, a BGA image to be detected is obtained, and the BGA image includes at least one candidate detection sub-image.

[0077] It should be noted that the acquisition of the BGA image to be detected can be performed by performing initial image acquisition on a target object including multiple small targets to be detected, and then performing image preprocessing on the initial image to obtain the BGA image to be detected, and the BGA image includes at least one candidate detection sub-image. Specifically, for example, since the size of the solder ball itself is small, some defects of the solder ball, including solder ball wear, solder ball burrs, solder ball scratches, etc., are smaller, so the required initial image, i.e., the solder ball photo, can be obtained by taking an image acquisition device such as an electron microscope. By performing image preprocessing on the solder ball photo, for example, image preprocessing operations such as image inversion, image translation, image segmentation, and image brightness enhancement can be performed on the solder ball photo to obtain the BGA image to be detected corresponding to the solder ball photo, and the BGA image includes at least one candidate detection sub-image, each of which corresponds to a target object to be detected in the solder ball photo, that is, equivalent to a solder ball.

[0078] It should be noted that the initial image may be a binary image, a grayscale image, an RGB three-channel image, etc. The embodiment of the present application does not limit the specific form of the collected initial image.

[0079] In step S120 of some embodiments, in order to improve the recognition accuracy of small target defect detection of solder balls, a data enhancement operation is performed on each candidate detection sub-image using a Mosaic function to obtain at least one target defect image in the BGA image. Specifically, a data enhancement model is obtained using the Mosaic function, and a data enhancement operation is performed on each candidate detection sub-image using the data enhancement model to obtain at least one target defect image in the BGA image.

[0080] It should be noted that the data enhancement method corresponding to the Mosaic function is mainly to randomly crop four pictures and then splice them into one picture as training data, thereby enriching the background of the picture. While obtaining a new picture, the corresponding recognition frame in this picture is also obtained. Then such a new picture is passed into the neural network for learning, thereby improving the recognition accuracy of the data enhancement model. The embodiment of the present application performs data enhancement operation on each candidate detection sub-image through the data enhancement model to obtain at least one target defect image in the BGA image, that is, at least one target defect image to be detected marked with a recognition frame is obtained in the BGA image.

[0081] In step S130 of some embodiments, in order to improve the accuracy and efficiency of small target defect detection of solder balls, defect detection is performed on each target defect image according to the pre-trained FPN architecture model, and the target defect category and target defect probability value corresponding to each target defect image are output.

[0082] It should be noted that the target defect categories of solder balls are mainly divided into two categories: solder ball size defects, solder ball damage defects and solder ball foreign matter defects. Among them, solder ball size defects mainly include: solder ball size defects, solder ball arrangement position defects, solder ball spacing defects, solder ball coplanarity defects, etc.; solder ball damage defects mainly include: solder ball flattening, solder ball wear, solder ball scratches, solder ball shedding, solder ball oxidation, solder ball shape defects, etc.; solder ball foreign matter defects mainly include: solder balls will be stained with impurities, or foreign hair will be adsorbed during transportation. Specifically, solder ball size defects can include solder ball height, volume, surface area and other defects; solder ball arrangement position defects are that the position of the solder ball cannot be offset; solder ball spacing defects are that the spacing between solder balls cannot exceed the size specification.

[0083] Please refer to Figure 2 In some embodiments, in order to improve the efficiency of target defect recognition, the embodiment of the present application performs target defect detection on the candidate detection sub-images in the BGA image according to the pre-trained FPN architecture model. Figure 2 is an optional training flow chart of the FPN architecture model provided in the embodiment of the present application. Figure 2 The training method may include but is not limited to steps S210 to S270.

[0084] Step S210, constructing a training sample set, where the training sample set includes a plurality of defect sample images;

[0085] Step S220, performing a downsampling operation on each defect sample image to obtain a second defect feature map having a plurality of different feature sizes corresponding to the defect sample image;

[0086] Step S230, constructing a multi-layer downsampling structure according to a plurality of second defect feature maps of different feature sizes;

[0087] Step S240, performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes;

[0088] Step S250, constructing a multi-layer upsampling structure according to a plurality of third defect feature maps of different sizes;

[0089] Step S260, performing a downsampling operation on each third defect feature map according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes;

[0090] Step S270, performing model training on the FPN architecture model according to each fourth defect feature map and the preset defect category until the precise value of the model training meets the preset training end condition to obtain the FPN architecture model.

[0091] In step S210 of some embodiments, in order to improve the efficiency of target defect recognition, the pre-trained FPN architecture model used in the present application is to perform a downsampling operation-upsampling operation-downsampling operation on the input image to be detected, thereby achieving parameter aggregation of different detection layers in the model from different backbone layers, and finally achieving the detection accuracy of small target defects of solder balls, greatly improving the detection rate and reducing the missed detection rate. Specifically, the training process is to first construct a training sample set, which includes multiple defect sample images.

[0092] In step S220 and step S230 of some embodiments, since the size of the feature map obtained after feature extraction of the initial defect sample image is large and contains strong position information and weak semantic feature information, a downsampling operation is performed on each defect sample image, that is, the feature map corresponding to each defect sample image is convolved to obtain a second defect feature map of multiple different feature sizes corresponding to the defect sample image. Among them, defect feature maps with the same feature size are classified into one layer, and a multi-layer downsampling structure is constructed based on multiple second defect feature maps with different feature sizes. Among them, the current downsampling structure is equivalent to the first feature pyramid, the initial defect sample image is located in the shallow layer of the first feature pyramid, and the defect feature map corresponding to the deep layer of the first feature pyramid is obtained after multiple convolutions are performed on each defect sample image, that is, the first feature pyramid realizes the sampling process from bottom to top for each defect sample image.

[0093] It should be noted that when downsampling each defect sample image, the backbone network of the feature selection network in the FPN architecture model is not limited to scaling the image by several times, and can be flexibly adjusted according to needs.

[0094] In step S240 and step S250 of some embodiments, since the deep feature map contains weaker position information and stronger semantic feature information, in order to enhance the semantic feature information corresponding to multiple size feature maps, each second defect feature map is upsampled according to the multi-layer downsampling structure to obtain multiple third defect feature maps of different sizes. Specifically, each second defect feature map is feature fused with a defect feature map of the same size in the first feature pyramid to obtain multiple third defect feature maps of different sizes. Therefore, the semantic feature information of the deep layer is transmitted to the defect feature map corresponding to the shallow layer, thereby enhancing the semantic feature information corresponding to the multiple size feature maps. A multi-layer upsampling structure is constructed based on the obtained multiple third defect feature maps of different sizes. Among them, the current upsampling structure is equivalent to the second feature pyramid, the initial second defect feature map is located in the deep layer of the second feature pyramid, and after multiple convolutions are performed on the second defect feature map corresponding to each defect sample image, the defect feature map corresponding to the shallow layer in the second feature pyramid is obtained, that is, the second feature pyramid realizes the sampling process from top to bottom for each defect sample image.

[0095] In step S260 of some embodiments, since the second feature pyramid only transmits the semantic feature information with stronger depth, in order to improve the transmission of higher position information and improve the positioning of defects in the image, each third defect feature map is downsampled according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes. Specifically, each third defect feature map is feature fused with a defect feature map of the same size as the feature map in the second feature pyramid to obtain a plurality of fourth defect feature maps of different sizes. Therefore, the position information with stronger depth is transmitted to the defect feature map corresponding to the deep layer, thereby enhancing the position information corresponding to the plurality of size feature maps to improve the positioning and detection capability of small target defects.

[0096] It should be noted that a multi-layer upsampling structure is constructed based on a plurality of fourth defect feature maps of different feature sizes. Among them, the current upsampling structure is equivalent to the third feature pyramid, the initial third defect feature map is located in the shallow layer of the third feature pyramid, and the defect feature map corresponding to the deep layer of the third feature pyramid is obtained after multiple convolution processes are performed on each third defect feature map, that is, the third feature pyramid realizes the sampling process of each third defect feature map from bottom to top. At the same time, the prediction of defects of different sizes can be realized according to the feature images corresponding to different layers in the third feature pyramid.

[0097] In step S270 of some embodiments, in order to improve the efficiency of the FPN architecture model in identifying defects, the FPN architecture model is trained according to each fourth defect feature map and the preset defect category until the precise value of the model training meets the preset training end condition to obtain the FPN architecture model.

[0098] Please refer to Figure 3 In some embodiments, step S240 specifically includes but is not limited to steps S310 to S330.

[0099] Step S310, obtaining the second defect feature map of the deepest layer in the multi-layer downsampling structure, where the second defect feature map of the deepest layer is obtained by performing a convolution operation on the second defect feature map of the previous layer corresponding to the second defect feature map of the deepest layer;

[0100] Step S320, performing feature fusion on the second defect feature map of the previous layer and the second defect feature map of the deepest layer to obtain a third defect feature map having a feature size equal to that of the second defect feature map of the previous layer;

[0101] Step S330, replacing the obtained third defect feature map with the second defect feature map of the deepest layer, and performing feature fusion with the second defect feature map of the previous layer corresponding to the current deepest second defect feature map, to obtain a plurality of third defect feature maps of different sizes.

[0102] In some embodiments, in steps S310 to S330, in order to enhance the semantic feature information corresponding to multiple size feature maps, a multi-layer downsampling structure is obtained, that is, the second defect feature map of the deepest layer in the first feature pyramid is obtained, and the second defect feature map of the deepest layer is obtained by convolution operation of the second defect feature map of the previous layer corresponding to the second defect feature map of the deepest layer in the first feature pyramid. In order to ensure the height and width of the expanded feature image, the upsampling operation samples the horizontal addition fusion, and the horizontal connection is to fuse the upsampling result with the feature map of the same size generated in the first feature pyramid, that is, the second defect feature map of the previous layer and the second defect feature map of the deepest layer are feature fused to obtain a third defect feature map with the same feature size as the second defect feature map of the previous layer. In the continuous upsampling process, the obtained third defect feature map is replaced with the second defect feature map of the deepest layer, and the second defect feature map of the previous layer corresponding to the current second defect feature map of the deepest layer is feature fused to obtain multiple third defect feature maps of different sizes. Therefore, the semantic feature information of the deep layer is transmitted to the defect feature map corresponding to the shallow layer, thereby enhancing the semantic feature information corresponding to multiple size feature maps.

[0103] It should be noted that the specific multiple used in the upsampling operation is not specifically limited and can be selected and adjusted according to the situation.

[0104] It should be noted that in order to eliminate the aliasing effect caused by the upsampling operation, after feature fusion, a set 3*3 convolution kernel can be used to convolve each fusion result.

[0105] It should be noted that when performing feature fusion on the second defect feature map of the previous layer and the second defect feature map of the deepest layer, in order to reduce the dimension of the fusion channel, the second defect feature map of the previous layer is first subjected to the corresponding convolution operation, and the second defect feature map of the deepest layer is subjected to the feature method so that the feature map at this time has the same size as the second defect feature map of the previous layer. Then, the second defect feature map of the previous layer after the convolution operation and the enlarged second defect feature map of the deepest layer are feature fused to obtain a third defect feature map with the same feature size as the second defect feature map of the previous layer.

[0106] It should be noted that since the second feature pyramid only transmits the deeper and stronger semantic feature information, in order to improve the transmission of higher position information and improve the positioning of defects in the image, each third defect feature map is downsampled according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes. Specifically, obtaining the multi-layer upsampling structure is to obtain the deepest third defect feature map in the second feature pyramid, and the deepest third defect feature map is obtained by convolution operation of the third defect feature map of the previous layer corresponding to the deepest third defect feature map in the second feature pyramid. The downsampling operation also samples the horizontal addition fusion, and the horizontal connection is to fuse the downsampling result with the feature map of the same size generated in the second feature pyramid, that is, to fuse the third defect feature map of the previous layer and the third defect feature map of the shallowest layer to obtain a fourth defect feature map with the same feature size as the third defect feature map of the previous layer. In the continuous downsampling process, the obtained fourth defect feature map is replaced by the shallowest third defect feature map, and the third defect feature map of the previous layer corresponding to the current shallowest third defect feature map is feature fused to obtain multiple fourth defect feature maps of different sizes.

[0107] Please refer to Figure 4 In some embodiments, step S130 specifically includes but is not limited to step S410 and step S420.

[0108] Step S410, performing defect detection on each target defect image according to the pre-trained FPN architecture model to obtain a candidate defect probability value corresponding to each target defect image under a candidate defect category;

[0109] Step S420 , numerically comparing multiple candidate defect probability values ​​to obtain a target defect category and a target defect probability value corresponding to each target defect image.

[0110] In step S410 and step S420 of some embodiments, in order to improve the efficiency and accuracy of target defect recognition, defect detection is performed on each target defect image according to the pre-trained FPN architecture model to obtain the candidate defect probability value corresponding to each target defect image under the candidate defect category. Specifically, when defect detection is performed on each target defect image according to the pre-trained FPN architecture model, the candidate defect probability value corresponding to each target defect image under the candidate defect category is output, and a plurality of candidate defect probability values ​​are numerically compared to select the target that actually best matches the target defect image to be detected, that is, the target defect category and target defect probability value corresponding to each target defect image are obtained. Because the target defect category and the corresponding target defect probability value of the solder ball are both identified through the network corresponding to the model, the efficiency and accuracy of target defect recognition are improved.

[0111] In some embodiments, the target defect detection method of the embodiment of the present application further includes: marking the target defect category and the target defect probability value on the corresponding target defect image in the BGA image.

[0112] Specifically, in some embodiments, in order to intuitively obtain the target defect category and target defect probability value corresponding to the target image to be detected, the obtained target defect category and target defect probability value are marked on the corresponding target defect image in the BGA image.

[0113] In a specific embodiment, please refer to Figure 5First, a BGA image 510 to be detected is obtained, and the BGA image 510 includes at least one candidate detection sub-image. A data enhancement operation is performed on each candidate detection sub-image using a Mosaic function to obtain at least one target defect image 520 in the BGA image 510. The pre-trained FPN architecture model includes a first feature pyramid 530, a second feature pyramid 540, and a third feature pyramid 550. Defect detection is performed on each target defect image 520 according to the pre-trained FPN architecture model. Specifically, the initial target defect image 520 is continuously downsampled in the first feature pyramid 530 to obtain a second defect feature map corresponding to a plurality of different feature sizes of the defect sample image, for example, including second defect feature maps with feature sizes of 512*512, 128*128, and 64*64 obtained in sequence. Then the 64*64 second defect feature map is the deepest second defect feature map. By performing feature fusion on the 128*128 second defect feature map in the first feature pyramid 530 and the deepest second defect feature map, a third defect feature map with a feature size of 128*128 in the second feature pyramid 540 is obtained. By further downsampling the 128*128 third defect feature map, a third defect feature map with a feature size of 512*512 is obtained, thereby constructing the second feature pyramid 540. Since the shallow feature map contains strong position information and weak semantic feature information, the 512*512 third defect feature map in the second feature pyramid 540 is the shallowest third defect feature map, and the third defect feature map of the previous layer corresponding to the current shallowest third defect feature map is feature fused to obtain a plurality of fourth defect feature maps of different sizes. For example, the third defect feature maps with feature sizes of 512*512, 128*128, and 64*64 obtained in sequence are included to construct the third feature pyramid 550. According to the images of different sizes in the third feature pyramid 550, the shallower position information can be transmitted to the deeper layer to enhance the positioning capability at multiple scales. For example, the feature map with a feature size of 512*512 in the shallow layer of the third feature pyramid 550 can predict smaller defects in the solder ball, such as burrs, scratches, etc., the feature map with a feature size of 128*128 can predict some medium defects in the solder ball, such as solder ball foreign matter, etc., and the feature map with a feature size of 64*64 can predict larger defects in the solder ball, such as solder ball shedding, etc., that is, by performing defect detection on each target defect image 520, the target defect category and target defect probability value corresponding to each target defect image 520 are output. Therefore, the embodiment of the present invention adds a multi-layer upsampling fusion layer to the traditional FPN architecture model, that is, through the downsampling-upsampling-downsampling model architecture, the shallow information containing more details is transmitted upward, the receptive field area is expanded, and the feature map corresponding to the input target defect image is amplified twice, thereby improving the efficiency and accuracy of target defect recognition.

[0114] The present application also provides a target defect detection device, please refer to Figure 6 , the above-mentioned target defect detection method can be implemented. The device includes: an image acquisition module 610, a data enhancement module 620 and a defect detection module 630.

[0115] An image acquisition module 610 is used to acquire a BGA image to be inspected, wherein the BGA image includes at least one candidate inspection sub-image;

[0116] A data enhancement module 620 is used to perform a data enhancement operation on each candidate detection sub-image using a Mosaic function to obtain at least one target defect image in the BGA image;

[0117] The defect detection module 630 is used to perform defect detection on each target defect image according to the pre-trained FPN architecture model, and output the target defect category and target defect probability value corresponding to each target defect image.

[0118] The target defect detection device proposed in the embodiment of the present application obtains at least one candidate detection sub-image in the BGA image by acquiring a BGA image to be detected. In order to improve the accuracy of defect detection on the image, a Mosaic function is used to perform data enhancement operations on each candidate detection sub-image to obtain at least one target defect image in the BGA image. Defect detection is performed on each target defect image according to the pre-trained FPN architecture model, and the target defect category and target defect probability value corresponding to each target defect image are output. The embodiment of the present application performs target defect detection on the candidate detection sub-images in the BGA image according to the pre-trained FPN architecture model, which can improve the accuracy and efficiency of small target defect detection of solder balls.

[0119] The target defect detection device of the embodiment of the present application is used to execute the target defect detection method in the above embodiment. Its specific processing process is the same as the target defect detection method in the above embodiment, and will not be repeated here.

[0120] In some embodiments, the target defect detection device provided by the embodiments of the present application further includes:

[0121] FPN architecture model module, the FPN architecture model corresponding to the FPN architecture model module is trained by the following method:

[0122] Constructing a training sample set, wherein the training sample set includes a plurality of defect sample images;

[0123] Performing a downsampling operation on each defect sample image to obtain a second defect feature map having a plurality of different feature sizes corresponding to the defect sample image;

[0124] Constructing a multi-layer downsampling structure according to a plurality of second defect feature maps with different feature sizes;

[0125] Performing an upsampling operation on each second defect feature map according to a multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes;

[0126] Constructing a multi-layer upsampling structure according to a plurality of third defect feature maps of different sizes;

[0127] Performing a downsampling operation on each third defect feature map according to a multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes;

[0128] The FPN architecture model is trained according to each fourth defect feature map and the preset defect category until the precise value of the model training meets the preset training end condition to obtain the FPN architecture model.

[0129] In some embodiments, the defect detection module 630 includes:

[0130] The defect detection submodule is used to perform defect detection on each target defect image according to the pre-trained FPN architecture model to obtain the candidate defect probability value corresponding to each target defect image under the candidate defect category;

[0131] The numerical comparison module is used to perform numerical comparison on multiple candidate defect probability values ​​to obtain the target defect category and target defect probability value corresponding to each target defect image.

[0132] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the method in the above embodiment of the present application.

[0133] Combine the following Figure 7 The hardware structure of the computer device is described in detail. The computer device includes: a processor 710 , a memory 720 , an input / output interface 730 , a communication interface 740 and a bus 750 .

[0134] The processor 710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0135] The memory 720 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 720 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 720, and the processor 710 calls and executes the target defect detection method of the embodiments of this application;

[0136] Input / output interface 730, used to implement information input and output;

[0137] Communication interface 740, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0138] bus 750 , which transmits information between the various components of the device (e.g., processor 710 , memory 720 , input / output interface 730 , and communication interface 740 );

[0139] The processor 710 , the memory 720 , the input / output interface 730 , and the communication interface 740 are connected to each other in communication within the device via the bus 750 .

[0140] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the target defect detection method of the embodiment of the present application.

[0141] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0143] It can be understood by those skilled in the art that Figures 1 to 5 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0144] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0146] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0147] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0152] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target defect detection method, characterized in that: The method comprises: Acquire a BGA image to be detected, wherein the BGA image includes at least one candidate detection sub-image, and each of the candidate detection sub-images corresponds to a solder ball to be detected; Using a Mosaic function to perform a data enhancement operation on each candidate detection sub-image to obtain at least one target defect image in the BGA image; Perform defect detection on each target defect image according to the pre-trained FPN architecture model, and output the target defect category and target defect probability value corresponding to each target defect image; The FPN architecture model is trained by the following method: Constructing a training sample set, wherein the training sample set includes a plurality of defect sample images; Performing a downsampling operation on each defect sample image to obtain a second defect feature map having a plurality of different feature sizes corresponding to the defect sample image; Constructing a multi-layer downsampling structure according to the plurality of second defect feature maps with different feature sizes; Performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes; Constructing a multi-layer upsampling structure according to the plurality of third defect feature maps of different sizes; Performing a downsampling operation on each third defect feature map according to the multi-layer upsampling structure to obtain a plurality of fourth defect feature maps of different sizes; The FPN architecture model is trained according to each fourth defect feature map and a preset defect category until the precise value of the model training meets a preset training end condition to obtain the FPN architecture model, wherein each of the fourth defect feature maps is used to predict defects of different sizes in solder balls.

2. The method according to claim 1, characterized in that The upsampling operation is performed on each second defect feature map according to the multi-layer downsampling structure to obtain a plurality of third defect feature maps of different sizes, including: Obtaining a second defect feature map in the deepest layer of the multi-layer downsampling structure, wherein the second defect feature map in the deepest layer is obtained by performing a convolution operation on a second defect feature map in a previous layer corresponding to the second defect feature map in the deepest layer; Performing feature fusion on the second defect feature map of the previous layer and the second defect feature map of the deepest layer to obtain a third defect feature map having a feature size equal to that of the second defect feature map of the previous layer; The obtained third defect feature map is replaced with the second defect feature map of the deepest layer, and feature fusion is performed with the second defect feature map of the previous layer corresponding to the current second defect feature map of the deepest layer to obtain multiple third defect feature maps of different sizes.

3. The method according to any one of claims 1 to 2, characterized in that: The method of performing defect detection on each target defect image according to the pre-trained FPN architecture model and outputting a target defect category and a target defect probability value corresponding to each target defect image includes: Perform defect detection on each target defect image according to the pre-trained FPN architecture model to obtain a candidate defect probability value corresponding to each target defect image under a candidate defect category; Numerical comparison is performed on the plurality of candidate defect probability values ​​to obtain a target defect category and a target defect probability value corresponding to each target defect image.

4. The method according to claim 3, characterized in that: The method further comprises: The target defect category and the target defect probability value are marked on the corresponding target defect image in the BGA image.

5. A target defect detection device, characterized in that: The device comprises: An image acquisition module, used to acquire a BGA image to be detected, wherein the BGA image includes at least one candidate detection sub-image; each of the candidate detection sub-images corresponds to a solder ball to be detected; A data enhancement module, used for performing a data enhancement operation on each candidate detection sub-image using a Mosaic function to obtain at least one target defect image in the BGA image; A defect detection module is used to perform defect detection on each target defect image according to a pre-trained FPN architecture model, and output a target defect category and a target defect probability value corresponding to each target defect image; FPN architecture model module, the FPN architecture model corresponding to the FPN architecture model module is trained by the following method: constructing a training sample set, the training sample set includes multiple defect sample images; performing a downsampling operation on each defect sample image to obtain a second defect feature map of multiple different feature sizes corresponding to the defect sample image; constructing a multi-layer downsampling structure according to the multiple second defect feature maps with different feature sizes; performing an upsampling operation on each second defect feature map according to the multi-layer downsampling structure to obtain a third defect feature map of multiple different sizes; constructing a multi-layer upsampling structure according to the multiple third defect feature maps of different sizes; performing a downsampling operation on each third defect feature map according to the multi-layer upsampling structure to obtain a fourth defect feature map of multiple different sizes; performing model training on the FPN architecture model according to each fourth defect feature map and a preset defect category until the precise value of the model training meets the preset training end condition to obtain the FPN architecture model, wherein each of the fourth defect feature maps is used to predict defects of different sizes in solder balls.

6. The device according to claim 5, characterized in that The defect detection module comprises: A defect detection submodule is used to perform defect detection on each target defect image according to the pre-trained FPN architecture model to obtain a candidate defect probability value corresponding to each target defect image under a candidate defect category; The numerical comparison module is used to perform numerical comparison on the plurality of candidate defect probability values ​​to obtain the target defect category and the target defect probability value corresponding to each target defect image.

7. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute: The method according to any one of claims 1 to 4.

8. A storage medium, the storage medium being a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is configured to perform: The method according to any one of claims 1 to 4.

Images