Neural network training method, related method, device, terminal and storage medium

Abstract

The present application discloses a neural network training method, a sharp corner detection method, a defect detection method, an optical detection device, an intelligent terminal, and a computer-readable storage medium. The neural network training method comprises: obtaining a training set including a plurality of images to be trained, wherein the images to be trained include the annotation information of the sharp corners in the circuit board; inputting the training set into the neural network for training, and obtaining a neural network for detecting the sharp corners in the circuit board image. The above method uses the sharp corners of the circuit board and the annotation information of the sharp corners in the input training set to train the neural network for detecting the sharp corners, which can improve the accuracy and efficiency of the sharp corners in the circuit board image detected by the neural network.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a neural network training method, related methods, equipment, terminal and storage medium. Background Art

[0002] With the widespread application of automotive electronics, communication equipment, transformers, inductors and power modules in life and the rapid development of electronic information technology and communication technology, the market has put forward higher requirements for high-transmission and high-voltage electronic products. As the basic carrier of electronic components, the performance of printed circuit boards (PCBs) directly affects the performance of products after the electronic components are installed.

[0003] Since there are inevitably a large number of defects in the PCB production process, and the existing defects are mainly located at the sharp corners of the PCB. Therefore, it is necessary to detect the sharp corners with defects, so as to carry out subsequent PCB repair work according to the detection results; however, the sharp corners on the PCB circuit are very small and dense, and manual marking will be very time-consuming and labor-intensive, and prone to errors. If the sharp corners are not detected and the defective sharp corners are directly put into the next process for manufacturing, the subsequent PCB repair cost will become higher and higher, and it will be more likely to be scrapped, resulting in a lot of waste costs. Summary of the invention

[0004] To solve the above problems, the present application provides a neural network training method, related methods, devices, terminals and storage media, which can improve the accuracy and efficiency of sharp corners in circuit board images detected by neural networks.

[0005] A technical solution adopted in the present application is: a neural network training method, the method comprising: obtaining a training set including a number of images to be trained, wherein the images to be trained include annotation information of sharp corners in a circuit board; inputting the training set into a neural network for training, and obtaining a neural network for detecting sharp corners in circuit board images.

[0006] Optionally, the neural network training method further includes: obtaining a sharp corner template; using the sharp corner template to traverse the circuit board area in the image to be trained, and marking the sharp corner area in the image to be trained that matches the sharp corner template.

[0007] Optionally, after obtaining a training set including several images to be trained, the neural network training method further includes: performing one or more image processing of flipping, rotating, enlarging, reducing, and chromaticity adjustment on the several images to be trained to obtain several extended images; and updating the training set using the several extended images.

[0008] Optionally, the neural network is iteratively trained using a sample-balanced loss function.

[0009] Another technical solution adopted in the present application is: to provide a method for detecting sharp corners, which comprises: obtaining an image to be detected of a circuit board; inputting the image to be detected into a pre-trained neural network to obtain detection information of sharp corners in the image to be detected; wherein the pre-trained neural network is trained by the neural network training method as described above.

[0010] Optionally, after acquiring the detection information of the sharp corners in the image to be detected, the sharp corner detection method further includes: using the detection information of the sharp corners to generate a labeling box of the sharp corners and its confidence level in the image to be detected.

[0011] Another technical solution adopted in the present application is: to provide a defect detection method, which includes: obtaining a design drawing and a captured image of a circuit board; obtaining an area where a sharp corner is located in the captured image; matching the design drawing and the captured image in an area outside the area where the sharp corner is located; and outputting defect information according to the matching result of the design drawing and the captured image; wherein the area where the sharp corner is located is obtained by detection through a pre-trained neural network.

[0012] Optionally, the design drawing and the captured image are matched in an area outside the area where the sharp corners are located, including: obtaining a mapping matrix between the design drawing and the captured image; mapping the area where the sharp corners are located on the captured image to the design drawing based on the mapping matrix to generate a sharp corner mask; and using the area outside the sharp corner mask in the design drawing to match the captured image.

[0013] Another technical solution adopted in the present application is: to provide an optical inspection device, which includes: an image acquisition module, used to acquire the design drawing and collected image of the circuit board; a region extraction module, used to acquire the area where the sharp corner is located in the collected image; wherein the area where the sharp corner is located is obtained by detection of a pre-trained neural network; a region matching module, used to match the design drawing and the collected image in the area outside the area where the sharp corner is located; a defect output module, used to output defect information according to the matching results of the design drawing and the collected image.

[0014] Another technical solution adopted in the present application is: to provide an intelligent terminal, which includes: a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls the program data stored in the memory to execute the neural network training method, sharp corner detection method or defect detection method as described above.

[0015] Another technical solution adopted in the present application is: providing a computer-readable storage medium, in which program data is stored. When the program data is executed by a processor, it is used to implement the neural network training method, sharp corner detection method or defect detection method as described above.

[0016] Different from the prior art, the neural network training method provided by the present application includes: obtaining a training set including a number of images to be trained, wherein the images to be trained include the annotation information of the sharp corners in the circuit board; inputting the training set into the neural network for training to obtain a neural network for detecting the sharp corners in the circuit board image. Through the above-mentioned neural network training method, the sharp corners of the circuit board and the annotation information of the sharp corners in the input training set are used to train the neural network for detecting the sharp corners, which can improve the accuracy and efficiency of the sharp corners in the circuit board image detected by the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0018] Figure 1 is a structural schematic diagram of an embodiment of an optical detection device provided by the present application;

[0019] Figure 2 It is a flow chart of an embodiment of a defect detection method provided by the present application;

[0020] Figure 3 It is a schematic diagram of the process of matching design drawings and captured images in an embodiment of the present application;

[0021] Figure 4 It is a flowchart of an embodiment of making a mapping matrix corresponding to a design drawing and a captured image in the present application;

[0022] Figure 5 It is a flowchart of an embodiment of a neural network training method provided by the present application;

[0023] Figure 6 It is a flow chart of an embodiment of a circuit board area marking method provided by the present application;

[0024] Figure 7 This is a schematic diagram of a process for obtaining a sharp corner template in an embodiment of the present application;

[0025] Figure 8 This is a schematic diagram of an interface of an embodiment of a sharp-angle area corresponding to an operation instruction marked in the present application;

[0026] Fig. 9 It is a schematic diagram of a process of establishing a sharp corner template in an embodiment of the present application;

[0027] Fig.10 It is a flowchart of an embodiment of updating a training set in the present application;

[0028] Fig.11 It is a flowchart of an embodiment of updating a training set using a plurality of extended images in the present application;

[0029] Fig.12 It is a flowchart of a sharp corner detection method provided by the present application;

[0030] Fig.13 It is a structural schematic diagram of a smart terminal provided by this application;

[0031] Fig.14 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0033] Reference to "embodiments" in an application means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] The steps in the embodiments of the present application do not necessarily have to be processed in the described order of steps. The steps can be selectively rearranged, or the steps in the embodiments can be deleted, or the steps in the embodiments can be added as needed. The step descriptions in the embodiments of the present application are only optional sequence combinations and do not represent all step sequence combinations in the embodiments of the present application. The order of steps in the embodiments cannot be considered as a limitation of the present application.

[0035] The term "and / or" in the embodiments of the present application refers to any and all possible combinations of one or more of the associated enumerated items. It should also be noted that when used in this specification, "include / comprise" specifies the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components and / or their groups.

[0036] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] In addition, although the terms "first", "second", etc. are used many times in this application to describe various operations (or various components or various applications or various instructions or various data), etc., these operations (or components or applications or instructions or data) should not be limited by these terms. These terms are only used to distinguish one operation (or component or application or instruction or data) from another operation (or component or application or instruction or data). For example, the first circuit element template can be called the second circuit element template, and the second circuit element template can also be called the first circuit element template. It is just that the scopes included by the two are different, and it does not deviate from the scope of this application. The first circuit element template and the second circuit element template are both sets of various circuit element templates, but the two are not sets of the same circuit element templates.

[0038] See also Figure 1 , Figure 1 1 is a schematic diagram of the structure of an embodiment of an optical inspection device provided in the present application. The optical inspection device 10A includes: an image acquisition module 11A, a region extraction module 12A, a region matching module 13A and a defect output module 14A.

[0039] In one embodiment, the optical inspection device 10A is an automated optical inspection device (AOI), which is a device that detects common defects encountered in welding production based on optical principles. During automatic inspection, the AOI machine automatically scans the PCB through a camera, collects images, compares the tested solder joints with qualified parameters in the database, and detects defects on the PCB through image processing, and displays / marks the defects through a display or automatic marking for maintenance personnel to repair.

[0040] Optionally, AOI's main frequency CPU (Central Processing Unit / Processor) is an Intel Core i5 or higher configuration processor; the memory is 8GB or higher configuration memory; the hard disk has more than 10G free disk space; the graphics card is GeForce RTX 3070Ti 8G or higher configuration video memory; the network card is a gigabit network card; the display device is a monitor that supports 1280x 1024 resolution; the software environment of the application software is Pytorch; the operating system is Microsoft Windows 10 Ultimate or Professional Edition.

[0041] In one embodiment, the image acquisition module 11A is used to acquire a design drawing of a circuit board and collect images.

[0042] Optionally, the image acquisition module 11A may be equipped with an image acquisition device such as a depth camera, a 3D camera, a monocular camera or a binocular camera, which may generate corresponding control information according to user input to acquire the captured image of the circuit board.

[0043] Optionally, the captured image of the circuit board is a PCB (Printed circuit boards) image. PCB is also called a printed circuit board, and its board surface is divided into a circuit board area and a non-circuit board area. PCB can be applied to many electronic components, including mobile terminals such as cameras and video recorders, mobile phones, smart phones, notebook computers, personal digital assistants (PDAs), tablet computers (PADs), etc., and can also be fixed terminals such as digital broadcast transmitters, digital TVs, desktop computers, servers, etc.

[0044] Among them, the circuit board area of ​​the PCB has a PCB circuit etched by a chemical, and on the PCB circuit are small and dense circuit elements and their sharp corners (that is, the sharp corner area protrudes outward from the four right-angled parts of the square area, and the two adjacent straight lines in the square area form mutually intersecting inclined lines at the ends close to each other, and the two adjacent inclined lines intersect each other to form a sharp corner; the inclined line is called a sharp corner line segment). Due to the tension of the chemical, a large number of defects are inevitably present in the production process of the PCB (due to the manufacturing process, defects such as missing holes, mouse-type corrosion, open circuits, short circuits, burrs, copper slag, etc. are inevitable in the production process of the circuit board), and the sharp corners on the PCB circuit are often false point sharp corners (due to the shiny openings of the sharp corners, the reflection of the substrate, local oxidation and dirty spots, etc., there will be more false points on the circuit elements).

[0045] Among them, the design drawing of the circuit board is the principle design drawing corresponding to the PCB. Compared with the PCB, there are no defects on the board surface and no false points or sharp corners at the sharp corners. It is the PCB board in the most ideal state.

[0046] In one embodiment, the region extraction module 12A is used to obtain the region where the sharp corner is located in the captured image; wherein the region where the sharp corner is located is obtained by detection using a pre-trained neural network.

[0047] Optionally, the area where the sharp corner is located is a sharp corner marked area that is pre-made for one or more types of PCBs and stored in the optical inspection device 10A or output by a third-party organization (such as a neural network model, a digital processing platform, a cloud server, an external terminal, etc.).

[0048] The outputted sharp corner annotation area includes the spatial shape (including plane shape and curved surface shape), position, size, type, purpose and other information of the corresponding sharp corner.

[0049] Optionally, the region extraction module 12A may input the captured image into a pre-trained neural network for sharp corner detection and labeling to obtain an output sharp corner labeling region.

[0050] Optionally, the region extraction module 12A may also extract at least one corresponding sharp corner marked region in the captured image based on the user's input.

[0051] For example, based on the user's expectation corresponding to the user input to extract X sharp-cornered marked areas of A position, B size, C type and D purpose in the acquired image, the area extraction module 12A executes extraction of the corresponding sharp-cornered marked areas in the acquired image in response to the user input.

[0052] In one embodiment, the area matching module 13A is used to match the design drawing and the captured image in an area other than the area where the sharp corner is located.

[0053] Optionally, the area matching module 13A matches the design drawing and the captured image in an area outside the area where the sharp corners are located to match an area where the design drawing and the captured image are inconsistent. This area is the defect generated during the production process of the PCB corresponding to the captured image.

[0054] Optionally, the area matching module 13A matches the design drawing and the captured image in the circuit board area outside the area where the sharp corner is located, so as to obtain the matching degree of each circuit element on the design drawing and the captured image in the circuit board area, and marks the corresponding circuit element area based on the matching degree of each circuit element. The matching degree is related to the spatial shape (including plane shape and curved surface shape), position, size and other information of the corresponding circuit element area.

[0055] In one embodiment, the defect output module 14A is used to output defect information according to the matching result between the design drawing and the captured image.

[0056] Optionally, the matching result of the design drawing and the captured image is to match the area outside the area where the sharp corner is located, where the design drawing and the captured image are inconsistent, that is, the real defect area of ​​the PCB, and the defect information corresponding to the real defect area includes the spatial shape (including plane shape and curved surface shape), position, size, type, purpose and other information of the corresponding defect area.

[0057] Different from the prior art, the optical inspection equipment provided in this embodiment includes: an image acquisition module, which is used to acquire the design drawing and the captured image of the circuit board; a region extraction module, which is used to acquire the region where the sharp corner is located in the captured image; wherein the region where the sharp corner is located is obtained by detection using a pre-trained neural network; a region matching module, which is used to match the design drawing and the captured image in the region outside the region where the sharp corner is located; and a defect output module, which is used to output defect information according to the matching result between the design drawing and the captured image. Through the above-mentioned optical inspection equipment, on the one hand, the region where the sharp corner is located in the captured image is acquired by using a pre-trained neural network, which can improve the accuracy and efficiency of capturing false point sharp corners in the PCB image. On the other hand, the design drawing and the captured image are matched in the region outside the region where the sharp corner is located, so as to optimize the process of detecting real defect areas in the PCB image and reduce the detection cost.

[0058] Optionally, by combining the above optional implementation modes and further optimizing and expanding based on the above technical solution, an implementation mode of the defect detection method provided in the present application can be obtained.

[0059] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a defect detection method provided by the present application. The method is applied to the optical detection device in the above embodiment to be executed by the optical detection device, and the method includes:

[0060] Step 11: Get the design drawing and capture image of the circuit board.

[0061] Specifically, the optical inspection device obtains the design drawing and captured image of the circuit board from its own storage medium or from a third-party organization (such as a digital processing platform, a cloud server, an external terminal, etc.).

[0062] Optionally, the optical inspection device itself may be equipped with an image acquisition device such as a depth camera, a 3D camera, a monocular camera or a binocular camera, and generate corresponding control information according to the user's input to obtain the captured image of the circuit board.

[0063] Optionally, the captured image of the circuit board is a PCB (Printed circuit boards) image, wherein the PCB surface is divided into a circuit board area and a non-circuit board area.

[0064] Step 12: Obtain the area where the sharp corner is located in the acquired image.

[0065] Optionally, the optical detection device may input the captured image into a pre-trained neural network for sharp corner detection and labeling to obtain output and labeled areas where sharp corners are located.

[0066] The outputted sharp corner annotation area includes the spatial shape (including plane shape and curved surface shape), position, size, type, purpose and other information of the corresponding sharp corner.

[0067] Optionally, the optical detection device may also perform manual labeling and extract at least one corresponding sharp-cornered labeling area in the captured image based on user input.

[0068] Specifically, the optical inspection device first crops each original captured image and uses Harris corner detection to collect the image to extract all corners. Then, a human annotator uses the labelme application to mark the area where the sharp corners are located by marking the bounding box to obtain the area where the sharp corners are located.

[0069] Among them, after the optical inspection equipment runs normally, three subfolders Crop, Imgs and Pos are created in its output folder. Among them, a folder is created in Crop with each input picture name, and each folder stores the cropped image information used for annotation in the subsequent labelme application. Imgs is a large picture with cropped images numbered. Pos saves all corner point coordinate information of all images. Among them, the original captured image is cropped because each captured image is obtained by scanning the actual board surface by the original AOI equipment camera, and its size is too large. When it is directly manually annotated or input into the neural network, the annotation will be too complicated and inconvenient for manual annotation. Therefore, in one embodiment, the captured image is evenly cropped into an image of size 224*224. The traditional Harris corner detection method is used to extract all corners in the image.

[0070] The optical inspection device imports the Crop folder generated after the raw data preprocessing into the open source software labelme to mark the sharp corners of the circuit components with rectangles. It is not necessary to mark all the sharp corners of the circuit components during the marking process, only a small number of sharp corners of different types of circuit components need to be marked. After the marking is completed, a json file with the same name is generated in the path where the marked image is located.

[0071] Optionally, the optical detection device extracts the 8*8 square area where the sharp corners are located in the manually marked bounding box, performs flipping and rotation transformations, obtains different sharp corner annotation areas, and forms a sharp corner template. Since an original captured image may be composed of multiple small plates, the positions of the small plates may be rotated, so there will be many identical elements that have been rotated and flipped. Here, a sharp corner template is expanded to achieve matching at different angles. And for each original captured image, each template match in the sharp corner template library is fully annotated, which greatly reduces the cost of manual annotation and obtains extremely accurate matching results, while avoiding missed labels and mislabeling.

[0072] Step 13: Match the design drawing and the captured image in the area other than the area where the sharp corners are located.

[0073] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of matching a design drawing and a captured image in the present application. Specifically, step 13 may include the following steps:

[0074] Step 131: Obtain a mapping matrix between the design drawing and the captured image.

[0075] Specifically, the optical inspection device can obtain the mapping matrix between the design drawing and the captured image by pre-making it through its own application and storing it in a storage medium; the optical inspection device can also directly obtain the mapping matrix between the design drawing and the captured image by receiving it from a third-party organization (such as a cloud server or a data processing platform).

[0076] See also Figure 4 , Figure 4 1 is a flow chart of an embodiment of making a mapping matrix corresponding to a design drawing and a captured image in the present application. Specifically, step 131 may include the following steps:

[0077] Step 1311: Perform homography transformation on the design drawing and the captured image.

[0078] Specifically, the homography transformation of the design drawing and the captured image is a two-dimensional projection transformation that maps points in a plane where an image is located to another plane to obtain homography coordinate points (x) corresponding to the design drawing and the captured image.

[0079] Here, plane refers to an image or a plane representation in three dimensions. For points in the plane where the image is located (e.g., points in two dimensions or three dimensions), they are represented by homogeneous coordinates. The homogeneous coordinates of a point are defined based on its scale. Therefore, the point x = [x, y, w] = [ax, ay, aw] = [x / w, y / w, 1] in the plane where the design drawing and the captured image are located both represent the same two-dimensional point.

[0080] Step 1312: Obtain a homography matrix based on the homography coordinate points.

[0081] Specifically, the homography coordinate point (x) is converted to the homography matrix H according to DLT (Direct Linear Transformation). Among them, the complete projective transformation of the design drawing or the acquired image needs to have 8 degrees of freedom. According to the corresponding homography point constraints, two equations can be written for each corresponding homography point pair, corresponding to the x and y coordinates respectively. Therefore, 4 corresponding homography point pairs are required to calculate the homography matrix H.

[0082] Among them, DLT is to first normalize the homography coordinate point (x) through the Haffine_from_points function to make its mean 0 and variance 1. Then use the corresponding homography point pairs to construct the matrix A (that is, Ah=0, where A is a matrix with twice the number of rows of corresponding point pairs). Therefore, the least squares solution of matrix A is the last row of the matrix V obtained after the matrix SVD decomposition, which is transformed to obtain the homography matrix H. Then the homography matrix H is processed and normalized, and the output is returned to obtain the homography matrix.

[0083] Step 1313: Perform affine transformation on the homography matrix to obtain a mapping matrix corresponding to the design drawing and the captured image.

[0084] Specifically, the homography matrix is ​​affine transformed by the Haffine_from_points function to obtain the mapping matrix corresponding to the design drawing and the collected image. Since the affine transformation has 6 degrees of freedom, three corresponding homography points are required to affine the matrix H. By setting the last two elements to 0, that is, h7=h8=0, the affine transformation can estimate the mapping matrix corresponding to the design drawing and the collected image through the above-mentioned DLT algorithm.

[0085] Step 132: Mapping the area where the sharp corners on the captured image are located to the design drawing based on the mapping matrix to generate a sharp corner mask.

[0086] Specifically, the optical detection device determines the corresponding mapping matrix range on the captured image based on the area where the sharp corner is located in the captured image, and maps the corresponding mapping matrix range to the design drawing to generate a mask covering the sharp corner area (i.e., a sharp corner mask) on the design drawing.

[0087] Step 133: Use the area outside the sharp corner mask in the design image to match with the captured image.

[0088] Specifically, the optical inspection device matches the design drawing and the captured image in the area outside the sharp-cornered mask in the design drawing to match the area where the design drawing and the captured image are inconsistent. This area is the defect generated in the production process of the PCB corresponding to the captured image.

[0089] Optionally, the optical inspection device matches the design drawing and the captured image in the area outside the sharp corner mask in the design drawing to obtain the matching degree of each circuit element on the design drawing and the captured image in the circuit board area, and marks the corresponding circuit element area based on the matching degree of each circuit element. The matching degree is related to the spatial shape (including plane shape and curved surface shape), position, size and other information of the corresponding circuit element area.

[0090] Step 14: Output defect information according to the matching results between the design drawing and the captured image.

[0091] Optionally, the optical inspection device outputs defect information according to the matching result of the design drawing and the collected image. The matching result of the design drawing and the collected image is to match the area where the design drawing and the collected image are inconsistent in the area outside the area where the sharp corner is located, that is, the real defect area of ​​the PCB. The defect information corresponding to the real defect area includes the spatial shape (including plane shape and curved surface shape), position, size, type, purpose and other information of the corresponding defect area.

[0092] Different from the prior art, the defect detection method provided in this embodiment includes: obtaining the design drawing and the captured image of the circuit board; obtaining the area where the sharp corner is located in the captured image; matching the design drawing and the captured image in the area outside the area where the sharp corner is located; outputting defect information according to the matching result of the design drawing and the captured image; wherein the area where the sharp corner is located is obtained by detecting a pre-trained neural network. Through the above-mentioned defect detection method, on the one hand, using a pre-trained neural network to obtain the area where the sharp corner is located in the captured image can improve the accuracy and efficiency of capturing false point sharp corners in the PCB image. On the other hand, matching the design drawing and the captured image in the area outside the area where the sharp corner is located can optimize the process of detecting real defect areas in the PCB image and reduce the detection cost.

[0093] Optionally, by combining the above optional implementation modes and further optimizing and expanding based on the above technical solution, an implementation mode of the neural network training method provided in the present application can be obtained.

[0094] See also Figure 5 , Figure 5 1 is a flow chart of an embodiment of a neural network training method provided by the present application. The method is applied to the optical detection device in the above embodiment to be executed by the optical detection device, and the method includes:

[0095] Step 21: Obtain a training set including a plurality of images to be trained, wherein the images to be trained include annotation information of sharp corners in the circuit board.

[0096] Specifically, the optical detection device obtains a training set including a plurality of images to be trained. The training set of images to be trained is a sharp corner region in the sharp corner region marked by the optical detection device that matches at least one circuit board region. The marking information of the sharp corner in the circuit board is marked by a circuit board region marking method.

[0097] See also Figure 6 , Figure 6 1 is a flow chart of an embodiment of a circuit board area marking method provided by the present application. The method is applied to the optical detection device in the above embodiment to be executed by the optical detection device, and the method includes:

[0098] Step 211: Obtain a sharp corner template.

[0099] See also Figure 7 , Figure 7 2 is a flow chart of obtaining a sharp corner template in an embodiment of the present application. Specifically, step 211 may include the following steps:

[0100] Step 2111: In response to the user's operation instruction, mark the sharp corner area corresponding to the operation instruction in the circuit board area in the image to be trained.

[0101] See also Figure 8 , Figure 8 It is a schematic diagram of an interface of an embodiment of annotating sharp-angle areas corresponding to operation instructions in the present application. Among them, a circuit board area A in the image to be trained is displayed in the interface, and there are multiple sharp-angle areas in the circuit board area A. The user inputs the corresponding annotation data to the optical detection device, or the user uses a mouse, electronic brush or touch screen input manually to mark multiple sharp-angle areas in the interface. As shown in the figure, the circuit board area P is marked with sharp-angle areas P1, sharp-angle areas P2, sharp-angle areas P3, and sharp-angle areas P4 corresponding to the operation instructions.

[0102] Step 2112: Create a sharp corner template using the sharp corner area corresponding to the operation instruction.

[0103] Specifically, the optical detection device sequentially performs feature recognition, feature segmentation, and feature extraction on the sharp-angle area marked by the user to establish a circuit component template.

[0104] See also Fig. 9 , Fig. 9 This is a schematic diagram of a process for establishing a sharp corner template in an embodiment of the present application. Specifically, the method may include the following steps:

[0105] Step 21121: Identify the marked circuit elements to perform feature segmentation on the marked circuit elements.

[0106] In one embodiment, the optical inspection device identifies the marked sharp-angle areas that need feature segmentation in corresponding directions, positions and angles based on the segmented circuit board areas, and then uses a feature extraction network (such as CNN, VGG, ResNet, etc.) to perform feature segmentation on the marked sharp-angle areas to segment out the corresponding sharp-angle areas.

[0107] Step 21122: Convert the segmented circuit elements into corresponding matrix vectors.

[0108] In one embodiment, the optical detection device converts the segmented sharp corner regions into corresponding N*S dimensional matrix vectors through a word-embedding network, and each segmented sharp corner region corresponds to a matrix vector. For example, a sharp corner region is converted into a corresponding 1*S dimensional matrix vector, where 1 represents the number of sharp corner regions and S represents the vector dimension of the matrix.

[0109] Step 21123: Input the matrix vector into the correction model to obtain the sharp corner template corresponding to the matrix vector.

[0110] In one embodiment, the pre-trained correction model may be a Transformer model, and the optical detection device inputs each matrix vector into the trained Transformer model for template correction to obtain a correction template corresponding to each segmented sharp-angle area, that is, to establish a sharp-angle area template corresponding to each segmented and labeled sharp-angle area.

[0111] Step 21124: Sort the corresponding sharp corner templates to obtain a sharp corner template set.

[0112] In one embodiment, the optical detection device inputs the sharp corner templates corresponding to all the sharp corner areas obtained into an image sorting network (such as Attention-based RNN, LSTM, etc.) to sort the sharp corner templates in the corresponding directions, positions and angles of the image to obtain a set of sharp corner templates.

[0113] In another embodiment, obtaining the sharp corner template may be obtaining a pre-stored sharp corner template; wherein the pre-stored sharp corner template includes sharp corner region information of at least one type of sharp corner region.

[0114] Specifically, the pre-stored sharp corner template can be made based on the circuit board area in the historical image to be trained, and its preparation method is similar to the above embodiment, which will not be repeated here; wherein the historical image to be trained is the image to be trained previously acquired by the optical detection device. The pre-stored sharp corner template can be pre-stored in the storage medium of the optical detection device or a third-party institution (such as a digital processing platform, a cloud server, an external terminal, etc.).

[0115] Step 212: traverse the circuit board area in the image to be trained using the sharp corner template, and mark the sharp corner area in the image to be trained that matches the sharp corner template.

[0116] Specifically, the optical detection device traverses and matches the acquired sharp corner template on the circuit board area in the training image to obtain the matching degree between all the sharp corner areas on the circuit board area and the sharp corner template, and marks the corresponding sharp corner areas based on the matching degree. The matching degree is related to the spatial shape (including plane shape and curved surface shape), position, size and other information of the corresponding sharp corner area.

[0117] Furthermore, the images to be trained may be expanded to update the training set.

[0118] See also Fig.10 , Fig.10 1 is a flow chart of an embodiment of updating a training set in the present application. Specifically, the method may include the following steps:

[0119] Step 2121: Perform one or more image processing steps of flipping, rotating, enlarging, reducing, and adjusting chromaticity on a plurality of images to be trained, to obtain a plurality of extended images.

[0120] Specifically, the optical detection device can use the image processing program carried by its own device, or the optical detection device can send the extracted several images to be trained to a third-party organization (such as an image processing platform, a cloud server, etc.) to perform one or more image processing including flipping, rotating, enlarging, reducing, and color adjustment on the several images to be trained to obtain several extended images to be trained.

[0121] Step 2122: Update the training set using several extended images.

[0122] See also Fig.11 , Fig.11 21 is a flow chart of an embodiment of updating a training set using a plurality of extended images in the present application. Specifically, step 2122 further includes the following steps:

[0123] Step 21221: Calculate the similarity between several extended images.

[0124] Among them, the method for calculating image similarity by the optical detection equipment can be based on the Euclidean distance method, black box distance method, Coxon distance method, etc. in the relevant methods, and no specific limitation is made here.

[0125] Step 21222: Eliminate extended images whose similarity is greater than or equal to a similarity threshold from a plurality of extended images.

[0126] Step 21223: Add the remaining extended images to the training set for updating.

[0127] Specifically, the optical detection device removes images in the extended image whose similarity is greater than a preset similarity threshold to reduce the redundancy of the stored circuit element template and improve the rate of matching circuit element information. The preset similarity threshold is 70%-100%, for example, 75%, 85%, 95%, etc.

[0128] Furthermore, the remaining extended images after the elimination are added to the training set to update the training set.

[0129] Step 22: Input the training set into the neural network for training, and obtain a neural network for detecting sharp corners in the circuit board image.

[0130] Among them, the neural network uses a sample balanced loss function for iterative training.

[0131] Specifically, the optical inspection equipment inputs the acquired training set into the encoding and decoding structure of the UNet convolutional neural network for training, and finally trains a stable semantic segmentation model, that is, a neural network for sharp corners in the circuit board image.

[0132] In one embodiment, the entire network of the UNet convolutional neural network is in the shape of a "U". The Unet network can be divided into two parts. One is the feature extraction part, which, like other convolutional neural networks, extracts image features through stacked convolutions and compresses feature maps through pooling. The other part is the image restoration part, which restores the compressed image through upsampling and convolution. The feature extraction part can use excellent networks, such as Resnet50, VGG, etc. Among them, if the sharp-angle area is used as a mask, its range is small, and the sample balance loss function (i.e., focal loss) is used as the loss function of the neural network to balance the positive and negative examples, and finally a stable semantic segmentation model is obtained in which the loss function of the output sharp-angle area image is no longer reduced.

[0133] In one embodiment, the UNet convolutional neural network converts a 224x224x3 image into a 112x112x64 feature map through the encoder part, and then enlarges the feature map to 224x224x32 through an upsampling method. Finally, the number of channels of the feature map is adjusted to the same as the number of categories through convolution. Optionally, the UNet convolutional neural network uses Mobinet as the backbone feature extraction network, and loads pre-trained weights to improve the feature extraction capability. Among them, the restoration method of the decoder is similar to the above-mentioned image restoration part.

[0134] See also Fig.12 , Fig.12 : is a flow chart of a method for detecting a sharp corner provided by the present application, wherein the method is applied to the smart terminal in the above embodiment to be executed by the smart terminal, and the method includes:

[0135] Step 31: Acquire the image of the circuit board to be inspected.

[0136] Optionally, the optical inspection device can obtain the image to be annotated by selecting the original input image and the output image path for generating auxiliary corner points through the AOI system. The optical inspection device selects the folder where the image to be processed is located as input, and the AOI system creates a folder named input folder + "_ouput" in the upper directory of the input folder as the output path by default.

[0137] Among them, after the program of the AOI system of the optical inspection device is started, there are two prompt messages. One is that there is a problem with the image data in the input folder, and the error message will be output in the text box; the other is that the code is successfully run. The reasons for the error include but are not limited to: (1) the input and output paths are not specified, (2) the folder and the pictures under it do not comply with the naming rules (for example, using Chinese names or containing special symbols); (3) the pictures in the folder are not image data that meet the requirements of sharp corner false points. Among them, a progress bar is displayed during the program startup process, and the progress bar shows the progress of the processing. The name of the currently processed image and the time taken are displayed in the output log.

[0138] Optionally, the optical detection equipment may be equipped with an image acquisition device such as a depth camera, a 3D camera, a monocular camera or a binocular camera, which can generate corresponding control information according to the user's input to obtain the image to be detected.

[0139] Optionally, the image to be detected is a PCB image. On the circuit board area, there is a PCB circuit etched by a reagent, and on the PCB circuit are small and dense circuit elements and their sharp corners. PCB can be applied to many electronic components, including mobile terminals such as cameras and video recorders, mobile phones, smart phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), etc., and can also be fixed terminals such as digital broadcast transmitters, digital TVs, desktop computers, servers, etc.

[0140] Optionally, after obtaining the detection information of the sharp corners in the image to be detected, the obtained detection information of the sharp corners may be used to generate a labeling box of the sharp corners and its confidence in the image to be detected.

[0141] Among them, the confidence of the sharp corners of the image to be detected can be calculated by the above-mentioned trained neural network model according to the matching degree between the sharp corner template and the sharp corner area. The confidence of the sharp corners of the image to be detected can also be calculated by a third-party organization (such as a data processing platform, cloud server, etc.).

[0142] Step 32: Input the image to be detected into a pre-trained neural network to obtain detection information of circuit components in the image to be detected.

[0143] Specifically, the intelligent terminal can input the image to be detected into the pre-trained UNet convolutional neural network model in the above embodiment to directly obtain the detection information of the circuit elements in the image to be detected from the convolutional neural network model.

[0144] In one embodiment, the UNet convolutional neural network can be divided into two parts, one is the feature extraction part, which, like other convolutional neural networks, extracts image features through stacked convolutions and compresses feature maps through pooling. The other part is the image restoration part, which restores the compressed image through upsampling and convolution. The feature extraction part can use excellent networks, such as Resnet50, VGG, etc. Among them, if the sharp-angle area is used as a mask, its range is small, and the sample balance loss function (i.e., focal loss) is used as the loss function of the neural network to balance the positive and negative examples, and finally a stable semantic segmentation model is obtained in which the loss function of the output sharp-angle area image is no longer reduced.

[0145] In the above embodiment, if the sharp-angle region labeling method is directly used, the sharp-angle region can be correctly labeled. However, since there are many templates in the sharp-angle template and the template matching operation is implemented on the CPU, the running speed is slow. If a stable neural network model is trained through a convolutional neural network, and the trained neural network model is used to directly label the sharp-angle region in the sharp-angle region that matches at least one sharp-angle template, the detection process and accuracy of AOI can be accelerated.

[0146] See also Fig.13 , Fig.13 It is a structural diagram of an intelligent terminal provided in the present application, wherein the intelligent terminal 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein the memory 102 stores program data, and the processor 101 retrieves the program data stored in the memory 102 to execute the above-mentioned neural network training method, sharp corner detection method or defect detection method.

[0147] Optionally, in one embodiment, the processor 101 is applied to an optical inspection device; the processor 101 is used to execute program data stored in the memory 102 to implement the following method: obtain a training set including several images to be trained, wherein the images to be trained include annotation information of sharp corners in the circuit board; input the training set into a neural network for training to obtain a neural network for detecting sharp corners in circuit board images.

[0148] Through the above method, on the one hand, the sharp corners of the circuit board and their annotation information in the input training set are used to train the neural network for detecting sharp corners, which can improve the accuracy and efficiency of the neural network. On the other hand, using the trained neural network to detect sharp corners in the circuit board image can optimize the process of sharp corner detection and improve the accuracy of sharp corner area detection.

[0149] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip having the ability to process signals. The processor 101 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] The memory 102 can be a memory stick, a TF card, etc., which can store all the information in the smart terminal 100, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory 102. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the smart terminal 100 has a memory function and can ensure normal operation. The memory 102 of the smart terminal 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose, and there is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but it is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the implementation of the intelligent terminal 100 described above is only illustrative, for example, in response to the user's operation instruction, the sharp corner area corresponding to the operation instruction is marked; the circuit element image is flipped, rotated, enlarged, reduced, chromaticity adjustment, etc., one or more image processing, etc., is performed, which is only a collection method, and there may be other division methods in actual implementation, such as the sharp corner image and the pre-stored sharp corner template can be combined or can be integrated into another system, or some features can be ignored or not executed.

[0152] In addition, each functional unit in each embodiment of the present application (such as an image acquisition module and a template extraction module, etc.) can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0153] See also Fig.14 , Fig.14 1 is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 110 stores program instructions 111 that can implement all the above methods.

[0154] If the integrated units of the functional units in the various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium 110. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer-readable storage medium 110 includes several instructions in a program instruction 111 to enable a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile terminal such as a mobile phone, a tablet computer, a wearable device, or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of various implementation methods of the present application.

[0155] Optionally, in one embodiment, program instructions 111 are applied to an optical inspection device; when the program instructions 111 are executed by a processor, they are used to implement the following method: obtaining a training set including a number of images to be trained, wherein the images to be trained include annotation information of sharp corners in a circuit board; inputting the training set into a neural network for training to obtain a neural network for detecting sharp corners in circuit board images.

[0156] Through the above method, on the one hand, the sharp corners of the circuit board and their annotation information in the input training set are used to train the neural network for detecting sharp corners, which can improve the accuracy and efficiency of the neural network. On the other hand, using the trained neural network to detect sharp corners in the circuit board image can optimize the process of sharp corner detection and improve the accuracy of sharp corner area detection.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media 110 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by a computer-readable storage medium 110. These computer-readable storage media 110 can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the program instructions 111 executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer-readable storage media 110 may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the program instructions 111 stored in the computer-readable storage medium 110 produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer-readable storage media 110 can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing program instructions 111 executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] In one embodiment, these programmable data processing devices include a processor and a memory. The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an electronic chip having signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0162] The memory can be a memory stick, TF card, etc. It stores and retrieves information according to the location specified by the processor. According to the purpose, the memory can be divided into primary memory (internal memory) and auxiliary memory (external memory). There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but it is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0163] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A defect detection method, characterized in that: The defect detection method comprises: Obtain the design drawings and captured images of the circuit board; Acquire the area where the sharp corner is located in the collected image; Matching the design drawing and the captured image in an area other than the area where the sharp corner is located; Outputting defect information according to the matching result between the design drawing and the acquired image; Wherein, the area where the sharp corner is located is detected by a pre-trained neural network; The matching of the design drawing and the acquired image in the area outside the area where the sharp corner is located includes: Acquire a mapping matrix between the design drawing and the acquired image; Mapping the area where the sharp corners on the collected image are located to the design drawing based on the mapping matrix to generate a sharp corner mask; The area outside the sharp corner mask in the design image is used to match the collected image.

2. A neural network training method, characterized in that: The neural network training method comprises: Obtaining a training set including a plurality of to-be-trained images, wherein the to-be-trained images include annotation information of sharp corners in a circuit board; Inputting the training set into the neural network for training to obtain a neural network for detecting sharp corners in a circuit board image; The neural network training method further includes: Get the sharp corner template; The circuit board area in the to-be-trained image is traversed by using the sharp-corner template, and the sharp-corner area in the to-be-trained image that matches the sharp-corner template is marked.

3. The neural network training method according to claim 2, characterized in that: After obtaining a training set including a plurality of images to be trained, the neural network training method further includes: Performing one or more image processing of flipping, rotating, enlarging, reducing, and adjusting chromaticity on the plurality of images to be trained to obtain a plurality of extended images; The training set is updated using the plurality of extended images.

4. The neural network training method according to claim 2, characterized in that: The neural network is iteratively trained using a sample balanced loss function.

5. A method for detecting sharp corners, characterized in that: The sharp corner detection method comprises: Acquire the image to be inspected of the circuit board; Inputting the image to be detected into a pre-trained neural network to obtain detection information of sharp corners in the image to be detected; Wherein, the pre-trained neural network is obtained by training using the neural network training method described in any one of claims 2 to 4.

6. The method for detecting sharp corners according to claim 5, characterized in that: After acquiring the detection information of the sharp corners in the image to be detected, the sharp corner detection method further includes: The detection information of the sharp corner is used to generate a labeling box of the sharp corner and its confidence level in the image to be detected.

7. An optical detection device, characterized in that: The optical detection device comprises: An image acquisition module is used to acquire the design drawing and collected images of the circuit board; A region extraction module, used for obtaining the region where the sharp corner is located in the collected image; wherein the region where the sharp corner is located is obtained by detecting a pre-trained neural network; An area matching module, used for matching the design drawing and the collected image in an area other than the area where the sharp corner is located; A defect output module, used for outputting defect information according to the matching result between the design drawing and the collected image; The area matching module is also used to obtain a mapping matrix between the design drawing and the acquired image; based on the mapping matrix, the area where the sharp corners on the acquired image are located is mapped to the design drawing to generate a sharp corner mask; and the area outside the sharp corner mask in the design drawing is used to match with the acquired image.

8. An intelligent terminal, characterized in that: The intelligent terminal includes a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls the program data stored in the memory to execute the defect detection method according to claim 1, the neural network training method according to any one of claims 2-4, or the sharp corner detection method according to any one of claims 5-6.

9. A computer-readable storage medium having program instructions stored therein, characterized in that: The program instructions are executed to implement the defect detection method according to claim 1, the neural network training method according to any one of claims 2-4, or the sharp corner detection method according to any one of claims 5-6.

Images