Circuit element labeling method, related method, device, terminal and storage medium

Abstract

The application discloses a circuit element labeling method, a neural network training method, a circuit element detection method, an optical detection device, an intelligent terminal and a computer readable storage medium. The circuit element labeling method is applied to the optical detection device, and the method comprises the following steps: acquiring a to-be-labeled image and acquiring a circuit element template; wherein the to-be-labeled image comprises a circuit board region; extracting at least one circuit element information in the circuit element template; and traversing the circuit board region in the to-be-labeled image by using the at least one circuit element information, and labeling a circuit element region in the circuit board region which is matched with the at least one circuit element information. According to the method, the circuit element information corresponding to the circuit element template is matched with the circuit board region in the to-be-labeled image, and the accuracy of circuit element region labeling can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for annotating circuit elements, related methods, devices, terminals, and storage media. Background Technology

[0002] With the widespread application of automotive electronics, communication equipment, transformers, inductors, and power modules in daily life, and the rapid development of electronic and communication technologies, the market has placed higher demands on high-transmission, high-voltage electronic products. As the fundamental carrier component of electronic components, the performance of the printed circuit board (PCB) directly affects the performance of the product after the electronic components are installed.

[0003] Because PCB manufacturing inevitably introduces numerous defects, primarily located at the circuit components, it's crucial to label these defective components for subsequent PCB repair. However, PCB components are typically very small and densely packed, making manual labeling extremely time-consuming, labor-intensive, and prone to errors. If defective components aren't labeled and are directly moved to the next manufacturing stage, subsequent PCB repair costs will escalate, increasing the likelihood of PCBs becoming unusable and resulting in significant cost waste. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, related methods, equipment, terminal, and storage medium for labeling circuit components, which can improve the accuracy of labeling circuit component areas.

[0005] One technical solution adopted in this application is: a circuit element annotation method, which is applied to an optical inspection device. The method includes: acquiring an image to be annotated and acquiring a circuit element template; wherein, the image to be annotated includes a circuit board area; extracting at least one circuit element information from the circuit element template; using the at least one circuit element information to traverse the circuit board area in the image to be annotated, and annotating the circuit element area in the circuit board area that matches the at least one circuit element information.

[0006] Optionally, obtaining a circuit element template includes: in response to a user's operation command, marking the circuit element corresponding to the operation command in the circuit board area of ​​the image to be annotated; and using the circuit element corresponding to the operation command to create a circuit element template.

[0007] Optionally, obtaining a circuit element template includes: obtaining a pre-stored circuit element template, wherein the circuit element template includes circuit element information of at least one type of circuit element; extracting at least one type of circuit element information from the circuit element template, including: identifying the type of circuit element in the image to be labeled; and extracting circuit element information of the same type of circuit element from the circuit element template based on the type of circuit element.

[0008] Optionally, after annotating the circuit element region in the circuit board area that matches at least one circuit element information, the circuit element annotation method further includes: extracting the circuit element image of the matching circuit element region; performing one or more image processing operations on the circuit element image, such as flipping, rotating, enlarging, reducing, and adjusting color, to obtain several processed circuit element images; and inputting the circuit element image and its corresponding processed circuit element image into a pre-stored circuit element template to update the circuit element template.

[0009] Optionally, the circuit element annotation method further includes: obtaining all circuit element images in the circuit element template; calculating the similarity between all circuit element images; and removing circuit element images from the circuit element template whose similarity is greater than or equal to a similarity threshold.

[0010] Optionally, the circuit element has a sharp corner.

[0011] Another technical solution adopted in this application is: providing a neural network training method, which includes: acquiring a training set including a plurality of images to be trained, wherein the circuit elements in the images to be trained are labeled by the circuit element labeling method described above; inputting the training set into the neural network for training to obtain a neural network for detecting circuit elements in circuit board images.

[0012] Another technical solution adopted in this application is: providing a circuit element detection method, which includes: acquiring an image of a circuit board to be detected; inputting the image to be detected into a pre-trained neural network to obtain detection information of circuit elements in the image to be detected; wherein, the pre-trained neural network is trained by the neural network training method described above.

[0013] Another technical solution adopted in this application is: providing an optical inspection device, which includes: an image acquisition module for acquiring an image to be labeled; wherein the image to be labeled includes a circuit board area; a template acquisition module for acquiring a circuit element template; an information extraction module for extracting at least one circuit element information from the circuit element template; and an annotation module for traversing the circuit board area in the image to be labeled using at least one circuit element information, and annotating the circuit element area in the circuit board area that matches at least one circuit element information.

[0014] Another technical solution adopted in this application is: to provide a smart terminal, which includes: a processor and a memory connected to the processor, wherein the memory stores program data, and the processor retrieves the program data stored in the memory to execute the circuit element annotation method, neural network training method or circuit element detection method as described above.

[0015] Another technical solution adopted in this application is: providing a computer-readable storage medium that stores program data, which, when executed by a processor, is used to implement the circuit element annotation method, neural network training method, or circuit element detection method as described above.

[0016] Unlike existing technologies, the circuit component annotation method provided in this application is applied to optical inspection equipment. The method includes: acquiring an image to be annotated and acquiring a circuit component template; wherein the image to be annotated includes a circuit board area; extracting at least one circuit component information from the circuit component template; and using the at least one circuit component information to traverse the circuit board area in the image to be annotated, annotating the circuit component areas in the circuit board area that match the at least one circuit component information. By using the circuit component information corresponding to the circuit component template to match the circuit board area in the image to be annotated, the accuracy of circuit component area annotation can be improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0018] Figure 1 This is a schematic diagram of the structure of an embodiment of the optical inspection equipment provided in this application;

[0019] Figure 2 This is a flowchart illustrating an embodiment of the circuit element labeling method provided in this application;

[0020] Figure 3 This is a flowchart illustrating an embodiment of obtaining a circuit element template in this application;

[0021] Figure 4 This is a schematic diagram of the interface of an embodiment of the circuit element corresponding to the operation command marked in this application;

[0022] Figure 5 This is a flowchart illustrating an embodiment of establishing a circuit element template in this application;

[0023] Figure 6 This is a flowchart illustrating an embodiment of extracting circuit element information in this application;

[0024] Figure 7 This is a flowchart illustrating an embodiment of updating a circuit element template in this application;

[0025] Figure 8 This is a flowchart illustrating another embodiment of updating the circuit element template in this application;

[0026] Figure 9 This is a flowchart illustrating a neural network training method provided in this application;

[0027] Figure 10 This is a flowchart illustrating an embodiment of updating the training set in this application;

[0028] Figure 11 This is a flowchart illustrating an embodiment of updating the training set using several extended images in this application;

[0029] Figure 12 This is a flowchart illustrating a circuit element testing method provided in this application;

[0030] Figure 13 This is a schematic diagram of the structure of a smart terminal provided in this application;

[0031] Figure 14 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0033] The reference to "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] The steps in the embodiments of this application are not necessarily processed in the order described. The steps can be rearranged, deleted, or added as needed. The step descriptions in the embodiments of this application are only optional combinations of sequences and do not represent all possible combinations of steps in the embodiments of this application. The order of steps in the embodiments should not be considered as a limitation of this application.

[0035] The term "and / or" in the embodiments of this application refers to any and all possible combinations including one or more of the associated listed items. It should also be noted that, when used in this specification, "including / comprising" specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or components and / or groups thereof.

[0036] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0037] Furthermore, although the terms "first," "second," etc., are used repeatedly in this application to describe various operations (or various elements, or various applications, or various instructions, or various data), these operations (or elements, or applications, or instructions, or data) should not be limited by these terms. These terms are only used to distinguish one operation (or element, or application, or instruction, or data) from another operation (or element, or application, or instruction, or data). For example, a first circuit element template can be called a second circuit element template, and a second circuit element template can be called a first circuit element template; the only difference is the scope they encompass, but this does not depart from the scope of this application. Both the first circuit element template and the second circuit element template are collections of various circuit element templates, but they are not the same collection of circuit element templates.

[0038] See Figure 1 , Figure 1 This is a schematic diagram of an embodiment of the optical inspection device provided in this application. The optical inspection device 10A includes: an image acquisition module 11A, a template acquisition module 12A, an information extraction module 13A, and an annotation module 14A.

[0039] In one embodiment, the optical inspection device 10A is an automated optical inspection (AOI) device, which is a device based on optical principles to detect common defects encountered in welding production. During automatic inspection, the AOI machine automatically scans the PCB with a camera, acquires images, compares the tested solder joints with qualified parameters in the database, processes the images, detects defects on the PCB, and displays / marks the defects on a monitor or with automatic markers for repair personnel to fix.

[0040] Optionally, the AOI's CPU (Central Processing Unit / Processor) is an Intel Core i5 or higher processor; the memory is 8GB or higher; the hard drive has at least 10GB of free disk space; the graphics card is a GeForce RTX 3070Ti with 8GB or higher video memory; the network card is a gigabit network card; the display device is a monitor that supports a 1280x1024 resolution; the application software environment is PyTorch; and the operating system is Microsoft Windows 10 Ultimate or Professional.

[0041] Specifically, the image acquisition module 11A is used to acquire the image to be labeled; wherein the image to be labeled includes a circuit board area.

[0042] Optionally, users can obtain the image to be annotated by selecting the original input image and the output image path for generating auxiliary corner points through the AOI system. Users select the folder containing the image to be processed as the input. By default, the AOI system creates a folder named "input folder + '_output'" in the parent directory of the input folder as the output path. Users can also specify the output path themselves. The AOI system recommends using the default output path; otherwise, a path must be manually selected in the corner annotation generation module.

[0043] After the AOI system program starts, there are two types of prompts: one indicates that there is a problem with the image data in the input folder, and an error message will be displayed in the text box; the other indicates that the code has been successfully executed. The reasons for the error include, but are not limited to: (1) no input and output paths are specified; (2) the folder and its images do not conform to naming rules (e.g., using Chinese characters or containing special symbols); (3) the images in the folder are not the required sharp-angled false point image data. During program startup, a progress bar is displayed, showing the processing progress. The output log displays the name of the currently processed image and the processing time.

[0044] Optionally, the image acquisition module 11A may be equipped with an image acquisition device, such as a depth camera, a 3D camera, a monocular camera, or a binocular camera, which can generate corresponding control information based on user input to acquire the image to be labeled.

[0045] Optionally, the image to be labeled is a PCB (Printed Circuit Board) image. A PCB, also known as a printed circuit board, has a surface divided into a circuit board area and a non-circuit board area. The circuit board area contains PCB lines etched with chemicals, and these lines contain small, densely packed circuit components with sharp corners. PCBs are used in many electronic components, including mobile terminals such as cameras and video recorders, mobile phones, smartphones, laptops, personal digital assistants (PDAs), and tablet computers (PADs), as well as fixed terminals such as digital broadcast transmitters, digital TVs, desktop computers, and servers.

[0046] Specifically, the template acquisition module 12A is used to acquire circuit element templates.

[0047] Optionally, the circuit element template is a standard image template of circuit elements prefabricated for one or more types of PCBs and stored on an optical inspection device 10A or a third-party organization (e.g., a digital processing platform, cloud server, external terminal, etc.). This standard image template contains multiple circuit elements and their information. For example, the standard image template may contain 100-10000, 500, 1000, or 5000 circuit elements and their information.

[0048] The circuit element information includes the spatial shape (including planar shape and curved surface shape), location, size, type, and purpose of the corresponding circuit element.

[0049] Optionally, the circuit element template is obtained by preprocessing the original PCB image using the AOI system.

[0050] Specifically, the AOI system first crops each original PCB image and extracts corners using Harris corner detection. Then, human annotators use the Labelme application to outline a small number of circuit component areas that need to be annotated. Subsequently, the neighborhood of these circuit components is used as a template, and each template is flipped and rotated at multiple angles to obtain diverse templates, establishing a template library. At the same time, similar circuit component areas are removed, finally resulting in circuit component templates.

[0051] After the AOI system is running normally, it creates three subfolders: Crop, Imgs, and Pos in its output folder. Crop creates a folder named after each input image, and each folder stores the cropped image information for subsequent annotation in the labelme application. Imgs contains large, numbered images of the cropped images. Pos stores the coordinates of all corner points of all images. The original PCB images are cropped because each PCB image is obtained by scanning the actual board surface using the original AOI device's camera; its size is too large, making manual annotation or input into a neural network overly complex and inconvenient for manual annotation. Therefore, in one embodiment, the image is uniformly cropped to a size of 224*224. The traditional Harris corner detection method is used to extract all corners (i.e., circuit elements) in the image because the sharp corners of circuit elements are a type of corner.

[0052] The AOI system imports the Crop folder, generated after preprocessing the raw data, into the open-source software labelme to perform rectangular annotations on the sharp corners of circuit components. During the annotation process, it is not necessary to annotate all the sharp corners of all circuit components; only a small number of sharp corners of different types need to be annotated. After annotation is completed, a JSON file with the same name is generated in the same path as the annotated image.

[0053] The AOI system extracts the 8x8 square area containing the sharp corners of circuit components within the manually annotated bounding box, performs flipping and rotation transformations to obtain different circuit component sharp corners, thus forming circuit component templates. Since an original PCB image may be composed of multiple small boards, and the positions of these boards may be rotated, there will be many identical elements after rotation and flipping. Here, a template is expanded to achieve matching at different angles. Furthermore, for each original PCB image, every template in the template library is matched and fully annotated, greatly reducing manual annotation costs and obtaining highly accurate matching results while avoiding omissions and errors.

[0054] Specifically, the information extraction module 13A is used to extract information about at least one circuit element from the circuit element template.

[0055] Optionally, the information extraction module 13A may extract information of at least one circuit element from the circuit element template based on user input.

[0056] For example, if the user expects to extract information on X circuit elements in the circuit element template based on the user input, namely, the location of the circuit element, the size of the circuit element, the type of the circuit element, and the purpose of the circuit element, then the information extraction module 13A responds to the user input and performs the extraction of the corresponding circuit element information in the circuit element template.

[0057] Specifically, the annotation module 14A is used to traverse the circuit board area in the image to be annotated using at least one circuit element information, and to annotate the circuit element area in the circuit board area that matches the at least one circuit element information.

[0058] Optionally, the annotation module 14A performs a traversal matching of the acquired circuit component information on the circuit board area in the image to be annotated, to determine the matching degree between the circuit component area on the circuit board area and the acquired circuit component information, and annotates the corresponding circuit component area based on the matching degree. This matching degree is related to information such as the spatial shape (including planar shape and curved surface shape), position, and size of the corresponding circuit component area.

[0059] Unlike existing technologies, the optical inspection device provided in this embodiment includes: an image acquisition module for acquiring an image to be labeled, wherein the image to be labeled includes a circuit board area; a template acquisition module for acquiring a circuit component template; an information extraction module for extracting at least one circuit component information from the circuit component template; and an annotation module for traversing the circuit board area in the image to be labeled using the at least one circuit component information, and annotating the circuit component areas in the circuit board area that match the at least one circuit component information. By using the aforementioned optical inspection device to match the circuit board area in the image to be labeled using the circuit component information corresponding to the circuit component template, the accuracy and efficiency of circuit component area matching can be improved.

[0060] Optionally, by combining the above-mentioned optional embodiments and further optimizing and expanding upon the above technical solutions, one embodiment of the circuit element labeling method provided in this application can be obtained.

[0061] See Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the circuit element marking method provided in this application. The method is applied to the optical inspection device in the above embodiment for execution by the optical inspection device, and includes:

[0062] Step 11: Obtain the image to be labeled and obtain the circuit component template; wherein, the image to be labeled includes the circuit board area.

[0063] Optionally, the image to be labeled acquired by the optical inspection equipment is a PCB image, and the acquired circuit component template is a sharp corner template, that is, the circuit component is a sharp corner.

[0064] The area containing the sharp corner protrudes outward from the four right-angled parts of the square area, causing two adjacent straight lines in the square area to form intersecting inclined lines at their close ends. The two adjacent inclined lines intersect to form a sharp corner; the inclined line is called the sharp corner line segment.

[0065] See Figure 3 , Figure 3 This is a schematic flowchart of an embodiment of obtaining a circuit element template in this application. Specifically, step 11 may include the following steps:

[0066] Step 111: In response to the user's operation command, mark the circuit components corresponding to the operation command in the circuit board area of ​​the image to be marked.

[0067] See Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the annotation operation instructions corresponding to circuit elements in this application. The interface displays a circuit board area A in the image to be annotated, and this area contains multiple circuit elements. The user annotates the multiple circuit elements on the interface by inputting corresponding annotation data into an optical inspection device, or manually using a mouse, electronic pen, or touchscreen input. As shown in the figure, circuit elements P1, P2, P3, and P4 corresponding to the operation instructions are annotated in the circuit board area P.

[0068] Step 112: Use the circuit elements corresponding to the operation instructions to create a circuit element template.

[0069] Specifically, the optical inspection equipment sequentially performs feature recognition, feature segmentation, and feature extraction on the circuit elements labeled by the user in order to establish circuit element templates.

[0070] Please see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of establishing a circuit element template in this application. Specifically, the method may include the following steps:

[0071] Step 1121: Identify the labeled circuit elements to perform feature segmentation on the labeled circuit elements.

[0072] In one embodiment, the optical inspection device identifies the marked circuit elements that require feature segmentation in the corresponding direction, position, and angle based on the segmented circuit board area, and then uses a feature extraction network (such as CNN, VGG, ResNet, etc.) to perform feature segmentation on the marked circuit elements in order to segment out the corresponding circuit elements.

[0073] Step 1122: Convert the segmented circuit elements into corresponding matrix vectors.

[0074] In one embodiment, the optical inspection device uses a word-embedding network to convert segmented circuit elements into corresponding N*S-dimensional matrix vectors, with each segmented circuit element corresponding to a matrix vector. For example, a circuit element is converted into a corresponding 1*S-dimensional matrix vector, where 1 represents the number of circuit elements and S represents the vector dimension of the matrix.

[0075] Step 1123: Input the matrix vector into the correction model to obtain the circuit element template corresponding to the matrix vector.

[0076] In one embodiment, the pre-trained correction model can be a Transformer model. The optical detection device inputs each matrix vector into the trained Transformer model for template correction to obtain the correction template corresponding to each segmented circuit element, that is, to establish the circuit element template corresponding to each segmented labeled circuit element.

[0077] Step 1124: Sort the corresponding circuit element templates to obtain a set of circuit element templates.

[0078] In one embodiment, the optical detection device inputs the circuit element templates corresponding to all obtained circuit elements into an image sorting network (such as an attention-based RNN, LSTM, etc.) to sort the circuit element templates in the image according to the corresponding direction, position, and angle, so as to obtain a set of circuit element templates.

[0079] In another embodiment, obtaining the circuit element template may be obtaining a pre-stored circuit element template; wherein the pre-stored circuit element template includes circuit element information of at least one type of circuit element.

[0080] Specifically, the pre-stored circuit component templates can be created based on the circuit board area in the historical image to be labeled, and the creation method is similar to that described in the above embodiments, so it will not be repeated here; wherein, the historical image to be labeled is the image to be labeled that was previously acquired by the optical inspection equipment. The pre-stored circuit component templates can be pre-stored in the storage medium of the optical inspection equipment or in a third-party institution (such as a digital processing platform, cloud server, external terminal, etc.).

[0081] Step 12: Extract information of at least one circuit element from the circuit element template.

[0082] Please see Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of extracting circuit element information in this application. Specifically, the method may include the following steps:

[0083] Step 121: Identify the type of circuit components in the image to be labeled.

[0084] Specifically, the types of circuit components in the image to be labeled can be classified into different types based on the shape, position, size, etc. of the circuit components; the types of circuit components can be obtained by manual classification by the user or automatically output by a component classification model trained by a neural network.

[0085] Step 122: Extract circuit element information of the same type from the circuit element template based on the circuit element type.

[0086] Specifically, the optical inspection equipment matches the type of the circuit element template determined by the user with the type of circuit element in the image to be labeled, and extracts the information of circuit elements with the same type from the circuit element template.

[0087] Step 13: Use at least one circuit element information to traverse the circuit board area in the image to be annotated, and annotate the circuit element area in the circuit board area that matches at least one circuit element information.

[0088] Specifically, the optical inspection equipment traverses and matches the acquired circuit component information across the circuit board area in the image to be labeled, in order to determine the matching degree between the circuit component area on the circuit board area and the acquired circuit component information, and labels the corresponding circuit component area based on the matching degree. This matching degree is related to information such as the spatial shape (including planar shape and curved surface shape), position, and size of the corresponding circuit component area.

[0089] Please see Figure 7 , Figure 7 This is a flowchart illustrating an embodiment of updating a circuit element template in this application. Specifically, after marking the circuit element area in the circuit board area that matches at least one circuit element information, the circuit element template can be updated, further including the following steps:

[0090] Step 131: Extract the circuit element images of the matching circuit element regions.

[0091] Specifically, the optical inspection equipment extracts the circuit element image corresponding to the circuit element area in the marked circuit board area that matches the information of at least one circuit element.

[0092] Step 132: Perform one or more image processing operations on the circuit element image, such as flipping, rotating, enlarging, reducing, and adjusting color, to obtain several processed circuit element images.

[0093] Specifically, optical inspection equipment can use its own built-in image processing program, or send the extracted circuit component images to a third-party organization (such as an image processing platform, cloud server, etc.) to perform one or more image processing operations such as flipping, rotating, enlarging, reducing, and color adjustment on the extracted circuit component images to obtain several processed circuit component images.

[0094] Step 133: Input the circuit element image and its corresponding processed circuit element image into the pre-stored circuit element template to update the circuit element template.

[0095] Please see Figure 8 , Figure 8 This is a flowchart illustrating another embodiment of updating the circuit element template in this application. Specifically, after inputting several processed circuit element images into a pre-stored circuit element template, the circuit element template can be updated again, further including the following steps:

[0096] Step 1331: Obtain images of all circuit elements in the circuit element template.

[0097] Specifically, the optical inspection equipment acquires images of all circuit elements in the circuit element template from its own storage medium or from a third-party organization (such as a digital processing platform, cloud server, external terminal, etc.).

[0098] Step 1332: Calculate the similarity between all circuit element images.

[0099] Among them, the methods for calculating image similarity using optical detection equipment can be based on Euclidean distance, black box distance, cosine distance, etc., and no specific limitations are made here.

[0100] Step 1333: Remove circuit element images from the circuit element template that have a similarity greater than or equal to the similarity threshold.

[0101] Specifically, the optical inspection device removes all circuit element images in the circuit element template whose similarity exceeds a preset similarity threshold, thereby reducing redundancy in storing the circuit element template and improving the speed of matching circuit element information. The preset similarity threshold is 70%-100%, for example, 75%, 85%, 95%, etc.

[0102] In another embodiment, the optical inspection device can simply select a pre-made circuit component template to annotate the image to be annotated; alternatively, it can simply select a manually annotated circuit component region based on the image to be annotated to form a corresponding manually annotated circuit component template, and then directly perform annotation based on the manually annotated circuit component template; furthermore, the optical inspection device can add the manually annotated circuit component template to the pre-made circuit component template and remove templates with a similarity higher than a preset threshold to further update the circuit component template, and perform annotation through further updating the circuit component template. Therefore, this further updating of the circuit component template can integrate new templates (manually annotated circuit component templates) into the original template library (pre-made circuit component templates), re-annotate or supplement the image to be annotated, and reduce unnecessary computational load on the optical inspection device through further updating the circuit component template.

[0103] Unlike existing technologies, the circuit component annotation method provided in this embodiment is applied to optical inspection equipment. The method includes: acquiring an image to be annotated and acquiring a circuit component template; wherein the image to be annotated includes a circuit board area; extracting at least one circuit component information from the circuit component template; traversing the circuit board area in the image to be annotated using the at least one circuit component information, and annotating the circuit component area in the circuit board area that matches the at least one circuit component information. Through the above-described circuit component annotation method, on the one hand, by using a pre-made circuit component template to match and annotate the corresponding circuit board area in the image to be annotated with at least one circuit component of the same type, the accuracy and efficiency of circuit component area matching can be improved, thereby optimizing the PCB area annotation process and increasing the diversity of annotated circuit components. On the other hand, by performing various image processing operations on the circuit component image and determining the corresponding circuit component image based on similarity, the circuit component image in the circuit component template is updated and optimized, improving the accuracy of subsequent circuit component area matching.

[0104] See Figure 9 , Figure 9 This is a flowchart illustrating a neural network training method provided in this application, wherein the method is applied to the smart terminal in the above embodiments for execution by the smart terminal, and the method includes:

[0105] Step 21: Obtain a training set that includes several images to be trained.

[0106] Specifically, the smart terminal acquires a training set comprising several images to be trained from the optical inspection device. The training set of images to be trained consists of circuit element regions in the labeled circuit board area of ​​the optical inspection device described in the above embodiments that match information of at least one circuit element. The circuit elements in the images to be trained are labeled using the circuit element labeling method described in the above embodiments.

[0107] Furthermore, the training images can be expanded to update the training set.

[0108] Please see Figure 10 , Figure 10 This is a flowchart illustrating an embodiment of updating the training set in this application. Specifically, the method may include the following steps:

[0109] Step 211: Perform one or more image processing operations on several training images, including flipping, rotating, enlarging, shrinking, and color adjustment, to obtain several extended images.

[0110] Specifically, optical inspection equipment can use its own built-in image processing program, or send several extracted training images to a third-party organization (such as an image processing platform, cloud server, etc.) to perform one or more image processing operations such as flipping, rotating, enlarging, reducing, and color adjustment on the training images to obtain several extended training images.

[0111] Step 212: Update the training set using several expanded images.

[0112] Please see Figure 11 , Figure 11 This is a flowchart illustrating an embodiment of updating the training set using several expanded images in this application. Specifically, step 212 further includes the following steps:

[0113] Step 2121: Calculate the similarity between several extended images.

[0114] Among them, the methods for calculating image similarity using optical detection equipment can be based on Euclidean distance, black box distance, cosine distance, etc., and no specific limitations are made here.

[0115] Step 2122: Remove extended images from a number of extended images whose similarity is greater than or equal to the similarity threshold.

[0116] Step 2123: Add the remaining extended images to the training set for updating.

[0117] Specifically, the optical inspection device removes images from the expanded image whose similarity exceeds a preset similarity threshold, thereby reducing redundancy in storing circuit component templates and improving the speed of matching circuit component information. The preset similarity threshold is 70%-100%, for example, 75%, 85%, 95%, etc.

[0118] Furthermore, the remaining extended images after removal are added to the training set to update the training set.

[0119] Step 22: Input the training set into the neural network for training to obtain a neural network for detecting circuit elements in circuit board images.

[0120] Specifically, the smart terminal inputs the acquired training set into the encoding and decoding structure of the UNet convolutional neural network for training, and finally trains a stable semantic segmentation model, that is, a neural network of circuit elements.

[0121] In one embodiment, the entire UNet convolutional neural network is U-shaped. The UNet network can be divided into two parts: a feature extraction part, which, like other convolutional neural networks, extracts image features through stacked convolutions and compresses the feature map through pooling; and an image reconstruction part, which restores the compressed image through upsampling and convolution. The feature extraction part can use high-quality networks such as ResNet50 and VGG. If the circuit element region is used as a mask, its area is relatively small. Focal loss is used as the loss function of the neural network to balance positive and negative examples, ultimately resulting in a stable semantic segmentation model where the loss function for outputting the circuit element region image no longer decreases.

[0122] In one embodiment, the UNet convolutional neural network transforms a 224x224x3 image into a 112x112x64 feature map through its encoder, and then upsamples the feature map to 224x224x32. Finally, convolution is used to adjust the number of channels in the feature map to match the number of classes. Optionally, the UNet convolutional neural network uses Mobinet as the backbone feature extraction network and loads pre-trained weights to improve feature extraction capabilities. The decoder's reconstruction method is similar to the image reconstruction section described above.

[0123] See Figure 12 , Figure 12 This is a flowchart illustrating a circuit element detection method provided in this application, wherein the method is applied to the smart terminal in the above embodiments for execution by the smart terminal, and the method includes:

[0124] Step 31: Obtain the image of the circuit board to be inspected.

[0125] Optionally, the smart terminal can obtain the image to be detected by selecting the original input image and the output image path for generating auxiliary corner points through the AOI system. Specifically, the smart terminal selects the folder containing the image to be processed as the input, and the AOI system by default creates a folder named "input folder + "_output" in the parent directory of the input folder as the output path.

[0126] After the AOI system program starts, there are two types of prompts: one indicates that there is a problem with the image data in the input folder, and an error message will be displayed in the text box; the other indicates that the code has been successfully executed. The reasons for the error include, but are not limited to: (1) no input and output paths are specified; (2) the folder and its images do not conform to naming rules (e.g., using Chinese characters or containing special symbols); (3) the images in the folder are not the required sharp-angled false point image data. During program startup, a progress bar is displayed, showing the processing progress. The output log displays the name of the currently processed image and the processing time.

[0127] Optionally, the smart terminal may be equipped with an image acquisition device, such as a depth camera, a 3D camera, a monocular camera, or a binocular camera, which can generate corresponding control information based on user input to acquire the image to be detected.

[0128] Optionally, the image to be inspected is a PCB image. The PCB area has etched circuitry using chemicals, and the PCB circuitry contains small, densely packed circuit elements and their sharp corners. PCBs can be used in a wide variety of electronic components, including mobile terminals such as cameras and video recorders, mobile phones, smartphones, laptops, personal digital assistants (PDAs), and tablet computers (PADs), as well as fixed terminals such as digital broadcast transmitters, digital TVs, desktop computers, and servers.

[0129] Step 32: Input the image to be detected into the pre-trained neural network to obtain the detection information of the circuit components in the image to be detected.

[0130] Specifically, the smart terminal can input the image to be detected into the pre-trained UNet convolutional neural network model in the above embodiment to directly obtain the detection information of the circuit elements in the image to be detected from the convolutional neural network model.

[0131] In one embodiment, the UNet convolutional neural network can be divided into two parts: a feature extraction part, which, like other convolutional neural networks, extracts image features through stacked convolutions and compresses the feature map through pooling; and an image reconstruction part, which restores the compressed image through upsampling and convolution. The feature extraction part can use high-quality networks such as ResNet50 and VGG. If a sharp corner region is used as a mask, its area is relatively small. A sample balancing loss function (focal loss) is used as the loss function of the neural network to balance positive and negative examples, ultimately resulting in a stable semantic segmentation model that can output images of sharp corner regions without further loss.

[0132] In the above embodiments, the correct annotation information for the circuit element region can be obtained by directly using the circuit element annotation method. However, since there are many templates in the circuit element template and the template matching operation is implemented on the CPU, the running speed is slow. If a stable neural network model is trained by a convolutional neural network, and the trained neural network model is used to directly annotate the circuit element region in the circuit board area that matches the information of at least one circuit element, the detection process and accuracy of AOI can be accelerated.

[0133] See Figure 13 , Figure 13 This is a schematic diagram of the structure of a smart terminal provided in this application. The smart terminal 100 includes a processor 101 and a memory 102 connected to the processor 101. The memory 102 stores program data. The processor 101 retrieves the program data stored in the memory 102 to execute the circuit element annotation method, neural network training method, or circuit element detection method described above.

[0134] Optionally, in one embodiment, the processor 101 is applied to an optical inspection device; the processor 101 is used to execute program data stored in the memory 102 to implement the following method: acquiring an image to be labeled, and acquiring a circuit element template; wherein the image to be labeled includes a circuit board area; extracting at least one circuit element information from the circuit element template; traversing the circuit board area in the image to be labeled using the at least one circuit element information, and labeling the circuit element area in the circuit board area that matches the at least one circuit element information.

[0135] The processor 101 can also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip with signal processing capabilities. The processor 101 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0136] The memory 102 can be a memory module, TF card, etc., and can store all the information in the smart terminal 100, including the input raw data, computer programs, intermediate running results, and final running results. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the smart terminal 100 has a memory function and can ensure normal operation. The memory 102 of the smart terminal 100 can be classified according to its purpose as main memory (RAM) and auxiliary memory (external storage), or it can be classified as external memory and internal memory. External storage is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage components on the motherboard, used to store currently executing data and programs, but it is only used for temporary storage of programs and data; the data will be lost when the power is turned off.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the implementation of the smart terminal 100 described above is merely illustrative. For instance, in response to a user's operation command, marking the circuit element corresponding to the operation command; performing one or more image processing operations such as flipping, rotating, enlarging, shrinking, and adjusting the color of the circuit element image, etc., is merely a collection method. In actual implementation, there may be other division methods. For example, the circuit element image and the pre-stored circuit element template can be combined or integrated into another system, or some features can be ignored or not executed.

[0138] Furthermore, the functional units (such as the image acquisition module and the information extraction module) in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] See Figure 14 , Figure 14 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 110 stores program instructions 111 capable of implementing all the above methods.

[0140] If the integrated units of the various functional units in the various embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium 110. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer-readable storage medium 110 includes several instructions in a program instruction 111 to cause a computer device (which may be a personal computer, system server, or network device, etc.), an electronic device (e.g., MP3, MP4, etc., or a mobile terminal such as a mobile phone, tablet, or wearable device, or a desktop computer, etc.), or a processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0141] Optionally, in one embodiment, program instructions 111 are applied to an optical inspection device; when executed by a processor, program instructions 111 are used to implement the following method: acquiring an image to be labeled and acquiring a circuit element template; wherein the image to be labeled includes a circuit board area; extracting at least one circuit element information from the circuit element template; using the at least one circuit element information to traverse the circuit board area in the image to be labeled, and labeling the circuit element area in the circuit board area that matches the at least one circuit element information.

[0142] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media 110 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable storage medium 110. These computer-readable storage media 110 can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that program instructions 111, executable by the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer-readable storage media 110 may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that program instructions 111 stored in the computer-readable storage medium 110 produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer-readable storage media 110 may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing program instructions 111 that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] In one embodiment, these programmable data processing devices include a processor and memory. The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an electronic chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0147] Memory can be a RAM module, a TF card, etc., and it stores and retrieves information according to the location specified by the processor. Memory can be classified according to its purpose into main memory (RAM) and auxiliary memory (external storage), or it can be classified into external memory and internal memory. External storage is usually magnetic media or optical discs, which can store information permanently. RAM refers to the storage components on the motherboard, used to store currently executing data and programs, but it is only used for temporary storage of programs and data; the data will be lost when the power is turned off.

[0148] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for marking circuit components, applied to optical inspection equipment, characterized in that, The method includes: Obtain the image to be labeled and obtain the circuit component template; wherein, the image to be labeled includes a circuit board area, and the circuit component is a sharp corner on the PCB line; Extract information of at least one circuit element from the circuit element template; Using the information of at least one circuit element, traverse the circuit board area in the image to be labeled, and label the circuit element area in the circuit board area that matches the information of at least one circuit element; Obtaining circuit component templates includes: Corner detection is performed on the image to be labeled to extract candidate circuit element regions; The circuit elements in the candidate circuit element region are labeled, and the square region where the labeled circuit elements are located is extracted as a template; The templates are flipped and rotated at multiple angles to obtain diverse templates, and a template library is established. Retrieve a pre-stored circuit element template from the template library, wherein the circuit element template includes circuit element information of at least one type of circuit element; Identify the types of circuit components in the image to be labeled; Based on the circuit element type, extract circuit element information of the same type from the circuit element template.

2. The circuit element labeling method according to claim 1, characterized in that, The acquisition of the circuit component template includes: In response to the user's operation command, the circuit element corresponding to the operation command is marked in the circuit board area of ​​the image to be marked; The circuit element template is established using the circuit elements corresponding to the operation instructions.

3. The circuit element labeling method according to claim 1, characterized in that, After labeling the circuit element region in the circuit board area that matches the information of the at least one circuit element, the circuit element labeling method further includes: Extract the circuit element image of the matching circuit element region; The circuit element image is subjected to one or more image processing methods, including flipping, rotating, enlarging, shrinking, and color adjustment, to obtain several processed circuit element images; The circuit element image and its corresponding processing circuit element image are input into the pre-stored circuit element template to update the circuit element template.

4. The circuit element labeling method according to claim 3, characterized in that, The circuit element labeling method further includes: Obtain images of all circuit elements in the circuit element template; Calculate the similarity between the images of all the circuit elements; Remove circuit element images from the circuit element template that have a similarity greater than or equal to the similarity threshold.

5. A neural network training method, characterized in that, The neural network training method includes: A training set comprising a plurality of images to be trained is obtained, wherein the circuit elements in the images to be trained are labeled by the circuit element labeling method according to any one of claims 1 to 4; The training set is input into the neural network for training to obtain a neural network for detecting circuit elements in a circuit board image.

6. A method for testing circuit components, characterized in that, The circuit element detection method includes: Acquire the image of the circuit board to be inspected; The image to be detected is input into a pre-trained neural network to obtain detection information of circuit elements in the image to be detected; The pre-trained neural network is obtained by training the neural network training method of claim 5.

7. An optical inspection device, characterized in that, The optical inspection equipment includes: An image acquisition module is used to acquire an image to be labeled; wherein, the image to be labeled includes a circuit board area, and the circuit element is a sharp corner on the PCB line; The template acquisition module is used to acquire circuit component templates; The information extraction module is used to extract information about at least one circuit element from the circuit element template; The annotation module is used to traverse the circuit board area in the image to be annotated using the at least one circuit element information, and annotate the circuit element area in the circuit board area that matches the at least one circuit element information; The template acquisition module is further configured to: perform corner detection on the image to be labeled to extract candidate circuit element regions; label the circuit elements in the candidate circuit element regions and extract the square regions where the labeled circuit elements are located as templates; perform multi-angle flipping and rotation processing on the templates to obtain diverse templates and establish a template library; obtain pre-stored circuit element templates from the template library, wherein the circuit element templates include circuit element information of at least one type of circuit element; identify the type of circuit element in the image to be labeled; and extract circuit element information of the same type of circuit element from the circuit element templates based on the type of circuit element.

8. A smart terminal, characterized in that, The smart terminal includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor retrieves the program data stored in the memory to execute the circuit element labeling method as described in any one of claims 1 to 4, the neural network training method as described in claim 5, or the circuit element detection method as described in claim 6.

9. A computer-readable storage medium internally storing program instructions, characterized in that, The program instructions are executed to perform the circuit element labeling method as described in any one of claims 1 to 4, the neural network training method as described in claim 5, or the circuit element detection method as described in claim 6.

Images