Vision-based method and system for identifying and locating non-standard components on printed circuit boards

By building a multi-layer neural network model to identify and position non-standard components of printed circuit boards, the problem of low positioning performance in the existing technology is solved, and the positioning of high precision and low false alarm rate is achieved, which is suitable for complex environments and different Image type, reducing assembly costs.

CN114092440BActive Publication Date: 2024-11-12BEIJING BRIO ELECTRONIC TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111382915.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-22
Publication Date
2024-11-12
Estimated Expiration
2041-11-22

AI Technical Summary

Technical Problem

The prior art has problems in the positioning of non-standard components of printed circuit boards with low positioning performance, high requirements for image quality, and susceptible to contamination and blur.

Method used

Using vision-based non-standard component recognition and positioning method of printed circuit boards, the multi-layer neural network model is constructed, and the preprocessed image data is used for training, so as to realize the identification and positioning of non-standard components, reducing the prior knowledge of component contour Dependency, suitable for components of any contour.

Benefits of technology

It improves positioning accuracy and speed, reduces false alarm rate, adapts to positioning performance in complex environments, reduces assembly costs, is suitable for color and grayscale images, and is highly adaptable.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114092440B_ABST
    Figure CN114092440B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a method and system for identifying and locating non-standard components of printed circuit boards based on vision, and the method includes the following steps: collecting image data and performing preprocessing; constructing a positioning neural network model for the non-standard components of the printed circuit board, and training the positioning neural network model using multiple groups of sample images collected and preprocessed; and using the trained positioning neural network model to identify and locate the non-standard components of the printed circuit board. The present invention uses multiple layered neural network models to model the printed circuit board image, and establishes a network model with the ability to identify and locate non-standard components by designing the connection mode and connection weights of neurons between adjacent layers of the neural network. It can automatically locate the contour area of ​​non-standard electronic components from images or videos containing printed circuit boards and non-standard electronic components, and has high positioning accuracy and positioning speed, achieving higher positioning performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, machine learning and image analysis combining computer technology, control technology and artificial intelligence technology, and in particular to a method and system for identifying and locating non-standard components of a printed circuit board based on vision. Background Art

[0002] In recent years, with the rapid development of the electronic chip industry, it has brought far-reaching impacts to the world economy and society. The lightweight and diversification of electronic chips are increasingly becoming the direction of future development. In the electronic chip industry, electronic chip assembly technology is an indispensable key technology. Its main task is to install electronic chips and components that realize various specific functions on printed circuit boards, realize the assembly and integration of electronic chips, and produce terminal electronic products. Electronic mounting equipment can realize the automatic assembly of standardized electronic components, but due to the large variety of non-standard special-shaped components, large differences in size and shape, and the relatively low utilization rate of single pieces, the automatic assembly of non-standard components has always been a difficult problem that the industry is looking forward to overcoming.

[0003] The positioning of electronic components is a key step in the automatic insertion of electronic chip components and is the basis for realizing high-precision electronic component automated assembly production lines. Effectively solving the positioning problem of electronic components can greatly improve the efficiency of electronic component assembly, reduce assembly costs, and has high industrial value.

[0004] In the prior art, the positioning methods for non-standard components on printed circuit boards usually use curve fitting, edge detection, and template matching methods. Among them, curve fitting usually requires prior knowledge of the component contour to determine the degree and number of terms of the curve equation; the edge detection method is easily disturbed by the complex image texture on the component or circuit board; and the template matching method has high requirements on image quality. If the component image is dirty or blurred, positioning deviation is more likely to occur. Although the above methods have their own characteristics, they each have certain problems and cannot achieve high positioning performance. Summary of the invention

[0005] The present specification provides a method and system for identifying and locating non-standard components on a printed circuit board based on vision, so as to overcome at least one technical problem existing in the prior art.

[0006] In a first aspect, according to an embodiment of this specification, a method for identifying and locating non-standard components on a printed circuit board based on vision is provided, the method comprising the following steps:

[0007] Collect image data and perform preprocessing:

[0008] Acquire an input image Input including a complete circuit board without components installed and a component slot, and collect input image data;

[0009] Acquire a reference image Reference of a reference circuit board to be processed that includes complete installed components, and collect reference image data; the number of channels of the reference image is consistent with the number of channels of the input image;

[0010] Performing a color space conversion preprocessing operation on the collected input image and reference image respectively to obtain a preprocessed input image Input′ and a preprocessed reference image Reference′;

[0011] Constructing a positioning neural network model for the non-standard components of the printed circuit board, and using the collected and pre-processed multiple groups of sample images to train the positioning neural network model:

[0012] The preprocessed input image Input′ and reference image Reference′ are used as a group of sample images;

[0013] Constructing the positioning neural network model; the positioning neural network model includes an input layer, six hidden layers and an output layer;

[0014] Each group of sample images is used as the input layer of the positioning neural network model, and the component installation marks corresponding to the input images in the sample image group are used as the model output layer, which are substituted into the positioning neural network model, and the positioning neural network model is trained using a back propagation algorithm;

[0015] Using the trained positioning neural network model, the non-standard components of the printed circuit board are identified and positioned:

[0016] When a printed circuit board without components installed is given and needs to be positioned, a printed circuit board with components installed is randomly taken out from the sample library;

[0017] Collecting the positioning image to be identified of the printed circuit board without components installed and the sample reference image of the printed circuit board with components installed; wherein the number of channels of the positioning image to be identified is consistent with the number of channels of the sample reference image;

[0018] The positioning image to be identified and the sample reference image are respectively input as the input image Input and the reference image Reference into the trained positioning neural network model, and the non-standard component position of the printed circuit board without installed components to be positioned is located according to the output of the positioning neural network model.

[0019] Optionally, the input image Input=[R1, G1, B1], and the reference image Reference=[R2, G2, B2]; the preprocessing operation of color space conversion is performed on the collected input image and reference image to obtain the preprocessed input image Input′ and the preprocessed reference image Reference′ specifically includes:

[0020] The input image and the reference image are converted from the RGB space to the HSI space by using a geometric derivation method to obtain three channel data I1, S1, H1 of the converted input image and three channel data I2, S2, H2 of the converted reference image;

[0021] The three channels after the conversion of the input image and the three channels after the conversion of the reference image are respectively subjected to two-dimensional convolution filtering processing to obtain the preprocessed input image Input′ and the reference image Reference′; the two-dimensional convolution filtering processing process is as follows:

[0022]

[0023]

[0024] In formula (1) and formula (2), K I , K S They are the convolution kernels used for the I and S channels respectively.

[0025] Further optionally, the convolution kernel

[0026] Further optionally, the nodes of the input layer of the positioning neural network model are defined as and Together, the nodes of the first hidden layer are defined as and Together, the nodes of the second hidden layer are defined as and Together, the nodes of the third hidden layer are defined as and Together, the nodes of the fourth hidden layer are defined as and Together, the nodes of the fifth hidden layer are defined as and Together, the nodes of the sixth hidden layer are defined as and Together, the output layer is defined as in,

[0027]

[0028]

[0029] In formula (3) and formula (4), A node representing a portion of the input layer related to the preprocessed input image Input′; represents the node of the input layer related to the preprocessed reference image Reference′; c=1, 2, 3 represent the three channels of the preprocessed input image Input′ and the reference image Reference′; (x, y) represents the coordinates in the two-dimensional image;

[0030]

[0031]

[0032] In formula (5) and formula (6), Represents the nodes of the corresponding coordinates in the input layer; Represents the nodes of the corresponding coordinates in the input layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b0 is the linear bias; σ(·) is the activation function;

[0033]

[0034]

[0035] In formula (7) and formula (8), Represent the nodes with corresponding coordinates in the first hidden layer; Represent the nodes with corresponding coordinates in the first hidden layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b1 is the linear bias; σ(·) is the activation function;

[0036]

[0037]

[0038] In formula (9) and formula (10), Nodes representing the corresponding coordinates in the second hidden layer; represents the node with the corresponding coordinates in the second hidden layer; max and min represent the minimum value of all pixels in the window centered at (4x, 4y); p and q represent the positive integer coordinates of the position in the window; b2 is the linear bias; σ(·) is the activation function;

[0039]

[0040]

[0041] In formula (11) and formula (12), Represent the nodes with corresponding coordinates in the third hidden layer; represents the node with the corresponding coordinates in the third hidden layer; max and min represent the minimum value of all pixels in the window centered at (4x, 4y); p and q represent the positive integer coordinates of the position in the window; b3 is the linear bias; σ(·) is the activation function;

[0042]

[0043]

[0044] In formula (13) and formula (14), Nodes representing the corresponding coordinates in the fourth hidden layer; Nodes representing the corresponding coordinates in the fourth hidden layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b4 is the linear bias; σ(·) is the activation function;

[0045]

[0046]

[0047] In formula (15) and formula (16), Nodes representing the corresponding coordinates in the fifth hidden layer; represents the node with the corresponding coordinates in the fifth hidden layer; W represents the total number of pixels under one channel of the convolution window; p and q represent the positive integer coordinates of the position in the convolution window; b5 is the linear bias; σ(·) is the activation function;

[0048]

[0049] In formula (17), p and q represent the positive integer coordinates of the position, which are the node range of the sixth hidden layer; c represents the number of channels (c = 1, 2, 3); d represents whether the node of the previous layer belongs to the u6 or v6 part. When d = 1, When d = 2, represents the corresponding node in the sixth hidden layer; b6 is the linear bias; σ(·) is the activation function; For the node The corresponding independent weights.

[0050] Further optionally, the activation function σ(·) is defined as:

[0051]

[0052] In formula (18), the parameter α is used to adjust the convergence speed of the function.

[0053] Further optionally, the adopting the back propagation algorithm to train the positioning neural network model specifically includes:

[0054] The positioning neural network model is trained using a back propagation algorithm to determine the weights w1, w2, w5, w6, b0, b1, b2, b3, b4, b5, and b6 in equations (5)-(17).

[0055] Further optionally, locating the non-standard component position of the printed circuit board without components installed to be located according to the output of the positioning neural network model specifically includes:

[0056] The localization neural network model recognizes the input and outputs a set of two-dimensional matrices The two-dimensional matrix The size of is the same as the size of the positioning image to be identified;

[0057] According to the two-dimensional matrix The value of the pixel at the middle position determines whether the position is a component;

[0058] The set of all component position pixels is used as the output result of non-standard component positioning.

[0059] Optionally, the input image and the reference image are both grayscale single-channel images, or the input image and the reference image are both color three-channel images.

[0060] In a second aspect, according to an embodiment of this specification, a vision-based printed circuit board non-standard component identification and positioning system is provided, and the system is used to implement the vision-based printed circuit board non-standard component identification and positioning method described in the first aspect above; the system includes:

[0061] A data acquisition module, used for collecting sample images and obtaining image data;

[0062] A preprocessing module, used for preprocessing the collected sample image;

[0063] A neural network model module is used to construct a positioning neural network model and train the positioning neural network model using a plurality of groups of preprocessed sample images;

[0064] The identification and positioning module is used to identify and locate non-standard components on the printed circuit board based on the trained positioning neural network model.

[0065] The beneficial effects of the embodiments of this specification are as follows:

[0066] The method for identifying and locating non-standard components on a printed circuit board takes an image or video of the printed circuit board and uses machine vision, machine learning and other technologies to identify and determine the contour area of ​​the non-standard components on the printed circuit board, thereby realizing the positioning of the non-standard components on the printed circuit board, so as to solve the problem of identifying and automatically locating non-standard components on the printed circuit board.

[0067] The non-standard component identification and positioning method for printed circuit boards adopts multiple layered neural network models to model printed circuit board images, establishes a neural network model with non-standard component identification and positioning capabilities by designing the connection mode and connection weights of neurons between adjacent layers of the neural network, and trains the neural network model. The trained neural network model is used to identify the non-standard components of the printed circuit board and output the positioning results. The positioning accuracy is higher and the false alarm rate is lower. The contour area of ​​the non-standard electronic components can be automatically located from the image or video containing the printed circuit board and the non-standard electronic components. It has high positioning accuracy and positioning speed, improves the assembly efficiency of the electronic components, and can achieve higher positioning performance in complex environments such as image contamination, blur, light, and texture interference where the automatic operation effect is poor, and can make full use of the prior information of the printed circuit board image, greatly reducing the reliance on the prior of the contour of the non-standard component itself in the prior art, and is applicable to non-standard components and standard components of arbitrary contours, with strong applicability and reduced assembly costs. In addition, the non-standard component identification and positioning method for printed circuit boards has no strict requirements on the size of the input image or video, and can use any image size, and can be applicable to color RGB images or grayscale single-channel images, with stronger adaptability.

[0068] The innovative features of the embodiments of this specification include:

[0069] 1. In this embodiment, multiple layered neural network models are built by designing the connection mode and connection weights of neurons between adjacent layers of the neural network, and multiple groups of sample images are used to train them, so that the trained neural network model has the ability to recognize and locate non-standard components. It only needs to input the to-be-identified and located images and the corresponding sample images with the same number of channels into the trained neural network model, and then a set of two-dimensional matrices with the same size as the input image can be output. Whether the pixel at the corresponding image position is a component is determined by the size of the element value in the matrix. The pixel set where all the pixels at the position are components is the positioning result of the non-standard component, and the positioning accuracy is higher and the false alarm rate is lower.

[0070] 2. In this embodiment, since the method for identifying and locating non-standard components on a printed circuit board is based on positioning and judging the position pixels in the image, it can automatically locate the contour area of ​​non-standard electronic components from an image or video containing printed circuit boards and non-standard electronic components, and has high positioning accuracy. Moreover, the method uses a trained model for judgment, which is more efficient and faster in positioning. Even in complex environments such as image contamination, blur, lighting, and texture interference where the automatic operation effect is poor, higher positioning performance can be achieved.

[0071] 3. In this embodiment, the neural network model in the method for identifying and locating non-standard components on a printed circuit board is trained using preprocessed image data, which minimizes noise such as illumination and motion, provides a good input data source, and can fully utilize the prior information of the printed circuit board image, greatly reducing the reliance on the prior information of the non-standard component's own contour in the prior art. It is applicable to non-standard components and standard components with arbitrary contours, has strong applicability, and reduces assembly costs.

[0072] 4. In this embodiment, since the method for identifying and locating non-standard components on a printed circuit board processes images by converting them into the HIS space, there is no restriction on the size of the image, so there is no strict requirement on the size of the input image or video. This solves the problem that some traditional methods require the template to be cropped to a specified size, and any image size can be used, which is highly adaptable.

[0073] 5. In this embodiment, in order to highlight the differences between different targets in the image, the collected images need to be converted into the HIS space. Therefore, the printed circuit board non-standard component identification and positioning method is suitable for color RGB images or grayscale single-channel images. In the specific application process, a more appropriate choice can be made according to the actual application situation, which solves the incompatibility problem of color RGB images and grayscale single-channel images and has stronger applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic diagram of the layout of the equipment and printed circuit board for collecting image data in the vision-based printed circuit board non-standard component identification and positioning method provided in the embodiments of this specification;

[0076] Figure 2A schematic diagram of a positioning neural network model involved in the vision-based printed circuit board non-standard component identification and positioning method provided in an embodiment of this specification;

[0077] Figure 3 A structural block diagram of a vision-based printed circuit board non-standard component identification and positioning system provided in an embodiment of this specification;

[0078] Explanation of the accompanying drawings: 1 is a shooting device, 2 is a sample image, 3 is a printed circuit board, 4 is a component, 5 is a data acquisition module, 6 is a preprocessing module, 7 is a neural network model module, and 8 is a recognition and positioning module. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of this specification and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0081] The embodiment of this specification discloses a method for identifying and locating non-standard components of a printed circuit board based on vision. The method for identifying and locating non-standard components of a printed circuit board comprises the following steps:

[0082] Step 1: Collect image data and perform preprocessing;

[0083] Step 2: constructing a positioning neural network model for the non-standard components of the printed circuit board, and training the positioning neural network model using the multiple sets of sample images collected and preprocessed;

[0084] Step 3: using the trained positioning neural network model to identify and locate the non-standard components on the printed circuit board.

[0085] The above steps are described in detail below.

[0086] Step 1: Collect image data and perform preprocessing.

[0087] That is, the acquisition and preprocessing of input image data of non-standard components on printed circuit boards refers to using a single camera to capture images or videos of printed circuit boards, complete the acquisition of input image data, and preprocess the input image data before component positioning. Image data acquisition and preprocessing are the first steps in general image processing and machine vision tasks, and they play a more critical role. Preprocessing and image acquisition methods are usually designed in combination to minimize noise caused by lighting, motion, etc. in the real world, providing a good input data source for machine vision tasks in the following steps.

[0088] It should be noted and understood that the images or videos taken by a single camera described in the embodiments of this specification, when used in steps 1, 2, and 3, are all based on "sheets" (images) or "frames" (videos) as basic units. When "images" or "videos" are used as references to processed data in the following text, the two terms are equivalent and refer to a basic unit of data. Single sheet / frame data includes three channels R, G, and B, generally referring to color images, which usually correspond to grayscale images (single channel) in the field of image processing. The method described in the embodiments of this specification is also applicable to single-channel grayscale images, and is adapted according to the method described later.

[0089] Step 1.1, obtaining an input image Input including a complete circuit board without components installed and a component slot, and collecting input image data.

[0090] The acquisition of input image data refers to using a single camera to shoot an image, or capturing a complete frame of image from the video captured by the camera as input image data. When the input image data is a three-channel color image, it is recorded as Input = [R1, G1, B1]; when the input image data is a single grayscale channel, it is recorded as Input = [I1]. In the above expression, each letter represents a channel.

[0091] The acquired image should include the entire board and component slots. In addition, as an optimization option, such as Figure 1 As shown, during shooting, the camera 1 can be appropriately away from the printed circuit board 3, and the image of the printed circuit board 3 can be positioned in the central area of ​​the sample image 2, away from the edge of the image, and symmetrical around the optical axis of the camera 1 to reduce the adverse effects of lens distortion, etc., and the component 4 is located on the printed circuit board 3. However, this is not a necessary condition for the method of the present invention to be established. As long as the image of the printed circuit board 3 is completely within the image of the camera 1, the method for identifying and locating non-standard components on a printed circuit board of the present invention can be implemented.

[0092] Step 1.2, obtain a reference image Reference of a reference circuit board to be processed that includes complete mounted components, and collect reference image data.

[0093] The acquisition of reference image data refers to preparing another circuit board that is the same as the circuit board to be operated, but has already installed the components to be installed (except for the installed components, other conditions are the same), and taking an image in the manner described in step 1.1. The number of channels of the image taken is the same as the number of channels of the image taken in step 1.1, which is used as the reference image data. When the input image data is a color three-channel, the reference image data is recorded as Reference = [R2, G2, B2], and when the input image data is a grayscale single channel, it is recorded as Reference = [I2].

[0094] Step 1.3: Perform color space conversion preprocessing operations on the collected input image and reference image respectively to obtain a preprocessed input image Input′ and a preprocessed reference image Reference′.

[0095] In the embodiment of the present specification, the preprocessing method and steps for the input image data and the reference image data in this step are completely consistent, so the preprocessing method is only introduced by taking the input image data Input as an example.

[0096] Step 1.3.1 uses geometric derivation to convert the input image from RGB space to HSI space in order to highlight the differences between different targets in the image. Assuming that the value range of a single channel of the image is [0,1], the formula of the geometric derivation method is as follows:

[0097]

[0098]

[0099]

[0100] In the above formula (20), min represents the minimum value of the three; in formula (21), θ is defined as follows:

[0101]

[0102] In formula (22), cos -1 Represents the inverse trigonometric cosine function.

[0103] When the input is single-channel, there is only I1 channel, and this step 1.3.1 is omitted.

[0104] Step 1.3.2: Perform two-dimensional convolution filtering on the three channels respectively.

[0105] Among them, the convolution kernel K used for the I1 channel I is an approximate Gaussian kernel, which aims to remove noise in the grayscale channel:

[0106]

[0107] The convolution kernel K used for the S1 channel S is a mean function, the purpose of which is to suppress the regional oversaturation phenomenon in the S channel:

[0108]

[0109] No convolution kernel is applied to channel H1.

[0110] The input image after preprocessing in steps 1.3.1 and 1.3.2 is Input′, and its channels are:

[0111]

[0112] in, is the convolution symbol. The corresponding reference image becomes:

[0113]

[0114] I2, S2, and H2 are the results obtained after spatial transformation of the parameter image Reference according to (19)–(21). When the input data is a single channel, Input′ and Reference′ only retain the first group I channel.

[0115] Step 2: construct a positioning neural network model for the non-standard components of the printed circuit board, and train the positioning neural network model using the multiple sets of sample images that have been collected and preprocessed.

[0116] That is, the construction and learning method of the neural network model for positioning non-standard components on a printed circuit board takes the preprocessed input image Input' and the reference image Reference' obtained in step 1 as input. Each Input' and Reference' (three channels or single channel) are regarded as a pair and input into the positioning neural network model together. The definition of the positioning neural network model is as follows.

[0117] like Figure 2 As shown in the figure, the positioning neural network model of non-standard components of printed circuit boards consists of an input layer, an output layer and a hidden layer. Each layer contains several nodes, each of which corresponds to a scalar data, also known as a neuron. A node in a certain layer has a connection with a node in the previous layer, corresponding to an operation function. The connection between any two nodes in two adjacent layers is not necessarily there, but any node in any layer must be connected to at least one node in the previous layer; if each node in a layer is connected to all nodes in the previous layer, then the layer is called a fully connected layer. The relationship between the input layer, the output layer, and the hidden layer is: the input layer is before the hidden layer, the hidden layer is before the output layer, and the hidden layer can recursively contain multiple layers.

[0118] In the embodiment of this specification, the input layer is the input image Input' and the reference image Reference', and the output layer is a two-dimensional matrix that identifies the position of the components, which has the same size as the input image and the reference image. The hidden layer is the intermediate step of the model.

[0119] S2.1 Definitions:

[0120]

[0121] In formula (3), Represents the nodes of the input layer related to the preprocessed input image Input′, wherein c=1, 2, 3 represent the three channels of the preprocessed input image Input′; (x, y) represents the coordinates in the two-dimensional image.

[0122]

[0123] In formula (4), Represents the nodes of the input layer that are related to the preprocessed reference image Reference′, wherein c=1, 2, 3 represent the three channels of the preprocessed reference image Reference′; and (x, y) represent the coordinates in the two-dimensional image.

[0124] and The nodes that together form the input layer. It can be seen that the input layer is composed of the pixel values ​​of all channels of the input image Input′ and the reference image Reference′.

[0125] S2.2 Definitions:

[0126]

[0127] In formula (5), is the convolution window corresponding to channel c (c=1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window. In the embodiment of this specification, The window size is 7×7, indicating The value range of p and q is 1 to 7; The window size is 5×5; The window size is 3×3; the value range of p and q varies with the window size; represents the node with the corresponding coordinate in the input layer; b0 is the linear bias; σ(·) is the activation function.

[0128]

[0129] In formula (6), Represents the nodes of the corresponding coordinates in the input layer; b0 has the same meaning as in formula (5).

[0130] and Together they form the nodes of the first hidden layer.

[0131] S2.3 Definitions:

[0132]

[0133] In formula (7), is the convolution window corresponding to channel c (c=1, 2, 3), p and q represent the positive integer coordinates of the position in the convolution window. In the embodiment of this specification, The window size is 5×5; The window size is 3×3; The window size is 3×3; the value range of p and q depends on the window size; represents the node with the corresponding coordinates in the first hidden layer; b1 is the linear bias; σ(·) is the activation function.

[0134]

[0135] In formula (8), Represent the nodes with corresponding coordinates in the first hidden layer; b1 has the same meaning as in formula (7).

[0136] and Together they form the nodes of the second hidden layer.

[0137] The two convolutional layers, the first hidden layer and the second hidden layer, are used to extract micro-scale features of the input image Input′ and the reference image Reference′.

[0138] S2.4 Definitions:

[0139]

[0140] In formula (9), max and min represent the minimum values ​​of all pixels in the window range centered at (4x, 4y); p and q represent the positive integer coordinates of the position in the window; in the embodiment of this specification, the window size of max and min is 4×4; the value range of p and q is [1,4]; represents the node with the corresponding coordinates in the second hidden layer; b2 is the linear bias; σ(·) is the activation function.

[0141]

[0142] In formula (10), represents the node with the corresponding coordinate in the second hidden layer; the meanings of max, min, and b2 are the same as those in formula (9).

[0143] and Together they make up the nodes of the third hidden layer. By definition, the number of nodes in the third hidden layer is approximately 1 / 16 of the number of nodes in the second hidden layer.

[0144] S2.5 Definitions:

[0145]

[0146] In formula (11), max and min represent the minimum values ​​of all pixels in the window range centered at (4x, 4y), and p and q represent the positive integer coordinates of the position in the window; in the embodiment of this specification, the window size of max and min is 4x4; the value range of p and q is [1,4]; represents the nodes with corresponding coordinates in the third hidden layer; b3 is the linear bias; σ(·) is the activation function.

[0147]

[0148] In formula (12), represents the node with the corresponding coordinate in the third hidden layer; the meanings of max, min, and b3 are the same as those in formula (11).

[0149] and Together they make up the nodes of the fourth hidden layer. By definition, the number of nodes in the fourth hidden layer is approximately 1 / 16 of the number of nodes in the third hidden layer.

[0150] The main functions of the third hidden layer and the fourth hidden layer are: first, to remove noise by minimizing the maximum value; second, to change the scale of the image so that the subsequent steps can extract image features at different scales.

[0151] S2.6 Definitions:

[0152]

[0153] In formula (13), is the convolution window corresponding to channel c (c=1, 2, 3), p and q represent the positive integer coordinates of the position in the convolution window. In the embodiment of this specification, The window size is 7×7; The window size is 5×5; The window size is 5×5; the value range of p and q depends on the window size; represents the node with the corresponding coordinates in the fourth hidden layer; b4 is the linear bias; σ(·) is the activation function.

[0154]

[0155] In formula (14), Nodes representing the corresponding coordinates in the fourth hidden layer; b4 has the same meaning as in formula (13).

[0156] and The nodes that together form the fifth hidden layer. The main purpose of the fifth hidden layer is to extract the features of the input image Input′ and the reference image Reference′ at a macro scale.

[0157] S2.7 Definitions:

[0158]

[0159] In formula (15), W represents the total number of pixels in one channel of the convolution window (the weights of the convolution window are all 1, which is equivalent to averaging the pixels in the convolution window), p and q represent the positive integer coordinates of the position in the convolution window, the convolution window is 4×4 here, and the value range of p and q is [1,4]; represents the nodes with corresponding coordinates in the fifth hidden layer; b5 is the linear bias; σ(·) is the activation function.

[0160]

[0161] In formula (16), represents the node with the corresponding coordinate in the fifth hidden layer; W and b5 have the same meaning as in formula (15).

[0162] and It should be noted that according to equations (15) and (16), the sixth hidden layer is reduced to 1 / 16 of the fifth hidden layer.

[0163] S2.8 Definitions:

[0164]

[0165] In formula (17), p and q represent the positive integer coordinates of the position, which are the node ranges of the sixth hidden layer, c represents the number of channels (c = 1, 2, 3); d represents whether the node of the previous layer belongs to the u6 or v6 part. When d = 1, When d = 2, represents the corresponding node in the sixth hidden layer; b6 is the linear bias; σ(·) is the activation function; For the node Corresponding independent weights. From this we can see that It represents a fully connected layer, which is connected to every node in the sixth hidden layer.

[0166] It should be noted that The size of each node is the same as that of a single channel of the input image and the reference image. The value of each node is 0 or 1. When the value is 0, it indicates that the pixel at this position (corresponding to the input image) is not a component; when the value is 1, it indicates that the pixel at this position is part of the component. All pixels with a value of 1 in jointly mark the location of non-standard components in the input image. Denoted as the output layer of the positioning neural network model.

[0167] Wherein, the σ(·) is called an activation function, which is used to enable the network to have the ability to classify nonlinear data sets. The activation function σ(·) in the embodiment of this specification is defined as follows:

[0168]

[0169] In formula (18), parameter α is used to adjust the convergence speed of the function. By adjusting the convergence speed of the function, it helps to improve the learning effect.

[0170] During learning, select several printed circuit boards that have not been installed with components and several printed circuit boards that have been installed with components, and take sample images. Each set of sample images includes an image of a printed circuit board that has not been installed with components and an image of a printed circuit board that has been installed with components, and only one printed circuit board appears in each image. Preferably, the shooting can be performed with reference to the shooting suggestions described in step 1.

[0171] Each group of sample images is used as the input layer of the positioning neural network model, and the component installation mark (0 or 1) corresponding to the input image in the sample image group (i.e., the printed circuit board image without components installed) is used as the output layer of the positioning neural network model. Substitute it into the positioning neural network model, and use the back propagation algorithm to train the positioning neural network model to determine the weights w1, w2, w5, w6, b0, b1, b2, b3, b4, b5, b6 in equations (5)-(17). After training, a neural network model that can be used for recognition is obtained. Thereby completing the training of the positioning neural network model.

[0172] The above is one of the innovative points of the embodiments of this specification. By designing the connection mode and connection weights of neurons between adjacent layers of the neural network to build multiple layered neural network models, the neural network model has the ability to identify and locate non-standard components.

[0173] In the embodiments of the present specification, the above steps describe the method according to the case of three channels (c=1, 2, 3). When the sample is a single-channel image, refer to the relevant steps and discard the parts about c=2 and c=3, which does not affect the implementation of the method of the embodiments of the present specification.

[0174] Step 3: using the trained positioning neural network model to identify and locate the non-standard components on the printed circuit board.

[0175] That is, the specific process of identifying and locating non-standard components on printed circuit boards. The neural network model is obtained according to step 2. When a printed circuit board without components to be located is given, a printed circuit board with components installed is randomly selected from the sample library. As a preferred method, the relative positions of the printed circuit boards in the two images are as consistent as possible. The number of channels of the image to be identified and located is consistent with the number of channels of the sample reference image. The above two images are respectively input and reference into the neural network model. The model recognizes the input and outputs a set of two-dimensional matrices Its size is the input image size and its value is between [0,1].

[0176] If the value is greater than 0.5, it means that the pixel at that position is a component; the set of all the aforementioned pixels is output as the result of non-standard component positioning, and the non-standard component positioning of the printed circuit board without components installed to be positioned is completed.

[0177] In addition, the printed circuit board non-standard component identification and positioning method in the embodiments of this specification has no strict requirements on the size of the input image or video. Some existing technologies, such as template matching methods, usually require the template to be cropped to a smaller fixed size such as 24×24 pixels. The present invention can use any image size, such as the commonly used 1920×1080 pixels or 800×600 pixels, provided that the computing resources for implementing the method permit.

[0178] Table 1 Comparison of positioning accuracy between the method of the present invention and the prior art

[0179]

[0180] The embodiment of this specification also discloses a vision-based printed circuit board non-standard component identification and positioning system, which is used to implement the above-mentioned vision-based printed circuit board non-standard component identification and positioning method. Specifically, the system includes a data acquisition module 5, a preprocessing module 6, a neural network model module 7 and an identification and positioning module 8.

[0181] Among them, the data acquisition module 5 is used to collect sample images and obtain image data; the preprocessing module 6 is used to preprocess the collected sample images; the neural network model module 7 is used to construct a positioning neural network model, and use the preprocessed multiple groups of sample images to train the positioning neural network model; the identification and positioning module 8 is used to identify and locate non-standard components on the printed circuit board according to the trained positioning neural network model.

[0182] In summary, this specification discloses a vision-based method and system for identifying and locating non-standard components on a printed circuit board. By taking an image or video of a printed circuit board and adopting machine vision, machine learning and other technologies, the contour area of ​​the non-standard components on the printed circuit board is identified and determined, and the positioning of the non-standard components on the printed circuit board is achieved, so as to solve the problem of identifying and automatically locating non-standard components on the printed circuit board.

[0183] The non-standard component identification and positioning method for printed circuit boards adopts multiple layered neural network models to model printed circuit board images, establishes a neural network model with non-standard component identification and positioning capabilities by designing the connection mode and connection weights of neurons between adjacent layers of the neural network, and trains the neural network model. The trained neural network model is used to identify the non-standard components of the printed circuit board and output the positioning results. The positioning accuracy is higher and the false alarm rate is lower. The contour area of ​​the non-standard electronic components can be automatically located from the image or video containing the printed circuit board and the non-standard electronic components. It has high positioning accuracy and positioning speed, improves the assembly efficiency of the electronic components, and can achieve higher positioning performance in complex environments such as image contamination, blur, light, and texture interference where the automatic operation effect is poor, and can make full use of the prior information of the printed circuit board image, greatly reducing the reliance on the prior of the contour of the non-standard component itself in the prior art, and is applicable to non-standard components and standard components of arbitrary contours, with strong applicability and reduced assembly costs. In addition, the non-standard component identification and positioning method for printed circuit boards has no strict requirements on the size of the input image or video, and can use any image size, and can be applicable to color RGB images or grayscale single-channel images, with stronger adaptability.

[0184] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0185] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying and locating non-standard components on a printed circuit board based on vision, characterized in that: The method comprises the following steps: Collect image data and perform preprocessing: Acquire an input image Input including a complete circuit board without components installed and a component slot, and collect input image data; Acquire a reference image Reference of a reference circuit board to be processed that includes complete installed components, and collect reference image data; the number of channels of the reference image is consistent with the number of channels of the input image; Performing a color space conversion preprocessing operation on the collected input image and reference image respectively to obtain a preprocessed input image Input′ and a preprocessed reference image Reference′; Constructing a positioning neural network model for the non-standard components of the printed circuit board, and using the collected and pre-processed multiple groups of sample images to train the positioning neural network model: The preprocessed input image Input′ and reference image Reference′ are used as a group of sample images; Constructing the positioning neural network model; the positioning neural network model includes an input layer, six hidden layers and an output layer; Each group of sample images is used as the input layer of the positioning neural network model, and the component installation marks corresponding to the input images in the sample image group are used as the model output layer, which are substituted into the positioning neural network model, and the positioning neural network model is trained using a back propagation algorithm; Using the trained positioning neural network model, the non-standard components of the printed circuit board are identified and positioned: When a printed circuit board without components installed is given and needs to be positioned, a printed circuit board with components installed is randomly taken out from the sample library; Collecting the positioning image to be identified of the printed circuit board without components installed and the sample reference image of the printed circuit board with components installed; wherein the number of channels of the positioning image to be identified is consistent with the number of channels of the sample reference image; The positioning image to be identified and the sample reference image are respectively input as the input image Input and the reference image Reference into the trained positioning neural network model, and the non-standard component position of the printed circuit board without installed components to be positioned is located according to the output of the positioning neural network model.

2. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 1, characterized in that: The input image Input = [R1, G1, B1], the reference image Reference = [R2, G2, B2]; the preprocessing operation of color space conversion is performed on the collected input image and reference image to obtain the preprocessed input image Input' and the preprocessed reference image Reference' specifically including: The input image and the reference image are converted from the RGB space to the HSI space by using a geometric derivation method to obtain three channel data I1, S1, H1 of the converted input image and three channel data I2, S2, H2 of the converted reference image; The three channels after the conversion of the input image and the three channels after the conversion of the reference image are respectively subjected to two-dimensional convolution filtering processing to obtain the preprocessed input image Input′ and the reference image Reference′; the two-dimensional convolution filtering processing process is as follows: In formula (1) and formula (2), K I , K S They are the convolution kernels used for the I and S channels respectively.

3. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 2, characterized in that: The convolution kernel 4. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 2, characterized in that: The nodes of the input layer of the positioning neural network model are defined as and Together, the nodes of the first hidden layer are defined as and Together, the nodes of the second hidden layer are defined as and Together, the nodes of the third hidden layer are defined as and Together, the nodes of the fourth hidden layer are defined as and Together, the nodes of the fifth hidden layer are defined as and Together, the nodes of the sixth hidden layer are defined as and Together, the output layer is defined as in, In formula (3) and formula (4), A node representing a portion of the input layer related to the preprocessed input image Input′; represents the node of the input layer related to the preprocessed reference image Reference′; c=1, 2, 3 represent the three channels of the preprocessed input image Input′ and the reference image Reference′; (x, y) represents the coordinates in the two-dimensional image; In formula (5) and formula (6), Represents the nodes of the corresponding coordinates in the input layer; Represents the nodes of the corresponding coordinates in the input layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b0 is the linear bias; σ(·) is the activation function; In formula (7) and formula (8), Represent the nodes with corresponding coordinates in the first hidden layer; Represent the nodes with corresponding coordinates in the first hidden layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b1 is the linear bias; σ(·) is the activation function; In formula (9) and formula (10), Nodes representing the corresponding coordinates in the second hidden layer; represents the node with the corresponding coordinates in the second hidden layer; max and min represent the minimum value of all pixels in the window centered at (4x, 4y); p and q represent the positive integer coordinates of the position in the window; b2 is the linear bias; σ(·) is the activation function; In formula (11) and formula (12), Represent the nodes with corresponding coordinates in the third hidden layer; represents the node with the corresponding coordinates in the third hidden layer; max and min represent the minimum value of all pixels in the window centered at (4x, 4y); p and q represent the positive integer coordinates of the position in the window; b3 is the linear bias; σ(·) is the activation function; In formula (13) and formula (14), Nodes representing the corresponding coordinates in the fourth hidden layer; Nodes representing corresponding coordinates in the fourth hidden layer; is the convolution window corresponding to channel c (c = 1, 2, 3); p and q represent the positive integer coordinates of the position in the convolution window; b4 is the linear bias; σ(·) is the activation function; In formula (15) and formula (16), Nodes representing the corresponding coordinates in the fifth hidden layer; represents the node with the corresponding coordinates in the fifth hidden layer; W represents the total number of pixels under one channel of the convolution window; p and q represent the positive integer coordinates of the position in the convolution window; b5 is the linear bias; σ(·) is the activation function; In formula (17), p and q represent the positive integer coordinates of the position, which are the node range of the sixth hidden layer; c represents the number of channels (c = 1, 2, 3); d represents whether the node of the previous layer belongs to the u6 or v6 part. When d = 1, When d = 2, represents the corresponding node in the sixth hidden layer; b6 is the linear bias; σ(·) is the activation function; For the node The corresponding independent weights.

5. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 4, characterized in that: The activation function σ(·) is defined as: In formula (18), the parameter α is used to adjust the convergence speed of the function.

6. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 4, characterized in that: The adopting back propagation algorithm to train the positioning neural network model specifically includes: The positioning neural network model is trained using a back propagation algorithm to determine the weights w1, w2, w5, w6, b0, b1, b2, b3, b4, b5, and b6 in equations (5)-(17).

7. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 4, characterized in that: The method of locating the non-standard component position of the printed circuit board without components installed to be located according to the output of the positioning neural network model specifically includes: The localization neural network model recognizes the input and outputs a set of two-dimensional matrices The two-dimensional matrix The size of is the same as the size of the positioning image to be identified; According to the two-dimensional matrix The value of the pixel at the middle position determines whether the position is a component; The set of all component position pixels is used as the output result of non-standard component positioning.

8. The method for identifying and locating non-standard components on a printed circuit board based on vision according to claim 1, characterized in that: The input image and the reference image are both grayscale single-channel images, or the input image and the reference image are both color three-channel images.

9. A vision-based printed circuit board non-standard component identification and positioning system, characterized in that: The system is used to implement the vision-based printed circuit board non-standard component identification and positioning method described in any one of claims 1-8 above; the system comprises: A data acquisition module is used to collect sample images and obtain image data; A preprocessing module, used for preprocessing the collected sample image; A neural network model module is used to construct a positioning neural network model and train the positioning neural network model using a plurality of groups of preprocessed sample images; The identification and positioning module is used to identify and locate non-standard components on the printed circuit board based on the trained positioning neural network model.