A quartz crystal precise slot detection method and system based on deep learning

Abstract

The application relates to the technical field of the cross of electronic component manufacturing, machine vision and deep learning, and provides a quartz crystal oscillator precision slot detection method and system based on deep learning, which comprises the following steps: collecting crystal oscillator images of different detection stations, and pre-processing the crystal oscillator images to obtain input images; constructing a deep learning network model; performing feature extraction on the input images through a backbone network to obtain a high-dimensional feature map; performing deep-shallow layer feature fusion and further feature extraction on the high-dimensional feature map through a neck network to obtain a prediction input feature map; performing feature extraction on the prediction input feature map through a head network to obtain a prediction feature map; performing post-processing on the prediction feature map to obtain an optimal prediction frame; and performing crystal oscillator slot detection according to the optimal prediction frame. The application realizes the accurate detection and recognition of whether the crystal oscillator in the crystal oscillator image is in the slot, and has the advantages of high detection precision, strong generalization ability, high detection efficiency and low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic component manufacturing, machine vision and deep learning, and in particular to a quartz crystal precise slot detection method and system based on deep learning. BACKGROUND

[0002] As a core component in electronic devices, the manufacturing process of quartz crystal involves multiple complex steps, and whether the crystal is accurately placed in the tray slot is a key link to ensure the accuracy and continuity of the subsequent production process. However, traditional crystal slot detection methods usually rely on manual visual inspection, mechanical sensor detection or simple image processing methods, etc. Due to the complexity and variability of the state of the crystal on the tray, these methods are susceptible to human subjective factors, have low detection efficiency, low detection accuracy, poor robustness and generalization, and cannot adapt to large-scale production, etc., making it difficult to meet the high precision and high efficiency requirements of modern electronic component manufacturing.

[0003] Traditional quartz crystal slot detection methods have poor adaptability to different light source conditions, such as different types of ring light, coaxial light, bar light, etc., and complex background changes, such as tray damage, dirt, scratches, characters, image noise interference, different specifications, sizes, and attitudes of crystals, etc. This leads to unsatisfactory detection results. In addition, when the crystal surface is not uniform in color or has shadows due to light or other reasons, or when two or more crystals overlap, the crystal features change significantly, which significantly reduces the detection accuracy, and even makes it impossible to detect. The detection effect is greatly affected by feature changes, and the robustness is poor, so traditional image processing algorithms are only suitable for situations with obvious features and simple backgrounds, and the recognition effect depends largely on the human-set feature template.

[0004] Each different specification of crystal and tray requires a set of templates and a separate detection process, which cannot well identify the various states of different specifications of crystals on different types of trays (such as multiple crystal stacking, crystal on the edge beam, and slightly overlapping crystal, etc.), has poor adaptability, and cannot identify crystals outside the positioning slot range. Due to the similarity of different slot features and the large difference between different state crystal features, the template matching accuracy is poor. Existing traditional methods and deep learning methods have low detection accuracy for small size and multiple crystal stacking under large field of view, and high miss detection rate. Some crystals in different states have small inter-class difference and large intra-class difference, making it difficult to obtain high recognition accuracy through traditional methods. SUMMARY

[0005] This invention aims to solve at least one of the technical problems existing in related technologies. To this end, this invention provides a deep learning-based method and system for accurate slot placement detection of quartz crystal oscillators. This method achieves accurate detection and identification of whether a crystal oscillator is in its slot in a crystal oscillator image. It has advantages such as high detection accuracy, strong generalization ability, high detection efficiency, and low cost. It has significant practical value in solving the problem of accurate slot placement detection and identification of quartz crystal oscillators and can be widely applied in the field of electronic component manufacturing to improve production efficiency and product quality.

[0006] This invention provides a deep learning-based method for accurate slot detection of quartz crystal oscillators, comprising:

[0007] S1: Acquire crystal oscillator images from different testing stations, and preprocess the crystal oscillator images to obtain the input image;

[0008] S2: Construct a deep learning network model, which includes a Backbone network, a Neck network, and a Head network;

[0009] S3: The Backbone network extracts features from the input image to obtain a high-dimensional feature map; the Neck network performs deep and shallow feature fusion and further feature extraction on the high-dimensional feature map to obtain a predicted input feature map; the Head network extracts features from the predicted input feature map to obtain a predicted feature map.

[0010] S4: The predicted feature map is generated by decoding the predicted bounding box; the predicted bounding box is redundantly removed by non-maximum suppression (NMS) to obtain the optimal predicted bounding box.

[0011] S5: Detect crystal oscillator insertion based on the optimal prediction frame.

[0012] Furthermore, the preprocessing includes geometric transformation, optical transformation, noise addition, and annotation information. The annotation information includes target category, target upper left corner coordinates, and target lower right corner coordinates. The target category includes empty slot without material OK_1, fully loaded slot OK_2, out of slot within the material slot range and needing to adjust the loading NG_1, and out of slot outside the material slot range or reversed material and needing to be picked up and discarded defective product boxes NG_2.

[0013] Furthermore, the Backbone network includes a CBS module, a CSSPX module containing one RES Block, a RES-SE module, a CSSPX module containing two RES Blocks, a RES-SE module, a CSSPX module containing four RES Blocks, a CSSPX module containing six RES Blocks, and a CSSPX module containing eight RES Blocks, stacked sequentially.

[0014] Further, the CBS module comprises a convolution layer, a BN layer and a SiLu activation layer stacked in sequence;

[0015] The RES Block comprises a CBS module with a convolution kernel of 1*1, a CBS module with a convolution kernel of 3*3 and a CBS module with a convolution kernel of 1*1 stacked in sequence;

[0016] The working process of the CSSPX module is as follows:

[0017] The input image of the CSSPX module is down-sampled by a CBS module to obtain a first feature map, the first feature map is extracted by a CBS module and a plurality of ES Block modules and then extracted by a CBS module to obtain a second feature map, the first feature map is extracted by a CBS module to obtain a third feature map, the second feature map and the third feature map are spliced and then extracted by a CBS module to obtain a CSSPX module output feature image.

[0018] Further, the RES-SE module is an EDC-SE module added in the RES Block;

[0019] The working process of the RES-SE module is as follows:

[0020] The output feature map of the CBS module with a convolution kernel of 3*3 is obtained by the EDC-SE module to obtain a weight vector, the weight vector is weighted to the output feature map of the original CBS module with a convolution kernel of 3*3 by multiplication operation, and then the feature obtained by the CBS module with a convolution kernel of 3*3 is added to the input feature of the original RES-SE module to obtain the output feature of the RES-SE module;

[0021] The input feature of the EDC-SE module is divided into two paths, one of which is obtained by a global average pooling layer and a CBS module with a convolution kernel of 1*1 stacked in sequence to obtain a fourth feature, and the other of which is obtained by a global maximum pooling layer and a CBS module with a convolution kernel of 1*1 stacked in sequence to obtain a fifth feature, the fourth feature and the fifth feature are spliced, and then connected with an activation function to obtain the output feature of the EDC-SE module.

[0022] Further, the Neck network uses a deep and shallow layer feature fusion network structure based on FPN and PAN, and increases a SPPF module and a CSSPX module at the front,

[0023] The SPPF module is an improved spatial pyramid pooling layer,

[0024] The working process of the SPPF module is as follows:

[0025] The input feature map of the SPPF module first passes through a CBS module with a 1x1 convolution kernel to extract preliminary features. Then, it passes through three max pooling layers with different kernel sizes for downsampling. The resulting four feature maps are then concatenated by channel. Finally, the concatenated feature map passes through a CBS module with a 1x1 convolution kernel to extract features, thus obtaining the output feature map of the SPPF module.

[0026] Furthermore, the predicted input feature map is output as a predicted feature map of three sizes through the Head network.

[0027] Furthermore, step S5 includes:

[0028] S51: Convert the pixel coordinates of the optimal prediction boxes of the three sizes into actual coordinates in the camera coordinate system;

[0029] S52: Divide the crystal oscillator image under the current detection field of view into regions with the center point of the detection region as the center;

[0030] S53: Sort according to actual coordinates and encode the crystal oscillators falling within the area;

[0031] S54: Traverse all actual coordinates and encode all crystal oscillators falling within the region;

[0032] S55: Detect the crystal oscillator's placement in the slot based on the labeling information and coding.

[0033] Furthermore, in step S53,

[0034]

[0035] in, Encoding for crystal oscillators, The number of regions to divide the current image. for The coordinates of the lower left corner of the region for The coordinates of the lower left corner of the region The coordinates of any pixel in the source image.

[0036] This invention also provides a deep learning-based system for precise slotting detection of quartz crystal oscillators, used to execute the aforementioned deep learning-based method for precise slotting detection of quartz crystal oscillators, comprising:

[0037] The image acquisition module acquires crystal oscillator images from different detection stations and preprocesses the crystal oscillator images to obtain the input image.

[0038] A model construction module constructs a deep learning network model, which includes a Backbone network, a Neck network and a Head network;

[0039] A feature extraction module extracts features of an input image through the Backbone network to obtain a high-dimensional feature map, performs deep and shallow layer feature fusion and further feature extraction on the high-dimensional feature map through the Neck network to obtain a predicted input feature map, and extracts features of the predicted input feature map through the Head network to obtain a predicted feature map;

[0040] An optimization module is configured to generate a predicted bounding box through Decode decoding of the predicted feature map, and remove redundant predicted bounding boxes through NMS non-maximum suppression to obtain an optimal predicted bounding box.

[0041] A detection module detects whether a crystal oscillator is inserted into a slot according to the optimal predicted bounding box.

[0042] The one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0043] The present application extracts and classifies features of crystal oscillator images through a deep learning algorithm, ensures that crystal oscillators in various states under different lines, different fields of view, different sizes, different positions and distances can be accurately detected, significantly improves the accuracy of slot insertion detection, reduces the false detection rate and the missed detection rate, helps to reduce production costs and improve the market competitiveness of enterprises.

[0044] The present application realizes real-time detection of whether multiple crystal oscillators are accurately inserted into slots under multiple fields of view, triggers corresponding feedback mechanisms according to the detection results, realizes automatic control and adjustment, reduces manual intervention and downtime, reduces production costs, and improves the automation level and production efficiency of the production line.

[0045] The present application can adapt to crystal oscillators of different models, different sizes, different fields of view and front and back surfaces in various production environments, has strong robustness, generalization and scalability, and can adapt to new situations and new demands of the production environment through continuous updating and optimization of the model, ensuring long-term stable operation.

[0046] Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by practicing the present application. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to make the technical solutions in the present application or the prior art clearer, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and all other drawings obtained by those skilled in the art without creative labor based on the embodiments in the present application also belong to the protection scope of the present application. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0048] Figure 1 is a flowchart of a quartz crystal precise slot detection method based on deep learning provided by the present application.

[0049] Figure 2 is a structural schematic diagram of a CBS module and a RES Block module provided by the present application.

[0050] Figure 3 is a structural schematic diagram of a CSSPX module provided by the present application.

[0051] Figure 4 is a structural schematic diagram of an EDC-SE module provided by the present application.

[0052] Figure 5 is a structural schematic diagram of a RES-SE module provided by the present application.

[0053] Figure 6 is a structural schematic diagram of an SPPF module provided by the present application.

[0054] Figure 7 is a structural schematic diagram of a deep learning network model provided by the present application.

[0055] Figure 8 is a structural schematic diagram of a quartz crystal precise slot detection system based on deep learning provided by the present application.

[0056] Reference signs:

[0057] 101, image acquisition module; 102, model construction module; 103, feature extraction module; 104, optimization module; 105, detection module. DETAILED DESCRIPTION

[0058] In order to make the technical solutions in the present application or the prior art clearer, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and all other drawings obtained by those skilled in the art without creative labor based on the embodiments in the present application also belong to the protection scope of the present application. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0059] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0060] The application is described below in conjunction with Figures 1 to 8 A deep learning-based quartz crystal precise slot detection method and system are described.

[0061] As Figure 1 shown, a deep learning-based quartz crystal precise slot detection method includes:

[0062] S1: Collecting crystal vibration image at different detection stations, and pre-processing the crystal vibration image to obtain an input image;

[0063] In some specific embodiments of the present application, an industrial camera with a resolution of 2448*2048 is used in combination with a FA lens (Factory Automation Lens, industrial automation lens) to collect image data of the crystal vibration in different states on the online body detection station, including different number and specification of crystal vibration, different types of tray, different field of view, empty slot, complete slot, not complete slot, complete slot, different degree of edge, crystal vibration scattered in each position of the tray and various stacking materials, etc. The detection station includes point glue slot detection, fine adjustment slot detection, sealing and welding slot detection.

[0064] The collected crystal vibration image is pre-processed and high-quality labeled using a labeling tool. The label is two OK categories and two NG categories, respectively OK_1: empty slot without material, OK_2: complete slot, NG_1: out of slot, within the material slot range, need to adjust the slot, NG_2: out of slot, outside the material slot range, reverse material, need to suck and throw the defective product box.

[0065] The pre-processing includes geometric transformation, optical transformation, noise addition, and label information. The label information includes target category, target top-left corner coordinate, target bottom-right corner coordinate, the target category includes empty slot without material OK_1, complete slot OK_2, out of slot within the material slot range and need to adjust the slot NG_1, and out of slot outside the material slot range or reverse material need to suck and throw the defective product box NG_2. The above information is uniformly saved in a txt file.

[0066] Divide the pixel value by 255 to convert from 0-255 to 0-1 for normalization; finally, the image is scaled, translated to the center, and the redundant part is filled.

[0067] In some embodiments of the application, the input image is scaled by a factor of 1, the image resolution is converted from 2448*2048 to a set input image size of 608*608, then the scaled image is translated to the center and the redundant part of the scaled image is filled with 0. The whole process can be completed by an affine transformation matrix, and the calculation expression is:

[0068]

[0069] wherein, (x,y) is the coordinate of any pixel point on the target image, , is the width of the source image, is the height of the source image, is the width of the target image, is the height of the target image, is the scaling factor, which is the minimum value of the width ratio and the height ratio of the target image to the source image, and (x,y) is the coordinate of any pixel point on the source image. Through the matrix, the coordinates of any pixel value of the source image can be obtained.

[0070] S2: Construct a deep learning network model, which includes a Backbone network, a Neck network and a Head network; the deep learning network model is as shown in

[0071] . Figure 7

[0072] The Backbone network includes a CBS module, a CSSPX module containing 1 RES Block, a RES-SE module, a CSSPX module containing 2 RES Blocks, a RES-SE module, a CSSPX module containing 4 RES Blocks, a CSSPX module containing 6 RES Blocks, and a CSSPX module containing 8 RES Blocks, which are stacked in sequence.

[0073] As shown in Figure 2 , the CBS module includes a convolution layer (Conv), a BN layer and a SiLu activation layer, which are stacked in sequence;

[0074] The RES Block includes a CBS module with a convolution kernel of 1*1, a CBS module with a convolution kernel of 3*3, and a CBS module with a convolution kernel of 1*1, which are stacked in sequence.​

[0075] As shown in Figure 3 , the working process of the CSSPX module is as follows:

[0076] The image input by the CSSPX module is down-sampled by the CBS module to obtain a first feature map, the first feature map is extracted by a CBS module and a plurality of ES Block modules and then extracted by another CBS module to obtain a second feature map, the first feature map is extracted by a CBS module to obtain a third feature map, the second feature map and the third feature map are spliced (Concat) and then extracted by a CBS module to obtain a CSSPX module output feature image.

[0077] As shown in Figure 5 , the RES-SE module is an EDC-SE module added in the RES Block;

[0078] The working process of the RES-SE module is as follows:

[0079] The output feature map of the CBS module with a convolution kernel of 3x3 is obtained by the EDC-SE module to obtain a weight vector, the weight vector is weighted to the output feature map of the CBS module with a convolution kernel of 3x3 by a multiplication operation (MCL) channel by channel, and then the feature obtained by the CBS module with a convolution kernel of 3x3 is added (Add) to the input feature of the original RES-SE module to obtain the output feature of the RES-SE module.

[0080] As shown in Figure 4 , the input feature of the EDC-SE module is divided into two paths, one of which is obtained by sequentially stacking a global average pooling layer (Global AvgPool) and a CBS module with a convolution kernel of 1x1 to obtain a fourth feature, and the other of which is obtained by sequentially stacking a global maximum pooling layer (Global MaxPool) and a CBS module with a convolution kernel of 1x1 to obtain a fifth feature, the fourth feature and the fifth feature are spliced, and then connected with an activation function (Sigmoid) to obtain the output feature of the EDC-SE module.

[0081] The Neck network uses a deep and shallow feature fusion network structure based on FPN and PAN, and increases SPPF modules and CSSPX modules in the front part.

[0082] As shown in Figure 6 , the SPPF module is an improved spatial pyramid pooling layer,

[0083] The working process of the SPPF module is as follows:

[0084] The input feature map of the SPPF module is first subjected to a CBS module with a convolution kernel of 1x1 to extract preliminary features, and then subjected to three max-pooling layers with different kernel sizes for downsampling operation. Four feature maps obtained are subjected to a channel concatenation (Concat) operation, and the concatenated feature map is subjected to a CBS module with a convolution kernel of 1x1 to extract features, thereby obtaining the output feature map of the SPPF module.

[0085] Figure 6 The three max-pooling layers with different kernel sizes are MaxPool2d (k13, s1, p2), MaxPool2d (k9, s1, p2), and MaxPool2d (k5, s1, p2), wherein k13 is a kernel size of 13x13, corresponding to a receptive field (RF) of RF: 13x13, k9 is a kernel size of 9x9, corresponding to a receptive field (RF) of RF: 9x9, and k5 is a kernel size of 5x5, corresponding to a receptive field (RF) of RF: 5x5, s1 represents a stride of 1, and p2 represents a padding of 2.

[0086]

[0087] The model is trained using the prepared data set in the deep learning environment, and the model is improved and optimized in a targeted manner according to the loss curve and various performance indicators of the validation set, thereby obtaining optimal weights and model parameters. Specifically, the model is evaluated in terms of performance using five performance indicators, namely, precision (P), recall (R), F1 value, AP (Average Precision) value, and MAP (Mean Average Precision) value.

[0088] ​Before the formal training begins, the model is initialized using pre-trained weights. During training, warm-up training is used, that is, a smaller learning rate initial value is used at the beginning of training, so that the model can understand the data distribution as 0 at the beginning, and not overfit to the small batch size of the image. In the present application, the learning rate initial value is set to 0.001, and the learning rate gradually rises to 0.01 in the first 6 rounds of training. The input image size is set to 608*608 pixels, the Epoch is set to 200, the Batch size is set to 8, and the weight decay is set to 0.0005. The network weight file is saved once after each Epoch of training, and the model is monitored in real time using the Tensorboard tool to record the loss value change of the model and the performance indicators on the validation set. Since the validation loss tends to be stable and does not decrease after 120 rounds, it indicates that the model has been fitted at this time. After 120 rounds, the validation loss fluctuates and rises, while the training loss is still decreasing, indicating that the model has been overfitted at this time. Since the model has been fitted at around 120 rounds, and the MAP value takes into account the accuracy and recall rate of all categories, the present application takes the maximum value of MAP of each 10 rounds around 120 rounds, and the maximum value is 99.7% which appears in the 117th round. The weight obtained in this round is taken as the final model weight.

[0089] The test set of the present application contains crystal vibration images in various complex situations, which basically conforms to the distribution of the overall data set. The MAP value of the finally optimized model on the test set can reach 99.5%, and the AP value of each category can reach 99.5%; the average detection speed on the RTX4060 GPU is 45.6 frames / s, and the detection accuracy and speed meet the actual production requirements; the F1 value is the harmonic mean of accuracy and recall, which changes with the confidence threshold value. When the confidence threshold value is 0.683, the F1 value reaches a maximum value of 1, at which time the performance of the model on the test set is best. After the final model training is completed, the optimal weight file is obtained.

[0090] S3: extracting features from the input image through the Backbone network to obtain a high-dimensional feature map; performing deep and shallow feature fusion and further feature extraction on the high-dimensional feature map through the Neck network to obtain a predicted input feature map; and extracting features from the predicted input feature map through the Head network to obtain a predicted feature map;

[0091] The input image is subjected to feature extraction by the Backbone network, and the input image will be subjected to a series of convolution, BN, activation and other calculations, and after 32 times down-sampling, a high-dimensional feature map with a resolution of 19*19 is generated. The calculation expression of the high-dimensional feature map is:

[0092]

[0093]

[0094] wherein, is the height of the high-dimensional feature map, is the width of the high-dimensional feature map, is the height of the input image, is the width of the input image, is the padding width, is the stride adopted in the convolution, is the width of the convolution kernel, is the height of the convolution kernel. By modifying the size of the convolution kernel and padding and stride, the size of the high-dimensional feature map can be adjusted.

[0095] In the model of the application, only the stride of the convolution kernel in the first CBS module of the CSSPX module in the Backbone network and the two CBS modules in the Neck network is 2, which is 2 times down-sampling, and the rest is 1, which does not down-sample.

[0096] As Figure 7As shown, the input image size is set to 608x608 pixels, the feature map size is reduced to half after passing through each CSSPX module in the backbone network (X represents that there are X RES blocks inside the CSSPX module), the stride s of the CSSPX modules in the neck network and the head network is 1, and no down-sampling is performed. After passing through two CBS modules and one SPPF module in the backbone network and the neck network and performing up-sampling, the feature map with a size of 38x38 output by the CSSP6 module in the backbone network is spliced and fused, and then after passing through one CSSP1 module and five CBS modules and performing up-sampling, the feature map with a size of 76x76 output by the CSSP4 in the backbone network is spliced and fused, and then after passing through one CSSP1 module in the head network, the prediction feature map y1 with a size of 76x76 is obtained, which is used for detecting small crystals. Then, the output feature map y1 with a size of 76x76 is subjected to 2 times down-sampling by passing through a CBS module with a stride of 2, and then the feature map with a size of 38x38 output by the five CBS modules in the neck network is spliced and fused, and then after passing through one CSSP1 module in the head network, the prediction feature map y2 with a size of 38x38 is obtained, which is used for detecting medium-sized crystals. Finally, the output feature map y2 with a size of 38x38 is subjected to 2 times down-sampling by passing through a CBS module with a stride of 2, and then the feature map with a size of 19x19 output by the CBS module before up-sampling in the neck network is spliced and fused, and then after passing through one CSSP1 module in the head network, the prediction feature map y3 with a size of 19x19 is obtained, which is used for detecting large crystals.

[0097] S4: The prediction feature map is decoded by a Decode of the prediction box to generate a prediction box, and the prediction box is subjected to redundancy elimination by NMS to obtain an optimal prediction box.

[0098] The channel number of the three prediction feature maps is 27 (each feature point in the feature map predicts three prior boxes, each prior box needs to predict four classes, one confidence and four positions, so a total of (4+1+4) x 3 = 27-dimensional prediction feature vectors are needed, and a total of (19x19+38x38+76x76) x 3 = 22743 prediction results are generated). The detection frame is also based on the preprocessed picture (i.e., the target image of 608x608), so it is necessary to map the detection frame on the target image back to the source image. This step of transformation can be realized by using the inverse matrix of the affine transformation matrix, and the calculation expression is:

[0099]

[0100] wherein, As the first auxiliary variable, As the second auxiliary variable, As a third auxiliary variable;

[0101] There are 22,743 predicted bounding boxes, most of which overlap. Therefore, redundant predicted bounding boxes need to be removed, and the optimal predicted bounding boxes need to be selected. Thus, all prediction results need to be fed into Soft NMS for calculation. After filtering out the redundant predicted bounding boxes, the final detection result is obtained. The calculation expression is:

[0102]

[0103] in, For the first The confidence score of each predicted bounding box. The prediction box with the highest confidence score among all prediction boxes. For the first The predicted bounding boxes being compared, This is the ratio of the intersection to the union of the two predicted boxes. The threshold is set manually, usually 0.5; σ is the coefficient of the continuous penalty term.

[0104] In some specific embodiments of the present invention .

[0105] S5: Detect crystal oscillator insertion into slot based on the optimal prediction frame;

[0106] S51: Convert the pixel coordinates of the optimal prediction boxes of the three sizes into actual coordinates in the camera coordinate system;

[0107] The pixel coordinates in the detection results are converted to actual coordinates in the camera coordinate system. The calculation expression is as follows:

[0108]

[0109] in, The physical size of a unit pixel on the horizontal axis. The physical size of a unit pixel on the vertical axis. Focal length This is the distance between the crystal oscillator and the optical center. These are the coordinates of the origin of the image coordinate system in the pixel coordinate system. Let be the coordinates of the origin of the image coordinate system in the pixel coordinate system. The coordinates of pixel point p in the pixel coordinate system are (u,v), and the coordinates in the camera coordinate system are (x,y,z).

[0110] S52: Divide the crystal oscillator image under the current detection field of view into regions with the center point of the detection region as the center;

[0111] The image under the current detection field of view is divided into multiple regions with the center point of the detection region as the center, and the crystal vibration coordinates output by the model are sorted, and the crystal vibrations falling in different regions are coded.

[0112] S53: According to the actual coordinates, the crystal vibrations falling in the region are coded;

[0113] The crystal vibration coordinates output by the model are sorted, and the crystal vibrations falling in different regions are coded, and the calculation expression is:

[0114]

[0115] Among them, is the crystal vibration coding, is the number of regions divided by the current image, is the left lower corner coordinate of the region, is the left lower corner coordinate of the region, is the coordinate of any pixel point on the source image.

[0116] S54: Traverse all actual coordinates, and code all crystal vibrations falling in the region;

[0117] Loop through all the prediction results, and as long as the center point falls in a certain region, it is considered as the prediction result of the current detection region.

[0118] S55: According to the labeled information and the coded detection crystal vibration into the slot;

[0119] According to the coding, adjust the NG_1 category into the slot; the NG_2 category is thrown into the defective box; after executing all the taking and placing operations, take a photo for re-inspection; if there is no result returned or the detection is timed out, sound and light alarm; after the second re-inspection without NG category, move to the next work station under the tray to take a photo and continue detection.

[0120] As shown in Figure 8 , a quartz crystal vibration precision slot detection system based on deep learning is used to execute a quartz crystal vibration precision slot detection method based on deep learning, which comprises:

[0121] The image acquisition module 101 collects crystal vibration images of different detection stations, and pre-processes the crystal vibration images to obtain input images;

[0122] The model construction module 102 constructs a deep learning network model, which comprises a Backbone network, a Neck network and a Head network;

[0123] The feature extraction module 103 extracts features of the input image through the Backbone network to obtain a high-dimensional feature map; performs deep and shallow feature fusion and further feature extraction on the high-dimensional feature map through the Neck network to obtain a predicted input feature map; and extracts features of the predicted input feature map through the Head network to obtain a predicted feature map;

[0124] The optimization module 104 is configured to decode the predicted feature map through a prediction box to generate a prediction box; and remove redundant prediction boxes through NMS non-maximum suppression to obtain an optimal prediction box.

[0125] The detection module 105 detects whether the crystal oscillator is inserted into the slot according to the optimal prediction box.

[0126] Through the cooperative work of the above modules, the present application extracts and classifies features of the crystal oscillator image through a deep learning algorithm, ensures that crystal oscillators in various states under different lines, different fields of view, different sizes, different positions and distances can be accurately detected, significantly improves the accuracy of the insertion detection, reduces the false detection rate and the missed detection rate, helps to reduce the production cost, and improves the market competitiveness of enterprises.

[0127] The present application realizes real-time detection of whether multiple crystal oscillators are accurately inserted into the slot under multiple fields of view, triggers the corresponding feedback mechanism according to the detection result, realizes automatic control and adjustment, reduces manual intervention and downtime, reduces production cost, and improves the automation level and production efficiency of the production line.

[0128] The present application can adapt to crystal oscillators of different models, different sizes, different field of view ranges and front and back surfaces in various production environments, has strong robustness, generalization and scalability, and can adapt to new situations and new needs of the production environment through continuous updating and optimization of the model, ensuring long-term stable operation.

[0129] The present application can be easily embedded into the vision and control system of the crystal oscillator manufacturing equipment, realizing seamless connection with the existing production line, and at the same time, providing rich interfaces and document support to facilitate secondary development and integration of users.

[0130] The implementation of the present application brings an intelligent upgrade opportunity to the crystal oscillator manufacturing process, promotes the further development and innovation of electronic component manufacturing technology; through the combination with big data, cloud computing and other technologies, higher level intelligent management and decision support can be realized, which helps enterprises to maintain technology leadership and cope with increasingly fierce market competition.

[0131] Automatic detection and control reduce manual operation and reduce safety risks caused by human factors, which helps to protect the personal safety of employees and the production safety of enterprises.

[0132] The successful application of the present application will promote the industrial upgrading and technological progress of the electronic component manufacturing industry, and promote the development and improvement of related industrial chains, which will help to improve the competitiveness and sustainable development ability of the entire industry. By reducing manual intervention and downtime, it helps to reduce energy consumption and reduce waste emissions, in line with the current global environmental protection and sustainable development trends.

[0133] In addition to the crystal oscillator manufacturing field, the present application can be applied to other fields that require high-precision automated detection and control, such as precision instrument processing, semiconductor manufacturing, electronic product assembly, product appearance defect detection, etc., and has a wide application prospect.

[0134] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning-based precision slotting detection method for a quartz crystal oscillator, characterized in that, The method comprises the following steps: S1: Collecting crystal oscillator images of different detection stations, and preprocessing the crystal oscillator images to obtain input images; S2: Constructing a deep learning network model, wherein the deep learning network model comprises a Backbone network, a Neck network, and a Head network; The Backbone network comprises a CBS module, a CSSPX module comprising one RES Block, an RES-SE module, a CSSPX module comprising two RES Blocks, an RES-SE module, a CSSPX module comprising four RES Blocks, a CSSPX module comprising six RES Blocks, and a CSSPX module comprising eight RES Blocks, which are stacked in sequence; The working process of the RES-SE module is as follows: The output feature map of the CBS module with a convolution kernel of 3*3 is subjected to the EDC-SE module to obtain a weight vector, the weight vector is weighted to the output feature map of the original CBS module with a convolution kernel of 3*3 through a multiplication operation, and then the feature obtained through the CBS module with a convolution kernel of 3*3 is added to the input feature of the original RES-SE module to obtain the output feature of the RES-SE module; The working process of the EDC-SE module is as follows: The input feature of the EDC-SE module is divided into two paths, one of which is subjected to a global average pooling layer and a CBS module with a convolution kernel of 1*1 to obtain a fourth feature, and the other of which is subjected to a global maximum pooling layer and a CBS module with a convolution kernel of 1*1 to obtain a fifth feature, the fourth feature and the fifth feature are spliced, and then connected with an activation function to obtain the output feature of the EDC-SE module; S3: Extracting high-dimensional features from the input images through the Backbone network, performing deep-shallow feature fusion and further feature extraction on the high-dimensional features through the Neck network to obtain a predicted input feature map, and extracting features from the predicted input feature map through the Head network to obtain a predicted feature map; S4: Generating a prediction box from the predicted feature map through the Decode decoding of the prediction box, and removing redundant prediction boxes through NMS non-maximum suppression to obtain an optimal prediction box; The calculation expression is as follows: wherein, is a confidence score of the th predicted box, is a predicted box with the highest confidence score among all predicted boxes, is a confidence score of the th predicted box, is a ratio of intersection over union of the two predicted boxes, is a manually set threshold, and σ is a coefficient of continuous penalty term. S5: Detecting the crystal oscillator into the slot according to the optimal prediction box.

2. The quartz crystal precise slotting detection method based on deep learning according to claim 1, characterized in that, The preprocessing comprises geometric transformation, optical transformation, noise addition, and annotation information, wherein the annotation information comprises target categories, target upper-left corner coordinates, and target lower-right corner coordinates, and the target categories comprise OK_1, OK_2, NG_1, and NG_2.

3. The quartz crystal oscillator precision slot detection method based on deep learning according to claim 1, wherein The CBS module comprises a convolution layer, a BN layer, and a SiLu activation layer which are stacked in sequence; The RES Block comprises a CBS module with a convolution kernel of 1*1, a CBS module with a convolution kernel of 3*3, and a CBS module with a convolution kernel of 1*1 which are stacked in sequence. The working process of the CSSPX module is as follows: The input image of the CSSPX module is down-sampled by the CBS module to obtain a first feature map, the first feature map is extracted by a CBS module and a plurality of ES Block modules, and then extracted by another CBS module to obtain a second feature map, the first feature map is extracted by a CBS module to obtain a third feature map, the second feature map and the third feature map are spliced, and then extracted by a CBS module to obtain a CSSPX module output feature image.

4. The quartz crystal precise slotting detection method based on deep learning according to claim 3, characterized in that, The RES-SE module is an EDC-SE module added in the RES Block.

5. The quartz crystal precise slotting detection method based on deep learning according to claim 4, characterized in that, The Neck network uses a deep-shallow layer feature fusion network structure based on FPN and PAN, and increases SPPF modules and CSSPX modules in the front part, The SPPF module is an improved spatial pyramid pooling layer, The working process of the SPPF module is as follows: The input feature map of the SPPF module is first extracted by a CBS module with a convolution kernel of 1x1, and then down-sampled by three maximum pooling layers with different pooling kernel sizes, the obtained four feature maps are spliced by channel, and the spliced feature map is extracted by a CBS module with a convolution kernel of 1x1 to obtain an output feature map of the SPPF module.

6. The quartz crystal precise slotting detection method based on deep learning according to claim 2, characterized in that, The prediction input feature map is output by the Head network to obtain three size prediction feature maps.

7. The quartz crystal precise slotting detection method based on deep learning according to claim 6, characterized in that, S5 step includes: S51: converting the pixel coordinates of the three size optimal prediction boxes into actual coordinates in the camera coordinate system; S52: dividing the region with the center point of the detection region as the center according to the crystal oscillator image in the current detection field of view; S53: sorting according to the actual coordinates, and encoding the crystal oscillator falling in the region; S54: traversing all actual coordinates, and encoding all crystal oscillators falling in the region; S55: detecting the crystal oscillator into the slot according to the label information and the encoding.

8. The quartz crystal precise slotting detection method based on deep learning according to claim 7, characterized in that, In the S53 step, wherein, is the crystal oscillator code, is the number of regions divided for the current image, is is the left lower corner coordinate of the region, is is the left lower corner coordinate of the region, is the coordinate of any pixel point on the source image. 9.A deep learning-based precision slotting detection system for a quartz crystal oscillator, characterized in that, A quartz crystal oscillator precision slot detection method based on deep learning is executed, and the method comprises the following steps: An image acquisition module is arranged to acquire crystal oscillator images of different detection stations, and to pre-process the crystal oscillator images to obtain input images; A model construction module is arranged to construct a deep learning network model, wherein the deep learning network model comprises a Backbone network, a Neck network and a Head network; A feature extraction module is arranged to extract features of the input images by the Backbone network to obtain high-dimensional feature maps, to perform deep-shallow layer feature fusion and further feature extraction on the high-dimensional feature maps by the Neck network to obtain prediction input feature maps, and to extract features of the prediction input feature maps by the Head network to obtain prediction feature maps; An optimization module is arranged to generate prediction boxes by decoding the prediction feature maps by a Decode, to remove redundant prediction boxes by NMS to obtain optimal prediction boxes; A detection module is arranged to detect the crystal oscillator into the slot according to the optimal prediction boxes.

Images