Yolo-based target detection model training method and its application and device

Abstract

The present invention relates to a Yolo-based target detection model training method and its application and device. The method includes the following steps: (a) loading a training set and a test set, performing regional elastic deformation data enhancement on the training set, and setting corresponding training parameters; (b) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model; (c) training the neural network model to obtain a trained target detection model; the application is: after obtaining a sample set to be tested, inputting it into the trained target detection model, which outputs a Yolo-format prediction annotation of the sample set to be tested; the device includes a data set annotation unit, a data set segmentation and preprocessing unit, a parameter tuning unit, a neural network architecture search unit, and a training unit. The method of the present invention simplifies the operation and realizes the standardization of the entire detection model training process; the device of the present invention has a simple structure and is easy to operate.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a Yolo-based target detection model training method and its application and device. Background Art

[0002] Chips are the brains of today's society, powering mobile phones, smart wearable devices, large servers, and sensors. Chip manufacturing is a core technology for aerospace and national defense, the foundation of intelligent manufacturing, and a key enabler of informatization. Appearance defects can severely impact chip performance, making chip appearance defect detection a crucial component of chip manufacturing. Chip appearance defects are irregular in shape, diverse in features, and appear in non-stationary locations, often accompanied by high background noise. The target to be detected is small relative to the background. Traditional chip defect detection typically relies on manual visual inspection, which has low efficiency and reliability, and significantly increases production costs.

[0003] In recent years, deep learning-based artificial intelligence (AI) technology has matured. Object detection is one of the hottest research areas in computer vision, with crucial applications in real-world scenarios such as intelligent surveillance, autonomous driving, and face detection. Currently, deep neural network-based object detection models, with their high recognition accuracy and rapid speed, have become the mainstream of object detection algorithms. Therefore, applying deep neural network-based object detection models to chip defect detection is crucial for improving chip production yield and reducing production costs. Traditional automated chip defect detection tasks are typically framed as an object detection problem, emphasizing the importance of more samples, but this approach lacks sufficient consideration of the problem itself and the actual application environment. Furthermore, deep learning technology presents a high barrier to entry. Practitioners require a solid foundation in programming, mathematical foundations, intelligent algorithms, and a thorough understanding of the dataset to design and optimize appropriate neural network models.

[0004] After thorough research across enterprises and production lines, users have shown a preference for deep neural network models that are quick to deploy and easy to upgrade. The less impact samples, sample annotation, and model deployment have on the entire production line, the better. When the detection target changes, users can quickly adjust and deploy the model without lengthy re-tuning. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a target detection model training method based on Yolo and its application and device.

[0006] To achieve the above object, the present invention adopts the following scheme:

[0007] A Yolo-based target detection model training method includes the following steps:

[0008] (a) Load the training set and test set and set the corresponding training parameters;

[0009] The training set includes chip appearance defect images and object detection labels in Yolo format for the chip appearance defect images;

[0010] The training parameters include the predefined number of chip appearance defect categories, the maximum number of training rounds, the learning rate, the image input width, the image input height, and the filters. The predefined number of chip appearance defect categories is the number of types of chip appearance defects, the predefined number of chip appearance defect categories is set to n ≥ 1, the maximum number of training rounds is set to be greater than or equal to 2000 × n, the learning rate is set to 0.00111, the image input width is set to 512, the image input height is set to 512, and the filters is set to (n+5) × 3;

[0011] (b) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model;

[0012] The training set in step (a) is divided into training data set D train and validation dataset D val , construct a meta-structure search space, and model the method of differentiable neural network architecture search, using the training dataset D train Train a differentiable neural network architecture search model;

[0013] During the training process, the structural weights of the meta-structure are firstly normalized globally, and then the structural weights of the meta-structure and the network parameters are double-layer optimized. val The loss value is used as the objective function of the optimization process, and the network parameters and structural weights of the meta-structure are adjusted simultaneously through the back propagation algorithm;

[0014] After the training is completed, the meta-structures are sorted according to their structural weights, the meta-structures with the largest weights are retained, and a deep neural network model is formed to obtain the neural network model;

[0015] (c) training the neural network model to obtain a trained target detection model;

[0016] After data enhancement is performed on the training set described in step (a), the model parameters of the neural network model are optimized according to the data-enhanced training set, test set and the training parameters to obtain a trained target detection model.

[0017] As the preferred technical solution:

[0018] In the above-mentioned Yolo-based object detection model training method, the specific process of step (c) is as follows:

[0019] (c1) After performing data augmentation on the training set described in step (a), the chip appearance defect images in the training set are input into the neural network model to obtain target detection label prediction values;

[0020] (c2) Calculate the loss function value using the target detection label true value and the target detection label predicted value;

[0021] (c3) Using the loss function value to update the model parameters (parameters are divided into hyperparameters and model parameters. Hyperparameters are set by humans, while model parameters are optimized by algorithms. The training parameters mentioned above are hyperparameters);

[0022] (c4) Input the chip appearance defect images in the test set into the neural network model to obtain the target detection label prediction value;

[0023] (c5) Calculate the loss function value and the test set accuracy using the target detection label true value and the target detection label predicted value;

[0024] (c6) Determine whether the test set accuracy is greater than the maximum accuracy R (range 0 to 100%). If so, save the neural network model, update R, and proceed to the next step; otherwise, proceed directly to the next step;

[0025] (c7) Determine whether the neural network model has converged (by observing whether the loss function values ​​of the training set and the test set gradually decrease to determine whether the model has converged). If so, proceed to the next step; otherwise, reduce the learning rate (the specific adjustment value is determined based on experience, for example, reduce it by 10 times, that is, adjust it to one tenth of the previous learning rate), and return to step (c1);

[0026] (c8) Determine whether the maximum number of training rounds has been reached. If so, end the process and output the trained target detection model; otherwise, return to step (c1).

[0027] As described above, a target detection model training method based on Yolo, the training data set D train and validation dataset D val The ratio of the number of data is 9:1.

[0028] In the above-mentioned Yolo-based target detection model training method, if the training set contains small targets, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, instance-level random copy and paste, regional elastic deformation, flipping, random scaling, and random brightness contrast transformation in sequence; otherwise, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, regional elastic deformation, flipping, and random brightness contrast transformation in sequence;

[0029] The small object is an appearance defect whose ratio of the width and height of the bounding box to the width and height of the image is less than 0.1, or an appearance defect whose resolution is less than 32 pixels×32 pixels.

[0030] In the above-mentioned object detection model training method based on Yolo, the specific steps of the region elastic deformation are as follows:

[0031] (i) Set the area ratio of the rectangular frame to the image to (r min , r max ), the aspect ratio range of the rectangular frame is (a min , a max );

[0032] (ii) Randomly select a coordinate point (x, y) in the image and min , r max ) randomly select the area ratio r within the range i , in (a min , a max ) randomly select the aspect ratio a within the range i ;

[0033] (iii) According to r i 、a i And the image area, calculate the length and width of the rectangular frame, take (x, y) as the center point of the rectangular frame, and determine the rectangular frame;

[0034] (iv) performing elastic deformation on the image within the rectangular frame;

[0035] (v) According to the target detection label, an image region containing the target is obtained, and steps (i) to (iv) are repeated for the image region.

[0036] In the above-mentioned object detection model training method based on Yolo, the steps for obtaining the training set and the test set are as follows:

[0037] (i) Label the chip appearance defect images to obtain a labeled dataset;

[0038] Predefine chip appearance defect categories and obtain predefined chip appearance defect category configurations; call the graphic image annotation tool labelImg to perform rectangular box annotation to obtain Yolo format target detection labels for the chip appearance defect images, and finally obtain a chip appearance defect dataset consisting of the chip appearance defect images and the target detection labels, i.e., obtain a labeled dataset;

[0039] (ii) preprocessing the labeled data set to obtain a training set and a test set;

[0040] Determine whether the chip appearance defect dataset contains the small target. If so, back up the chip appearance defect dataset, divide the chip appearance defect image into 64 sub-images, and divide the target detection label into sub-labels according to the rules corresponding to the chip appearance defect image. Extract the sub-images and sub-labels to obtain the segmented chip appearance defect dataset, and split them, that is, obtain the training set and the test set, and back them up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function; otherwise, directly extract the chip appearance defect images and target detection labels in the chip appearance defect dataset, and split them, that is, obtain the training set and the test set, and back them up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function.

[0041] In the Yolo-based target detection model training method, the segmentation adopts a random sampling method, and the ratio of the number of data in the training set to the test set is 9:1.

[0042] In the above-mentioned Yolo-based target detection model training method, the segmentation process is accelerated using a multi-threading method.

[0043] The present invention also provides an application of a Yolo-based target detection model training method, which obtains a sample set to be tested and inputs it into the trained target detection model, which outputs a Yolo-format prediction annotation of the sample set to be tested;

[0044] The process of acquiring the sample set to be tested is as follows: collecting an appearance defect picture of the chip to be tested, determining whether the appearance defect picture of the chip to be tested contains the small target; if so, backing up the appearance defect picture of the chip to be tested, dividing the appearance defect picture of the chip to be tested into 64 sub-pictures, extracting the sub-pictures, obtaining the sample set to be tested, and backing it up at the same time; otherwise, directly extracting the appearance defect picture of the chip to be tested, obtaining the sample set to be tested, and backing it up at the same time.

[0045] The present invention also provides a device using the above-mentioned Yolo-based target detection model training method, comprising:

[0046] A data set annotation unit is used to annotate images of chip appearance defects and obtain an annotated data set;

[0047] A data set segmentation and preprocessing unit, used to preprocess the labeled data set to obtain a training set and a test set;

[0048] Parameter tuning unit, used to load training sets and test sets and set corresponding training parameters;

[0049] A neural network architecture search unit, configured to construct a meta-structure search space based on the training set, perform a neural network architecture search, and obtain a neural network model;

[0050] The training unit is used to train the neural network model to obtain a trained target detection model.

[0051] As the preferred technical solution:

[0052] The apparatus as described above further includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The process of the computer program is as follows:

[0053] (S1) annotating chip appearance defect images to obtain an annotated dataset;

[0054] (S2) preprocessing the labeled data set to obtain a training set and a test set;

[0055] (S3) loading the training set and test set, and setting the corresponding training parameters;

[0056] (S4) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model;

[0057] (S5) Training the neural network model to obtain a trained target detection model.

[0058] The method of the present invention has the following four characteristics:

[0059] (1) Previous appearance inspection methods all went from training to inference, while the method of the present invention integrates data labeling, preprocessing, model optimization, and training, integrating the entire process of appearance inspection and unifying many algorithms and steps under a single framework. These algorithms were originally unrelated and required manual execution of each step and intermediate conversion operations. However, the present invention overcomes the incompatibility between algorithms and connects the various algorithms. For users, the present invention greatly simplifies the cumbersome operation process, reduces the difficulty of applying these algorithms, improves the efficiency of the model from training to deployment, and is more adaptable to the complex and changing scenarios of the production site.

[0060] (2) To address the problem of small and irregular chip appearance defects, the present invention designs a small target mode. That is, when a chip appearance defect image contains a small target, the image is segmented into 64 sub-images, the labels are segmented according to the rules corresponding to the image, and then the sub-images and sub-labels are extracted and divided into training and test sets, and then corresponding data enhancement is performed respectively; the small target mode allows the model to focus more on the chip appearance defects rather than the large amount of background, thereby improving the detection accuracy;

[0061] (3) In the preprocessing stage of the labeled dataset, the segmentation process is accelerated by a multi-threaded method. After the segmentation is completed, the copy function is rewritten in C language during the backup process and called by the preprocessing module in the form of a DLL library function to solve the problem of the slow execution speed of the Python copy function, thereby doubling the preprocessing speed of the system. When the operating environment is CPU: I7-10700, hard disk: NVME KIOXIA 256G, 223 chip appearance defect images containing small targets (image size is 6464x4852) are preprocessed, and the preprocessing time is reduced from 6 hours to less than 20 minutes.

[0062] (4) The neural network architecture search is used to obtain the optimized network structure, which realizes the automatic optimization of the target detection model, reduces the dependence on expert knowledge, and greatly reduces the model's dependence on the data set.

[0063] Regarding feature (1), prior art 1 (CN202110944135) mainly solves the technical problem that the Yolo network cannot converge when detecting the same target data set with different category feature richness, and there is no labeling and preprocessing method, while the content of the present invention is aimed at the chip appearance defect detection task, and has made adjustments, such as the small target mode, and integrates labeling and preprocessing methods; prior art 2 (CN202110905324) mainly proposes a sorting system with a simple built-in algorithm, while the focus of the present invention is from data labeling to model training, and does not involve mechanical structure; prior art 3 (CN113410154A) uses the method of regional division to determine whether the chip is qualified. This method is completely different from the method used in the present invention. It can only classify but cannot accurately identify the unqualified chip area, which is essentially different from the method used in the present invention. Difference; Existing technology 4 (IC chip appearance inspection system based on machine vision, South China University of Technology.) mainly increases the positioning accuracy of chip pins through the light source setting of image acquisition and image processing methods, and detects pin defects and printed information clarity defects of SOP type chips, while the present invention does not involve mechanical facilities such as image acquisition. The present invention can detect various types of appearance defects and is a universal detection model training framework. This is essentially different; Existing technology 5 (QFP chip appearance visual inspection system and inspection method. China Mechanical Engineering 24 (3): 290-294, 301.) mainly uses the Canny operator edge detection algorithm to process the image of QFP pins, focusing on the pin stack height detection method and the pin coplanarity detection method based on the three-point method. Similar to the previous invention, it uses traditional image processing methods to detect pins.

[0064] Regarding feature (2), prior art 6 (CN202110880257) mainly realizes small chip defect detection through transfer learning, while the present invention improves the detection accuracy of small targets by segmenting high-resolution images, and at the same time searches for a model suitable for the data set through a neural network architecture search method; prior art 7 (Carrier chip defect detection based on lightweight convolutional neural network. Computer Engineering and Applications: 1-10.), for the real-time detection problem of three different types of defects, namely carrier chip cracks, positioning column damage, and waveguide stains, proposes a carrier chip defect detection algorithm YOLO-Efficientnet based on a lightweight convolutional neural network. However, this method is for three defects of carrier chips, while the method of the present invention is applicable to various chip defect detections, especially for small target appearance defects. It includes various processes from data labeling to preprocessing, and includes a complete system, which is not available in other inventions.

[0065] Regarding feature (3), there is no relevant literature mentioning the application of this type of acceleration method in chip appearance defect detection, because this is related to the small target mode of the present invention and is unique to the present invention.

[0066] Regarding feature (4), prior art 8 (CN202110642625) only detects chip solder balls and recognizes text at the same time. The detection content is single and lacks adaptability. The method of the present invention can rebuild the model at any time according to the changes in the training data set, and search for the optimal network structure based on the neural network architecture search method, which is very convenient and efficient; prior art 9 (Research on IC chip appearance defect recognition algorithm based on deep learning, Jiangnan University.) mainly studies the traditional convolutional neural network algorithm, which is different from the full-process system method proposed in the present invention, and is essentially different from Yolo and neural network architecture search.

[0067] In summary, a comparison with the aforementioned existing technologies reveals that, among the numerous methods employing deep learning technology, only the present invention integrates data labeling, preprocessing, parameter setting, neural network architecture search, and model training into a single system, forming a holistic approach to chip appearance defect detection. This approach is systematic, integrated, automated, and takes a holistic approach, while simultaneously optimizing various components, such as small target modes and preprocessing acceleration methods. Other methods, on the other hand, focus only on a single part of the detection system and cannot be directly implemented on production lines.

[0068] Compared to existing technologies, other approaches only optimize specific parts, while this invention takes a holistic approach, simplifying the operational process for better practical application. While existing methods have limitations, this invention meticulously designs methods such as preprocessing acceleration and small target mode, while also utilizing the latest neural network architecture search algorithms to optimize the underlying model. This invention cleverly integrates these processes and algorithms into a novel whole, rather than simply a combination.

[0069] Beneficial effects:

[0070] (1) The object detection model training method based on Yolo of the present invention simplifies the operation and standardizes the entire detection model training process, making the entire process highly automated, improving efficiency, and reducing dependence on expert knowledge;

[0071] (2) The object detection model training method based on Yolo of the present invention can enable front-line workers to independently and standardizedly complete data collection, labeling, and preprocessing according to the actual situation of the samples obtained; through a smaller data set and standardized data labeling and neural network architecture search module, a deep neural network model suitable for the current data set is obtained, and the model training and deployment are completed;

[0072] (3) The target detection model training device based on Yolo of the present invention has a simple structure and is easy to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flow chart of a Yolo-based target detection model training method of the present invention;

[0074] Figure 2 This is a schematic diagram of the partial structure of a Yolo-based target detection model training device of the present invention;

[0075] Figure 3 A flow chart of training a neural network model to obtain a trained target detection model in a Yolo-based target detection model training method of the present invention;

[0076] Figure 4 The specific steps of the regional elastic deformation algorithm of the present invention;

[0077] Figure 5 Schematic diagram of the deformation principle of regional elastic deformation of the present invention, wherein (a) is the original image and (b) is the image after deformation;

[0078] Figure 6 These are actual effect diagrams of the regional elastic deformation of the present invention, where (a) is actual effect 1, (b) is actual effect 2, (c) is actual effect 3, (d) is actual effect 4, (e) is actual effect 5, (f) is actual effect 6, (g) is actual effect 7, (h) is actual effect 8, and (i) is actual effect 9. DETAILED DESCRIPTION

[0079] Below in conjunction with specific embodiment, further elaborate the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the present invention.

[0080] A Yolo-based target detection model training method, such as Figure 1 The specific steps are as follows:

[0081] (a) Obtain training and test sets. The specific steps are as follows:

[0082] (i) Label the chip appearance defect images to obtain a labeled dataset;

[0083] Predefine chip appearance defect categories and obtain predefined chip appearance defect category configurations; call the graphic image annotation tool labelImg to perform rectangular box annotation to obtain Yolo format target detection labels for the chip appearance defect images, and finally obtain a chip appearance defect dataset consisting of the chip appearance defect images and the target detection labels, i.e., obtain a labeled dataset;

[0084] (ii) preprocessing the labeled data set to obtain a training set and a test set;

[0085] Determine whether the chip appearance defect dataset contains the small target. If so, back up the chip appearance defect dataset, divide the chip appearance defect image into 64 sub-images using a multi-threaded method, and simultaneously divide the target detection label into sub-labels according to the rules corresponding to the chip appearance defect image using a multi-threaded method, extract the sub-images and sub-labels, obtain the segmented chip appearance defect dataset, and divide it by a random sampling method, that is, obtain a training set and a test set with a data ratio of 9:1, and back it up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function; otherwise, directly extract the chip appearance defect image and the target detection label in the chip appearance defect dataset, and divide it by a random sampling method, that is, obtain a training set and a test set with a data ratio of 9:1, and back it up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function;

[0086] The small object is an appearance defect with a ratio of the width and height of the bounding box to the width and height of the image less than 0.1, or an appearance defect with a resolution less than 32 pixels × 32 pixels;

[0087] The training set includes chip appearance defect images and object detection labels in Yolo format for the chip appearance defect images;

[0088] (b) Load the training set and test set and set the corresponding training parameters;

[0089] The training parameters include the predefined number of chip appearance defect categories, the maximum number of training rounds, the learning rate, the image input width, the image input height, and the filters. The predefined number of chip appearance defect categories is the number of types of chip appearance defects, the predefined number of chip appearance defect categories is set to n ≥ 1, the maximum number of training rounds is set to be greater than or equal to 2000 × n, the learning rate is set to 0.00111, the image input width is set to 512, the image input height is set to 512, and the filters is set to (n+5) × 3;

[0090] (c) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model;

[0091] The training set in step (a) is divided into a training data set D with a data ratio of 9:1. train and validation dataset D val , construct a meta-structure search space, and model the method of differentiable neural network architecture search, using the training dataset D train Train a differentiable neural network architecture search model;

[0092] During the training process, the structural weights of the meta-structure are firstly normalized globally, and then the structural weights of the meta-structure and the network parameters are double-layer optimized. val The loss value is used as the objective function of the optimization process, and the network parameters and structural weights of the meta-structure are adjusted simultaneously through the back propagation algorithm;

[0093] After the training is completed, the meta-structures are sorted according to their structural weights, the meta-structures with the largest weights are retained, and a deep neural network model is formed to obtain the neural network model;

[0094] (d) training the neural network model to obtain a trained target detection model, such as Figure 3 The specific steps are as follows:

[0095] (d1) After data enhancement is performed on the training set described in step (a), the chip appearance defect image in the training set (i.e., the "training image" in the figure) is input into the neural network model to obtain the target detection label prediction value (i.e., the "prediction result" in the figure); wherein, if the training set contains small targets, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, instance-level random copy and paste, regional elastic deformation, flipping, random scaling and brightness contrast random transformation in sequence; otherwise, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, regional elastic deformation, flipping and brightness contrast random transformation in sequence;

[0096] (d2) Calculate the loss function value using the true value of the target detection label (i.e., the "training image label" in the figure) and the predicted value of the target detection label (i.e., the "prediction result" in the figure);

[0097] (d3) Using the loss function value to update the model parameters (parameters are divided into hyperparameters and model parameters. Hyperparameters are set by humans, while model parameters are optimized by algorithms. The training parameters mentioned above are hyperparameters);

[0098] (d4) Input the chip appearance defect images in the test set (i.e., the "test images" in the figure) into the neural network model to obtain the target detection label prediction value (i.e., the "prediction result" in the figure);

[0099] (d5) Calculate the loss function value and test set accuracy using the true value of the target detection label (i.e., the "test image label" in the figure) and the predicted value of the target detection label (i.e., the "prediction result" in the figure);

[0100] (d6) Determine whether the test set accuracy is greater than the maximum accuracy R (the initial value of R is 0%). If so, save the neural network model, update R, and proceed to the next step; otherwise, proceed directly to the next step;

[0101] (d7) Determine whether the neural network model has converged. If so, proceed to the next step. Otherwise, reduce the learning rate (the specific adjustment value is determined based on experience, for example, reduce it by 10 times, that is, adjust it to one tenth of the previous learning rate), and return to step (d1);

[0102] (d8) Determine whether the maximum number of training rounds has been reached. If so, end the process and output the trained target detection model. Otherwise, return to step (d1).

[0103] In the step (d1) mentioned above, one of the steps in data enhancement is regional elastic deformation, such as Figure 4 The specific steps are as follows:

[0104] (i) Set the area ratio of the rectangular frame to the image to (r min , r max ), the aspect ratio range of the rectangular frame is (a min , a max );

[0105] (ii) Randomly select a coordinate point (x, y) in the image and min , r max ) randomly select the area ratio r within the range i , in (a min , a max ) randomly select the aspect ratio a within the range i ;

[0106] (iii) According to r i 、a i And the image area, calculate the length and width of the rectangular frame, take (x, y) as the center point of the rectangular frame, and determine the rectangular frame;

[0107] (iv) performing elastic deformation on the image within the rectangular frame;

[0108] (v) According to the target detection label, an image region containing the target is obtained, and steps (i) to (iv) are repeated for the image region.

[0109] The mathematical principle of elastic deformation is based on bilinear interpolation, such as Figure 5 shown.

[0110] An application of the Yolo-based target detection model training method as described above: after obtaining the sample set to be tested, input it into the trained target detection model, which outputs the Yolo format prediction annotation of the sample set to be tested; wherein, the process of obtaining the sample set to be tested is: collecting the appearance defect picture of the chip to be tested, judging whether the appearance defect picture of the chip to be tested contains the small target, if so, backing up the appearance defect picture of the chip to be tested, dividing the appearance defect picture of the chip to be tested into 64 sub-pictures, extracting the sub-pictures, obtaining the sample set to be tested, and backing it up at the same time; otherwise, directly extracting the appearance defect picture of the chip to be tested, obtaining the sample set to be tested, and backing it up at the same time.

[0111] The device using the Yolo-based target detection model training method as described above includes a data set annotation unit, a data set segmentation and preprocessing unit, a parameter tuning unit, a training unit, a memory, a processor, and a computer program stored in the memory and executable on the processor; wherein the partial structural diagram of the device is as follows Figure 2 As shown;

[0112] A data set annotation unit is used to annotate images of chip appearance defects and obtain an annotated data set;

[0113] A data set segmentation and preprocessing unit, used to preprocess the labeled data set to obtain a training set and a test set;

[0114] Parameter tuning unit, used to load training sets and test sets and set corresponding training parameters;

[0115] A neural network architecture search unit, configured to construct a meta-structure search space based on the training set, perform a neural network architecture search, and obtain a neural network model;

[0116] A training unit, configured to train the neural network model to obtain a trained target detection model;

[0117] The flow of the computer program is as follows:

[0118] (S1) annotating chip appearance defect images to obtain an annotated dataset;

[0119] (S2) preprocessing the labeled data set to obtain a training set and a test set;

[0120] (S3) loading the training set and test set, and setting the corresponding training parameters;

[0121] (S4) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model;

[0122] (S5) Training the neural network model to obtain a trained target detection model.

[0123] The following example illustrates the training method and application of a Yolo-based target detection model. The task is to detect scratches on chip surfaces. The specific steps are as follows:

[0124] (a) Obtain training and test sets;

[0125] A chip appearance defect category file is predefined. The file content is "huashang", which is the category label of chip appearance defects. First, a batch of chip appearance defect images are collected and opened with the graphic image annotation tool labelImg to annotate them. The annotation content is scratch defects, and the annotation form is a rectangular box. When annotating, the rectangular box must just cover the scratch. After annotating all images containing scratches, the initial data set is obtained.

[0126] The original chip appearance defect image has a size of 6464×4852, and the size of the scratch defect is less than 600×480, which meets the conditions of a small target. Therefore, the image and label are segmented; the chip appearance defect image is segmented into 64 sub-images, and the scratch label is segmented into sub-labels according to the rules corresponding to the image. Then, the sub-image containing scratches and the corresponding sub-labels are extracted to obtain a small target chip appearance defect dataset, or chip appearance scratch dataset.

[0127] The small target chip appearance defect dataset is split into a training set and a test set in a ratio of 9:1.

[0128] (b) Load the training set and test set and set the corresponding training parameters;

[0129] Set the number of predefined chip appearance defect categories n to 1, the maximum training rounds to 2000, the learning rate to 0.00111, the image input width to 512, the image input height to 512, and the filter to 18;

[0130] (c) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model;

[0131] The training set obtained above is further divided into training data set D according to the ratio of 9:1 train and validation dataset D val, under the overall architecture of Yolo, a meta-structure search space is constructed, and a method for searching differentiable neural network architecture is modeled, using the training dataset D train Train a differentiable neural network architecture search model;

[0132] During the training process, the structural weights of the meta-structure are firstly normalized globally, and then the structural weights of the meta-structure and the network parameters are double-layer optimized. val The loss value is used as the objective function of the optimization process, and the network parameters and structural weights of the meta-structure are adjusted simultaneously through the back propagation algorithm;

[0133] After the training is completed, the meta-structures are sorted according to their structural weights, the meta-structures with the largest weights are retained, and a deep neural network model is formed to obtain the neural network model;

[0134] (d) Training the neural network model to obtain a trained target detection model, the specific steps are as follows:

[0135] (d1) First, the data enhancement methods of Cutmix, mosaic data enhancement, class label smoothing, instance-level random copy and paste, regional elastic deformation, flipping, random scaling, and random brightness contrast transformation are applied to the training set in sequence to obtain enhanced training set images and corresponding scratch labels; then, the data enhancement methods of flipping, random scaling, and random brightness contrast transformation are applied to the test set data to obtain enhanced test set images and corresponding scratch labels; finally, the enhanced training set chip appearance defect images are input into the neural network to obtain the scratch label prediction value;

[0136] (d2) Calculate the loss function value using the true value of the scratch label and the predicted value of the scratch label;

[0137] (d3) Update the model parameters using the loss function value;

[0138] (d4) Inputting the chip appearance scratch images in the test set into the neural network model to obtain the scratch prediction value of the test set;

[0139] (d5) Using the actual value of the scratch label in the test set and the predicted value of the scratch label in the test set, calculate the loss function value and the test set accuracy;

[0140] (d6) Determine whether the test set accuracy is greater than the maximum accuracy R (the initial value of R is 0%). If so, save the neural network model, update R, and proceed to the next step; otherwise, proceed directly to the next step;

[0141] (d7) Determine whether the neural network model has converged. If so, proceed to the next step. Otherwise, reduce the learning rate (the specific adjustment value is determined based on experience, for example, reduce it by 10 times, that is, adjust it to one tenth of the previous learning rate), and return to step (d1);

[0142] (d8) Determine whether the maximum number of training rounds, 2000, has been reached. If so, terminate the process and output the trained target detection model. Otherwise, return to step (d1).

[0143] (e) Output the label of the image of the appearance defect of the chip under test;

[0144] Collect the appearance defect images of the chip to be tested, divide them into 64 sub-images, and obtain the segmented chip appearance defect images; extract the segmented chip appearance defect images to obtain the sample set to be tested, and back them up at the same time;

[0145] Input the sample set to be tested into the trained neural network model to obtain the scratch prediction label of the sample set to be tested;

[0146] There are 100 pictures of appearance defects of the chip to be tested, of which 110 scratch defects exist, including 90 small target scratch defects; the test samples are input into the model of the present invention and the model of the prior art (faster-rcnn) to obtain detection results; the model of the present invention detects 100 pictures and detects 102 scratch defects, including 85 small target scratch defects, at a detection speed of 35 pictures per second; the model of the prior art (faster-rcnn) detects 93 scratch defects, including 79 small target scratch defects, at a detection speed of 5 pictures per second.

[0147] In addition to chip datasets, the present invention also has good results on other public datasets. On the PASCAL VOC 2007 target detection dataset, the AP50 index of the method of the present invention is improved by 0.5 percentage points compared with the basic model. In addition, the regional elastic deformation algorithm of the present invention can also be applied to image classification tasks, and the actual effect is as follows: Figure 6 On the CIFAR-10 dataset, the accuracy of the original ResNet18 model was 94.92%, and after using the region elastic deformation algorithm, the accuracy was 95.86%. On the CIFAR-100 dataset, the accuracy of the original ResNet50 model was 80.60%, and after using the region elastic deformation algorithm, the accuracy was 81.68%.

[0148] Advantages of the method of the present invention: Compared with the original Yolo network, the present invention inherits the efficient detection speed of the Yolo network model, not only has good accuracy for small targets, but also can adjust the neural network model structure according to the data set; compared with faster-rcnn, the method of the present invention has a faster detection speed and a simpler operation method; from the perspective of data enhancement, the present invention designs a regional elastic deformation data enhancement method, which simulates the local elastic deformation of objects in the real world, increases the richness of samples, and improves the robustness of the model; from the perspective of standardization, compared with other methods that only improve the model structure, the present invention targets a variable data set, connects the data set construction and model training processes, reduces the complexity of data preprocessing, and provides a more standard and complete model training method.

Claims

1. A Yolo-based target detection model training method, characterized in that: The following steps are involved: (a) Load the training set and test set and set the corresponding training parameters; The training set includes chip appearance defect images and object detection labels in Yolo format for the chip appearance defect images; The training parameters include the predefined number of chip appearance defect categories, the maximum number of training rounds, the learning rate, the image input width, the image input height, and the number of filters. The predefined number of chip appearance defect categories is set to n ≥ 1, the maximum number of training rounds is set to be greater than or equal to 2000 × n, the learning rate is set to 0.00111, the image input width is set to 512, the image input height is set to 512, and the filters are set to (n + 5) × 3. (b) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model; The training set in step (a) is divided into training data set D train and validation dataset D val , construct a meta-structure search space, and model the method of differentiable neural network architecture search, using the training dataset D train Train a differentiable neural network architecture search model; During the training process, the structural weights of the meta-structure are firstly normalized globally, and then the structural weights of the meta-structure and the network parameters are double-layer optimized. val The loss value is used as the objective function of the optimization process, and the network parameters and structural weights of the meta-structure are adjusted simultaneously through the back propagation algorithm; After the training is completed, the meta-structures are sorted according to their structural weights, the meta-structures with the largest weights are retained, and a deep neural network model is formed to obtain the neural network model; (c) training the neural network model to obtain a trained target detection model; After data enhancement is performed on the training set described in step (a), the model parameters of the neural network model are optimized according to the data-enhanced training set, test set and the training parameters to obtain a trained target detection model.

2. The object detection model training method based on Yolo according to claim 1, characterized in that The specific process of step (c) is as follows: (c1) After performing data augmentation on the training set described in step (a), the chip appearance defect images in the training set are input into the neural network model to obtain target detection label prediction values; (c2) Calculate the loss function value using the target detection label true value and the target detection label predicted value; (c3) Update the model parameters using the loss function value; (c4) Input the chip appearance defect images in the test set into the neural network model to obtain the target detection label prediction value; (c5) Calculate the loss function value and the test set accuracy using the target detection label true value and the target detection label predicted value; (c6) Determine whether the test set accuracy is greater than the maximum accuracy R. If so, save the neural network model, update R, and proceed to the next step; otherwise, proceed directly to the next step; (c7) Determine whether the neural network model has converged. If so, proceed to the next step; otherwise, reduce the learning rate and return to step (c1); (c8) Determine whether the maximum number of training rounds has been reached. If so, end the process and output the trained target detection model; otherwise, return to step (c1).

3. The object detection model training method based on Yolo according to claim 1, characterized in that Training dataset D train and validation dataset D val The ratio of the number of data is 9:

1.

4. The object detection model training method based on Yolo according to claim 1, characterized in that If the training set contains small objects, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, instance-level random copy and paste, region elastic deformation, flipping, random scaling, and brightness contrast random transformation in sequence; otherwise, the data enhancement process is to perform Cutmix, mosaic data enhancement, class label smoothing, region elastic deformation, flipping, and brightness contrast random transformation in sequence; The small object is an appearance defect whose ratio of the width and height of the bounding box to the width and height of the image is less than 0.1, or an appearance defect whose resolution is less than 32 pixels×32 pixels.

5. The object detection model training method based on Yolo according to claim 4 is characterized in that The specific steps of regional elastic deformation are as follows: (i) Set the area ratio of the rectangular frame to the image to (r min , r max ), the aspect ratio range of the rectangular frame is (a min , a max ); (ii) Randomly select a coordinate point (x, y) in the image and min , r max ) randomly select the area ratio r within the range i , in (a min , a max ) randomly select the aspect ratio a within the range i ; (iii) According to r i 、a i And the image area, calculate the length and width of the rectangular frame, take (x, y) as the center point of the rectangular frame, and determine the rectangular frame; (iv) performing elastic deformation on the image within the rectangular frame; (v) According to the target detection label, an image region containing the target is obtained, and steps (i) to (iv) are repeated for the image region.

6. The object detection model training method based on Yolo according to claim 4, characterized in that The steps for obtaining the training set and test set are as follows: (i) Label the chip appearance defect images to obtain a labeled dataset; Predefine chip appearance defect categories and obtain predefined chip appearance defect category configurations; call the graphic image annotation tool labelImg to perform rectangular box annotation to obtain Yolo format target detection labels for the chip appearance defect images, and finally obtain a chip appearance defect dataset consisting of the chip appearance defect images and the target detection labels, i.e., obtain a labeled dataset; (ii) preprocessing the labeled data set to obtain a training set and a test set; Determine whether the chip appearance defect dataset contains the small target. If so, back up the chip appearance defect dataset, divide the chip appearance defect image into 64 sub-images, and divide the target detection label into sub-labels according to the rules corresponding to the chip appearance defect image. Extract the sub-images and sub-labels to obtain the segmented chip appearance defect dataset, and split them, that is, obtain the training set and the test set, and back them up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function; otherwise, directly extract the chip appearance defect images and target detection labels in the chip appearance defect dataset, and split them, that is, obtain the training set and the test set, and back them up at the same time. During the backup process, the copy function is rewritten in C language and called by the preprocessing module in the form of a DLL library function.

7. The object detection model training method based on Yolo according to claim 6, characterized in that The segmentation adopts a random sampling method, and the ratio of the number of data in the training set and the test set is 9:1; the process of dividing the chip appearance defect image into 64 sub-images adopts a multi-threading method to speed up, and the target detection label is divided into sub-labels according to the rules corresponding to the chip appearance defect image. The multi-threading method is also used to speed up.

8. The application of the Yolo-based target detection model training method according to claim 6 or 7, wherein: After obtaining the sample set to be tested, it is input into the trained target detection model, which outputs the predicted annotation of the sample set to be tested in Yolo format; The process of acquiring the sample set to be tested is as follows: collecting an image of the appearance defect of the chip to be tested, determining whether the image of the appearance defect of the chip to be tested contains the small target, and if so, backing up the image of the appearance defect of the chip to be tested, dividing the image of the appearance defect of the chip to be tested into 64 sub-images, extracting the sub-images, obtaining the sample set to be tested, and backing up the images at the same time; Otherwise, the appearance defect picture of the chip to be tested is directly extracted to obtain a sample set to be tested, and a backup is performed at the same time.

9. A device using a Yolo-based target detection model training method as described in claim 6 or 7, characterized in that: include: A data set annotation unit is used to annotate images of chip appearance defects and obtain an annotated data set; A data set segmentation and preprocessing unit, used to preprocess the labeled data set to obtain a training set and a test set; Parameter tuning unit, used to load training sets and test sets and set corresponding training parameters; A neural network architecture search unit, configured to construct a meta-structure search space based on the training set, perform a neural network architecture search, and obtain a neural network model; The training unit is used to train the neural network model to obtain a trained target detection model.

10. The device according to claim 9, characterized in that The system also includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The process of the computer program is as follows: (S1) annotating chip appearance defect images to obtain an annotated dataset; (S2) preprocessing the labeled data set to obtain a training set and a test set; (S3) loading the training set and test set, and setting the corresponding training parameters; (S4) constructing a meta-structure search space based on the training set, performing a neural network architecture search, and obtaining a neural network model; (S5) Training the neural network model to obtain a trained target detection model.

Images