Scratch Detection Method, Device, Computer Equipment and Storage Medium

Abstract

This application relates to a scratch detection method, apparatus, computer device, and storage medium. The method includes: obtaining a chip image to be detected, and determining a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes; determining a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes; and obtaining a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model. Using this method can improve the accuracy of scratch detection for chips.

Description

Technical Field

[0001] The present application relates to the technical field of defect detection, and particularly to a scratch detection method, device, computer device, and storage medium. Background Art

[0002] With the development of electronic product technology, as the core component of electronic products, the quality inspection of chips is crucial.

[0003] In the prior art, deep learning algorithms can be used to detect scratches on chips. However, in the prior art, based on deep learning algorithms, a neural network model is trained using an image set of large target objects, and good scratch detection results can be obtained in the scenario of defect detection for large targets, but it is not applicable to small target objects such as chips, and the accuracy of the existing scratch detection methods for detecting chip scratches is relatively low. Summary of the Invention

[0004] Based on this, it is necessary to provide a scratch detection method, device, computer device, and storage medium that can improve the accuracy of chip scratch detection for the above technical problems.

[0005] In a first aspect, the present application provides a scratch detection method. The method includes:

[0006] Obtain a chip image to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0007] Determine a second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0008] Obtain a scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model.

[0009] In one embodiment, the improved VGG16 network includes cascaded: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, and a seventh convolution module; the first feature map set includes a first feature map, a second feature map, and a third feature map; the determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes:

[0010] Input the chip image into the improved VGG16 network, use the output item of the second convolutional module as the first feature map, use the output item of the third convolutional module as the second feature map, and use the output item of the seventh convolutional module as the third feature map.

[0011] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; determining the second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model includes:

[0012] Input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map.

[0013] In one embodiment, obtaining the scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model includes:

[0014] Input the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0015] Perform non-maximum suppression processing on the multiple detection feature maps to obtain the scratch detection result.

[0016] In one embodiment, before determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, it further includes:

[0017] Obtain a scratch training image set, the scratch training image set includes multiple scratch images, and the annotation image of each scratch image, and each scratch image is an image of a chip;

[0018] Train the scratch detection model based on the scratch training image set, and obtain the trained scratch detection model after the training ends.

[0019] In one embodiment, the obtaining of the scratch training image set includes:

[0020] Shoot the chip to obtain multiple original images, and use the images with scratches in the multiple original images as scratch images;

[0021] Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images;

[0022] Annotate each scratch image to obtain an annotated image corresponding to each scratch image.

[0023] In a second aspect, the present application also provides a scratch detection device. The device includes:

[0024] A first feature map set determination module, configured to obtain a chip image of a chip to be detected, and determine a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0025] A second feature map set determination module, configured to determine a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0026] A scratch detection result determination module, configured to obtain a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model.

[0027] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0028] Obtain a chip image of a chip to be detected, and determine a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0029] Determine a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0030] Obtain a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model.

[0031] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, the following steps are implemented:

[0032] Obtain a chip image of a chip to be detected, and determine a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0033] Determine a second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0034] Obtain a scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model.

[0035] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Obtain a chip image to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0037] Determine a second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0038] Obtain a scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model.

[0039] For the above-mentioned scratch detection method, device, computer device, storage medium, and computer program product, a first feature map set is obtained based on a chip image and an improved VGG16 network. The first feature map set includes three feature maps with different image sizes. A second feature map set is obtained based on the first feature map set and an additional feature extraction network. The second feature map set includes three feature maps with different image sizes. A scratch detection result is obtained based on the first feature map set, the second feature map set, and the detection network. Compared with the existing SSD model, the existing SSD model extracts six feature maps, where two feature maps are obtained through the improved VGG16 network and four feature maps are obtained through the additional feature extraction network; in the present application, the first feature map set and the second feature map set include six feature maps; the feature map with the largest image size among the six feature maps of the existing SSD model is smaller than the feature map with the largest image size in the present application, and the feature map with the smallest image size among the six feature maps is smaller than the feature map with the smallest image size in the present application. Generally speaking, the image features included in the six feature maps extracted by the existing SSD model are fewer than the image features included in the six feature maps in the present application, which is more suitable for scratch detection of small target images such as chip images and improves the accuracy of scratch detection for chip images. Description of the Drawings

[0040] Figure 1 It is a schematic flowchart of a scratch detection method in an embodiment;

[0041] Figure 2 It is a schematic diagram of a trained scratch detection model in an embodiment;

[0042] Figure 3 It is a schematic diagram showing the difference between real scratches and artificial scratches in an embodiment;

[0043] Figure 4 It is a structural block diagram of a scratch detection device in an embodiment;

[0044] Figure 5 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] In one embodiment, as Figure 1 shown, a scratch detection method is provided. In this embodiment, the method is exemplified by being applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0047] S102, obtain a chip image to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model.

[0048] Among them, the first feature map set includes three feature maps with different image sizes. The trained scratch detection model includes an improved VGG16 network. The structure of the trained scratch detection model is the same as that of the SingleShot Multi-Box Detector (SSD) model.

[0049] Specifically, a SCARA robot can be used to drive a camera device to photograph a chip to be detected to obtain a chip image.

[0050] The VGG16 network includes three cascaded fully connected layers. Replace the first two fully connected layers with convolutional layers, delete the last fully connected layer, and delete the dropout layers in the VGG16 network. The VGG16 network includes five max pooling layers. According to the cascading order of the VGG16 network, define the operation attribute of the third max pooling layer as rounding up, and modify the convolutional kernel size and stride of the last max pooling layer to obtain an improved VGG16 network.

[0051] Input the chip image into the improved VGG16 network, and obtain a first feature map set from three convolutional modules of the improved VGG16 network; the chip image is a small target image, and the first feature map set includes first feature maps with larger image sizes. The first feature maps include more features of the chip image, which can improve the accuracy of scratch detection.

[0052] S104, based on the first feature map set and the additional feature extraction network in the trained scratch detection model, determine a second feature map set.

[0053] Among them, the second feature map set includes three feature maps with different image sizes.

[0054] Specifically, input the feature map with the smallest size in the first feature map set into the additional feature extraction network, and obtain the second feature map set through some layers of the additional feature extraction network. The additional feature extraction network includes three feature extraction modules. Input the feature map with the smallest size in the first feature map set into the additional feature extraction network, and obtain three feature maps with different sizes through the three feature extraction modules.

[0055] S106, based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model, obtain a scratch detection result.

[0056] Specifically, input the first feature map set and the second feature map set into the detection network, detect the predicted bounding boxes of each feature map in the first feature map set and the second feature map set, and then determine the scratch detection result. The scratch detection result includes a detection image and a scratch detection flag. The scratch detection flag can reflect whether the chip image includes scratches or the chip image does not include scratches; if the scratch detection flag indicates that the chip image includes scratches, the detection image includes a detection box of the scratches, and the detection box is the bounding box of the scratches.

[0057] In the above scratch detection method, a first feature map set is obtained based on the chip image and the improved VGG16 network. The first feature map set includes three feature maps with different image sizes. A second feature map set is obtained based on the first feature map set and the additional feature extraction network. The second feature map set includes three feature maps with different image sizes. Based on the first feature map set, the second feature map set, and the detection network, a scratch detection result is obtained. Compared with the existing SSD model, the existing SSD model extracts six feature maps. Among them, two feature maps are obtained through the improved VGG16 network, and four feature maps are obtained through the additional feature extraction network. In this application, the first feature map set and the second feature map set include six feature maps. The feature map with the largest image size among the six feature maps of the existing SSD model is smaller than the feature map with the largest image size in this application, and the feature map with the smallest image size among the six feature maps is smaller than the feature map with the smallest image size in this application. Generally speaking, the image features included in the six feature maps extracted by the existing SSD model are fewer than the image features included in the six feature maps in this application, which is more suitable for scratch detection of small target images such as chip images and improves the accuracy of scratch detection for chip images.

[0058] In one embodiment, referring to Figure 2 , the improved VGG16 network includes cascaded: a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a sixth convolutional module, and a seventh convolutional module; the first feature map set includes a first feature map, a second feature map, and a third feature map. In S102, determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes:

[0059] S201, input the chip image into the improved VGG16 network, use the output item of the second convolutional module as the first feature map, use the output item of the third convolutional module as the second feature map, and use the output item of the seventh convolutional module as the third feature map.

[0060] Among them, the first convolutional module includes: the first convolutional layer conv1-1, the second convolutional layer con1-2, and the first max pooling layer maxpooling-1; the second convolutional module includes: the third convolutional layer conv2-1, the fourth convolutional layer conv2-2, and the second max pooling layer maxpooling-2; the third convolutional module includes: the fifth convolutional layer conv3-1, the sixth convolutional layer conv3-2, the seventh convolutional layer conv3-3, and the third max pooling layer maxpooling-3; the fourth convolutional module includes: the eighth convolutional layer conv4-1, the ninth convolutional layer conv4-2, the tenth convolutional layer conv4-3, and the fourth max pooling layer maxpooling-4; the fifth convolutional module includes: the eleventh convolutional layer conv5-1, the twelfth convolutional layer conv5-2, the thirteenth convolutional layer conv5-3, and the fourth max pooling layer maxpooling-4; the sixth convolutional module includes: the fourteenth convolutional layer conv6; the seventh convolutional module includes: the fifteenth convolutional layer conv7. Among them, it is set that ceil_mode of maxpooling-3 = True, that is to say, the operation attribute of maxpooling-3 is set to round up; the convolution kernel size of maxpooling-5 is set to 3×3, the stride is 1, and the atrous algorithm is used to fill in the missing features.

[0061] Specifically, the output item of the second convolutional module is the output item of the second max pooling layer maxpooling-2, the output item of the third convolutional module is the output item of the third max pooling layer maxpooling-3, and the output item of the seventh convolutional module is the output item of the fifteenth convolutional layer conv7.

[0062] The size of the chip image is converted to a preset size, the preset size is 300×300, the chip image of the preset size is input into the improved VGG16 network, a first feature map with a size of 75×75 is output through maxpooling-2, a second feature map with a size of 38×38 is output through maxpooling-3, and a third feature map with a size of 19×19 is output through conv7.

[0063] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module, and the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map. S104 includes:

[0064] S401. Input the third feature map into the additional feature extraction network. Use the output item of the first feature extraction module as the fourth feature map, the output item of the second feature extraction module as the fifth feature map, and the output item of the third feature extraction module as the sixth feature map.

[0065] Among them, the first feature extraction module includes: the sixteenth convolutional layer conv8-1 and the seventeenth convolutional layer conv8-2; the second feature extraction module includes: the eighteenth convolutional layer conv9-1 and the nineteenth convolutional layer conv9-2; the third feature extraction module includes: the twentieth convolutional layer conv10-1 and the twenty-first convolutional layer conv10-2. The output item of the first feature extraction module is the output item of the seventeenth convolutional layer conv8-2, the output item of the second feature extraction module is the output item of the nineteenth convolutional layer conv9-2, and the output item of the third feature extraction module is the output item of the twenty-first convolutional layer conv10-2.

[0066] Specifically, input the third feature map with a size of 19×19 into the additional feature extraction network. Output the fourth feature map with a size of 9×9 through conv8-2, output the fifth feature map with a size of 5×5 through conv9-2, and output the sixth feature map with a size of 3×3 through conv10-2.

[0067] Based on a chip image with a size of 300×300, through the improved VGG16 network and the additional feature extraction network in the trained scratch detection model, the following can be obtained: the first feature map with a size of 75×75, the second feature map with a size of 38×38, the third feature map with a size of 19×19, the fourth feature map with a size of 9×9, the fifth feature map with a size of 5×5, and the sixth feature map with a size of 3×3.

[0068] In one embodiment, S106 includes:

[0069] S601. Input the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0070] S602. Perform non-maximum suppression processing on the multiple detection feature maps to obtain the scratch detection result.

[0071] Specifically, the detection network includes a convolutional predictor, which is used to predict multiple detection bounding boxes (default boxes) for each feature map, and obtain a detection feature map corresponding to each feature map; the number of multiple detection bounding boxes for each feature map is equal to: m×n×k, where m×n is the size of the feature map, and k is the number of detection bounding boxes corresponding to each point in the preset feature map; k corresponding to the first feature map can be set to 3, and k corresponding to other feature maps can be set to 6; based on a chip image with a size of 300×300, the detection feature map corresponding to the first feature map can be obtained to include 75×75×3 detection bounding boxes, the detection feature map corresponding to the second feature map includes 38×38×6 detection bounding boxes, the detection feature map corresponding to the third feature map includes 19×19×6 detection bounding boxes, the detection feature map corresponding to the fourth feature map includes 10×10×6 detection bounding boxes, the detection feature map corresponding to the fifth feature map includes 5×5×6 detection bounding boxes, and the detection feature map corresponding to the sixth feature map includes 3×3×6 detection bounding boxes. Perform non-maximum suppression processing on multiple detection feature maps to obtain the scratch detection result.

[0072] Compared with the existing SSD model, the number of detection bounding boxes in the multiple detection feature maps obtained by the existing SSD model is less than that in the multiple detection feature maps obtained in this application. For small target images such as chip images, relatively fine scratches can be detected through the multiple detection feature maps of this application, improving the accuracy of scratch detection for chip images.

[0073] A suitable training data set can improve the quality of model training and the detection accuracy of the trained model. Therefore, the scratch training image set used for training the scratch detection model is crucial. Before S102, it further includes:

[0074] S101, obtain the scratch training image set.

[0075] Among them, the scratch training image set includes multiple scratch images and the annotation images of each scratch image, and each scratch image is an image of a chip.

[0076] Specifically, photograph the chip to obtain multiple original images. The images with scratches among the multiple original images are scratch images, and the images without scratches are defect-free images. Since the number of chip samples is small, the number of obtained scratch images is small. It is necessary to amplify the original images to obtain a large number of scratch images, and then annotate the scratch images to obtain annotation images.

[0077] S101 includes:

[0078] S1011, photograph the chip to obtain multiple original images, and use the images with scratches among the multiple original images as scratch images.

[0079] Specifically, an electronic chip image capturing device can be used to capture images of the chip surface. For example, a CARA robot can be used to drive a lens to capture chip samples, obtaining multiple original images. The images with scratches among the multiple original images are used as scratch images.

[0080] S1012. Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images.

[0081] Specifically, the rotation processing includes: horizontal flipping, vertical flipping, and rotating 90 degrees to the left or right. Color enhancement processing can also be performed on each scratch image to amplify the scratch images. Convert the scratch images to a preset size, and the preset size is 300×300.

[0082] In one implementation, in order to increase the scratch images, scratches can be artificially made on the captured chips. The differences between the artificially made scratch defects and the real scratch defects are as Figure 3 shown.

[0083] In one implementation, in order to increase the scratch images, for any defect-free image, randomly select target pixel points on the defect-free image, determine the preset area corresponding to the target pixel points, and replace the pixel values of all pixel points in the preset area with a preset pixel value to obtain a scratch image.

[0084] Specifically, in order to amplify the defect images, "scratches" can be created by setting the pixel values of the preset area in the defect-free image. The target pixel points can be any pixel point in any defect-free image. The preset area includes the target pixel points, and the shape of the preset area is similar to the shape of the scratch. For example, the preset area is a linear area; replace the pixel values of all pixel points in the preset area with a preset pixel value. The preset pixel value can be 0 or 255. Replace the pixel values of all pixel points in the preset area with a preset pixel value so that the preset area is different from other areas to obtain a scratch image.

[0085] In a specific embodiment, through the above S1011 and S1012, a total of 176 scratch images are obtained, including the scratch images captured, the scratch images obtained by artificially making scratches on the chips, and the scratch images obtained by replacing the pixel values of the preset area in the defect-free images. Magnification processing, Gaussian noise processing, shearing processing, rotation processing, and color enhancement processing are performed on the scratch images, so that the number of scratch images is amplified to 1056.

[0086] S1013. Perform annotation on each scratch image to obtain an annotation image corresponding to each scratch image.

[0087] Specifically, an image annotation software LabelImg can be used to annotate the scratch area in the scratch image to obtain an annotated image, and the annotated image includes the bounding box coordinate information of the scratch.

[0088] S111. Based on the scratch training image set, train the scratch detection model, and obtain the trained scratch detection model after the training is completed.

[0089] Specifically, the model result of the scratch detection model is as Figure 2 shown. The scratch detection model includes an initial improved VGG16 network, an initial additional feature extraction network, and an initial detection network. Input the scratch images in the scratch training image set into the initial improved VGG16 network, and obtain the first training feature map with a size of 75×75, the second training feature map with a size of 38×38, and the third training feature map with a size of 19×19 through the initial improved VGG16 network; input the third training feature map into the initial additional feature extraction network to obtain the fourth training feature map with a size of 9×9, the fifth training feature map with a size of 5×5, and the sixth training feature map with a size of 3×3; input the first training feature map, the second training feature map, the third training feature map, the fourth training feature map, the fifth training feature map, and the sixth training feature map into the initial detection network to obtain the scratch prediction result; calculate the loss function according to the scratch prediction result and the annotated image of the scratch image, and adjust the parameters of the scratch detection model according to the loss function. Thus, one training is completed. Train the scratch detection model multiple times until the scratch detection model converges to obtain the trained scratch detection model.

[0090] In this embodiment, a first feature map set is obtained based on the chip image and the improved VGG16 network. The first feature map set includes three feature maps with different image sizes. A second feature map set is obtained based on the first feature map set and the additional feature extraction network. The second feature map set includes three feature maps with different image sizes. Based on the first feature map set, the second feature map set, and the detection network, the scratch detection result is obtained. Compared with the existing SSD model, the existing SSD model extracts six feature maps. Among them, two feature maps are obtained through the improved VGG16 network, and four feature maps are obtained through the additional feature extraction network; in this application, the first feature map set and the second feature map set include six feature maps; the feature map with the largest image size among the six feature maps of the existing SSD model is smaller than the feature map with the largest image size in this application, and the feature map with the smallest image size among the six feature maps is smaller than the feature map with the smallest image size in this application. Generally speaking, the image features included in the six feature maps extracted by the existing SSD model are fewer than the image features included in the six feature maps in this application. This application is more suitable for scratch detection of small target images such as chip images, and improves the accuracy of scratch detection for chip images.

[0091] Since the image size of the feature map in this application is larger than that of the feature map extracted by the existing SSD model, the number of detection bounding boxes in the multiple detection feature maps obtained in this application is also larger than that in the multiple detection feature maps obtained by the existing SSD model. For small target images such as chip images, relatively fine scratches can be detected through the multiple detection feature maps of this application, improving the accuracy of scratch detection for chip images.

[0092] When training the scratch detection model in this application, a scratch training image set was obtained. By artificially creating scratches on the chip, creating scratch images by replacing pixel values on the image without missing items, and performing processing such as magnifying, Gaussian noise adding, shearing, and rotating on the scratch images, the number of scratch images was increased, enabling the scratch detection model to learn more features of scratch detection, improving the quality of the trained scratch detection model, and enhancing the accuracy of scratch detection.

[0093] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless clearly stated in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0094] Based on the same inventive concept, an embodiment of this application also provides a scratch detection device for implementing the above-mentioned scratch detection method. The solution provided by this device for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the scratch detection device provided below can refer to the limitations on the scratch detection method in the above text and will not be elaborated here.

[0095] In one embodiment, as Figure 4 shown, a scratch detection device is provided, including:

[0096] A first feature map set determination module, configured to obtain a chip image of a chip to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0097] A second feature map set determination module, configured to determine a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0098] A scratch detection result determination module, configured to obtain a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model.

[0099] In one embodiment, the improved VGG16 network includes cascaded: a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a sixth convolutional module, and a seventh convolutional module; the first feature map set includes a first feature map, a second feature map, and a third feature map; the first feature map set module includes:

[0100] A first feature map set determination unit, configured to input the chip image into the improved VGG16 network, use the output item of the second convolutional module as the first feature map, use the output item of the third convolutional module as the second feature map, and use the output item of the seventh convolutional module as the third feature map.

[0101] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; the second feature map set module includes:

[0102] A second feature map set unit, configured to input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map.

[0103] In one embodiment, the scratch detection result determination module includes:

[0104] A detection feature map determination unit, configured to input multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0105] A scratch detection result determination unit, configured to perform non-maximum suppression processing on the multiple detection feature maps to obtain a scratch detection result.

[0106] In one embodiment, the device further includes:

[0107] A scratch training image set acquisition module for acquiring a scratch training image set, where the scratch training image set includes a plurality of scratch images and an annotation image for each scratch image, and each scratch image is an image of a chip;

[0108] A training module for training a scratch detection model based on the scratch training image set and obtaining a trained scratch detection model after the training ends.

[0109] In one embodiment, the scratch training image set acquisition module includes:

[0110] A photographing unit for photographing a chip to obtain a plurality of original images, and using the images with scratches in the plurality of original images as scratch images;

[0111] An amplification unit for performing magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain a plurality of amplified scratch images;

[0112] An annotation unit for annotating each scratch image to obtain an annotation image corresponding to each scratch image.

[0113] Each module in the above scratch detection device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0114] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a scratch detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0115] Those skilled in the art can understand,Figure 5 The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0117] Obtain a chip image to be detected, and based on the chip image and the improved VGG16 network in the trained scratch detection model, determine a first feature map set, where the first feature map set includes three feature maps with different image sizes;

[0118] Based on the first feature map set and the additional feature extraction network in the trained scratch detection model, determine a second feature map set, where the second feature map set includes three feature maps with different image sizes;

[0119] Based on the first feature map set, the second feature map set and the detection network in the trained scratch detection model, obtain a scratch detection result.

[0120] In one embodiment, the improved VGG16 network includes cascaded: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module and a seventh convolution module; the first feature map set includes a first feature map, a second feature map and a third feature map; the determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes:

[0121] Input the chip image into the improved VGG16 network, use the output item of the second convolution module as the first feature map, use the output item of the third convolution module as the second feature map, and use the output item of the seventh convolution module as the third feature map.

[0122] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map and a sixth feature map; the determining the second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model includes:

[0123] Input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map.

[0124] In one embodiment, obtaining the scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model includes:

[0125] Input the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0126] Perform non-maximum suppression processing on the multiple detection feature maps to obtain the scratch detection result.

[0127] In one embodiment, before determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, it further includes:

[0128] Obtain a scratch training image set, where the scratch training image set includes multiple scratch images and the annotation image of each scratch image, and each scratch image is an image of a chip;

[0129] Based on the scratch training image set, train the scratch detection model, and obtain the trained scratch detection model after the training ends.

[0130] In one embodiment, the obtaining of the scratch training image set includes:

[0131] Take pictures of the chips to obtain multiple original images, and use the images with scratches in the multiple original images as scratch images;

[0132] Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images;

[0133] Annotate each scratch image to obtain the annotation image corresponding to each scratch image.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0135] Obtain a chip image to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0136] Determine a second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0137] Obtain a scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model.

[0138] In one embodiment, the improved VGG16 network includes cascaded: a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a sixth convolutional module, and a seventh convolutional module; the first feature map set includes a first feature map, a second feature map, and a third feature map; the determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes:

[0139] Input the chip image into the improved VGG16 network, use the output item of the second convolutional module as the first feature map, use the output item of the third convolutional module as the second feature map, and use the output item of the seventh convolutional module as the third feature map.

[0140] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; the determining the second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model includes:

[0141] Input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map.

[0142] In one embodiment, the obtaining the scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model includes:

[0143] Input the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0144] Perform non-maximum suppression processing on the multiple detection feature maps to obtain a scratch detection result.

[0145] In one embodiment, before determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, it further includes:

[0146] Obtain a scratch training image set, where the scratch training image set includes a plurality of scratch images and the annotation image of each scratch image, and each scratch image is an image of a chip;

[0147] Based on the scratch training image set, train the scratch detection model, and obtain the trained scratch detection model after the training ends.

[0148] In one embodiment, the obtaining of the scratch training image set includes:

[0149] Take pictures of the chip to obtain multiple original images, and use the images with scratches in the multiple original images as scratch images;

[0150] Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images;

[0151] Annotate each scratch image to obtain the annotation image corresponding to each scratch image.

[0152] In one embodiment, a computer program product is provided, including a computer program, which when executed by a processor implements the following steps:

[0153] Obtain a chip image to be detected, and determine a first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes;

[0154] Determine a second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes;

[0155] Obtain a scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model.

[0156] In one embodiment, the improved VGG16 network includes cascaded: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, and a seventh convolution module; the first feature map set includes a first feature map, a second feature map, and a third feature map; the determining of the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes:

[0157] Input the chip image into the improved VGG16 network, use the output item of the second convolutional module as the first feature map, use the output item of the third convolutional module as the second feature map, and use the output item of the seventh convolutional module as the third feature map.

[0158] In one embodiment, the additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; determining the second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model includes:

[0159] Input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map.

[0160] In one embodiment, obtaining the scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model includes:

[0161] Input the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes;

[0162] Perform non-maximum suppression processing on the multiple detection feature maps to obtain the scratch detection result.

[0163] In one embodiment, before determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, it further includes:

[0164] Obtain a scratch training image set, which includes multiple scratch images and the annotation image of each scratch image, and each scratch image is an image of a chip;

[0165] Train the scratch detection model based on the scratch training image set, and obtain the trained scratch detection model after the training ends.

[0166] In one embodiment, the obtaining of the scratch training image set includes:

[0167] Take pictures of the chip to obtain multiple original images, and use the images with scratches in the multiple original images as scratch images;

[0168] Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images;

[0169] Each scratch image is labeled to obtain a labeled image corresponding to each scratch image.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0172] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0173] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A scratch detection method, characterized in that, The method includes: Obtaining a chip image to be detected, and determining a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, wherein the first feature map set includes three feature maps with different image sizes; Determining a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, wherein the second feature map set includes three feature maps with different image sizes; Obtaining a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model; The improved VGG16 network includes cascaded: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, and a seventh convolution module; the first feature map set includes a first feature map, a second feature map, and a third feature map; determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model includes: Inputting the chip image into the improved VGG16 network, taking the output item of the second convolution module as the first feature map, taking the output item of the third convolution module as the second feature map, and taking the output item of the seventh convolution module as the third feature map; The additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; determining the second feature map set based on the first feature map set and the additional feature extraction network in the trained scratch detection model includes: Inputting the third feature map into the additional feature extraction network, taking the output item of the first feature extraction module as the fourth feature map, taking the output item of the second feature extraction module as the fifth feature map, and taking the output item of the third feature extraction module as the sixth feature map, and the third feature map is the feature map with the smallest size in the first feature map set.

2. The method according to claim 1, characterized in that, Obtaining the scratch detection result based on the first feature map set, the second feature map set, and the detection network in the trained scratch detection model includes: Inputting the multiple feature maps included in the first feature map set and the second feature map set into the detection network to obtain multiple detection feature maps, and each detection feature map includes multiple detection bounding boxes; Performing non-maximum suppression processing on the multiple detection feature maps to obtain the scratch detection result.

3. The method according to claim 1, characterized in that, Before determining the first feature map set based on the chip image and the improved VGG16 network in the trained scratch detection model, it further includes: Obtaining a scratch training image set, which includes multiple scratch images and the annotation image of each scratch image, and each scratch image is an image of a chip; Training the scratch detection model based on the scratch training image set, and obtaining the trained scratch detection model after the training ends.

4. The method according to claim 3, characterized in that, The obtaining the scratch training image set includes: Shoot a chip to obtain multiple original images, and use the images with scratches among the multiple original images as scratch images; Perform magnification processing, Gaussian noise processing, shearing processing, and rotation processing on each scratch image to obtain multiple amplified scratch images; Perform annotation on each scratch image to obtain an annotated image corresponding to each scratch image.

5. A scratch detection device, characterized in that, The device includes: A first feature map set determination module, configured to obtain a chip image of a chip to be detected, and determine a first feature map set based on the chip image and an improved VGG16 network in the trained scratch detection model, where the first feature map set includes three feature maps with different image sizes; A second feature map set determination module, configured to determine a second feature map set based on the first feature map set and an additional feature extraction network in the trained scratch detection model, where the second feature map set includes three feature maps with different image sizes; A scratch detection result determination module, configured to obtain a scratch detection result based on the first feature map set, the second feature map set, and a detection network in the trained scratch detection model; The improved VGG16 network includes cascaded: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, and a seventh convolution module; the first feature map set includes a first feature map, a second feature map, and a third feature map; the first feature map set determination module includes: A first feature map set determination unit, configured to input the chip image into the improved VGG16 network, use the output item of the second convolution module as the first feature map, use the output item of the third convolution module as the second feature map, and use the output item of the seventh convolution module as the third feature map; The additional feature extraction network includes cascaded: a first feature extraction module, a second feature extraction module, and a third feature extraction module; the second feature map set includes a fourth feature map, a fifth feature map, and a sixth feature map; the second feature map set determination module includes: A second feature map set determination unit, configured to input the third feature map into the additional feature extraction network, use the output item of the first feature extraction module as the fourth feature map, use the output item of the second feature extraction module as the fifth feature map, and use the output item of the third feature extraction module as the sixth feature map, and the third feature map is the feature map with the smallest size in the first feature map set.

6. A computer device, including a memory and a processor, the memory stores a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer program product, including a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images