Defect detection method and related equipment
The method improves lithium battery defect detection efficiency and accuracy by cropping images into segments, using classification and detection models to filter and identify defects, addressing the inefficiencies of existing methods.
Patent Information
- Application Number
- CN202411025224.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-10
AI Technical Summary
Existing defect detection methods for lithium batteries suffer from low efficiency and accuracy due to the varied forms and sizes of defects, leading to missed or incorrect detections, especially when dealing with large images.
A method involving ROI area determination, cropping the image into smaller segments, using an image classification model to filter out defect-free images, and employing a defect detection model to identify defects in these segments, thereby improving detection efficiency and accuracy.
Enhances defect detection efficiency and accuracy by cropping large images into smaller segments, filtering out defect-free images, and utilizing specialized models for precise defect identification, while reducing time consumption for large-resolution image processing.
Smart Images
Figure CN119107283A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a defect detection method, device, electronic device, storage medium and computer program product. Background Art
[0002] As one of the most widely used batteries at present, lithium batteries are a type of battery that uses lithium metal or lithium alloy as positive / negative electrode materials and non-aqueous electrolyte solutions. Its performance is closely related to the stacking quality.
[0003] At present, poor appearance often occurs during the stacking process of lithium batteries. There are many manifestations of poor appearance, such as scratches, breakage, bubbles, metal leakage and foreign matter. The shapes and colors of these lithium battery appearance defects vary greatly and vary in size. Defect detection solutions based on traditional algorithms are difficult to be compatible with so many types and forms of defect detection in one test because they all perform overall detection on the original image. There is a high possibility of missed detection or wrong detection and low defect detection efficiency. Summary of the invention
[0004] The embodiment of the present application provides a defect detection method to solve the problem that the existing defect detection scheme has low defect detection efficiency and accuracy when performing defect detection on the appearance of a lithium battery.
[0005] Embodiments of the present application also provide a defect detection device, an electronic device, a computer-readable storage medium, and a computer program product.
[0006] The present application embodiment adopts the following technical solutions:
[0007] In a first aspect, the present application provides a defect detection method, comprising:
[0008] Determine the ROI area of the target image to be defect-detected, and crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image;
[0009] Determining a first number of defect thumbnails from the specified number of thumbnails by an image classification model;
[0010] Determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model;
[0011] Determine the defect information of the target image based on the defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails.
[0012] In a second aspect, the present application provides a defect detection device, comprising:
[0013] An image cropping module is used to determine the ROI area of the target image to be inspected for defects, and to crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image;
[0014] An image classification module, configured to determine a first number of defect thumbnails from the specified number of thumbnails by using an image classification model;
[0015] A defect detection module, used to determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model;
[0016] A defect determination module is used to determine the defect information of the target image based on the defect information and cropping coordinate information of each defect sub-image in the first number of defect sub-images.
[0017] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the defect detection method described above when executed by the processor.
[0018] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the defect detection method as described above are implemented.
[0019] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps in the defect detection method as described above when executed by a processor.
[0020] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0021] By adopting the method provided in the embodiment of the present application, when performing defect detection on the target image, since the ROI area of the target image to be defect-detected can be determined, and the ROI area of the target image can be cropped, a specified number of small pictures and the cropping coordinate information of each small picture can be obtained; then, through the image classification model, a first number of defective small pictures are determined from the specified number of small pictures, and the small pictures without defects are filtered out, and only the small pictures with defects are retained; and through the defect detection model, the defect information of each defective small picture in the first number of defective small pictures is determined; based on the defect information and cropping coordinate information of each defective small picture in the first number of defective small pictures, the defect information of the target image can be determined. Since the original target image can be cropped into multiple small pictures, and the small pictures without defects are filtered out, only the defect detection model is allowed to detect the defect information of the defective small pictures, compared with the prior art, the defect detection efficiency and accuracy of the defect detection for the entire large picture can be effectively improved. Moreover, by adopting the method provided in the embodiment of the present application, a single defect detection model can also be compatible with image detection of different sizes of large, medium and small, while solving the problem of high time consumption of large-resolution image reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0023] Figure 1 A schematic diagram of an implementation flow of a defect detection method provided in an embodiment of the present application;
[0024] Figure 2 A schematic diagram of marking defect information on a target image in a defect detection method provided in an embodiment of the present application;
[0025] Figure 3 A schematic diagram of equal division and cropping in the defect detection method provided in an embodiment of the present application;
[0026] Figure 4 A schematic diagram of sliding window clipping in the defect detection method provided in an embodiment of the present application;
[0027] Figure 5 A schematic diagram of a defect atlas and a non-defect atlas in a defect detection method provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of the structure of an image classification network in a defect detection method provided in an embodiment of the present application;
[0029] Figure 7 A schematic diagram of the YOLOv5-seg instance segmentation network structure in the defect detection method provided in an embodiment of the present application;
[0030] Figure 8 to Figure 12 A schematic diagram of the effect of implementing the defect detection method provided in an embodiment of the present application;
[0031] Fig.13 A schematic diagram of the structure of a defect detection device provided in an embodiment of the present application;
[0032] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0034] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0035] At present, the existing defect detection method needs to perform a defect detection on a complete image to be defect-detected through a trained defect detection model to identify the defect location and defect category in the image. This defect detection method relies on the recognition accuracy of the model. If a defect category that the model has not learned is encountered, problems such as missed detection or false detection may occur. Moreover, if the image is too large, the efficiency of defect detection is also low. To solve this problem, the embodiment of the present application provides a defect detection method.
[0036] The execution subject of the method may be various types of computing devices, or an application or application (Application, APP) installed on the computing device. The computing device may be, for example, a user terminal such as a mobile phone, a tablet computer, a smart wearable device, or a server.
[0037] For ease of description, the present application embodiment takes the execution subject of the method as an example of a server to introduce the method. Those skilled in the art can understand that the present application embodiment takes the server as an example to introduce the method, which is only an exemplary description and does not limit the scope of protection of the claims corresponding to the present solution.
[0038] First, the terms involved in one or more embodiments of the present application are explained.
[0039] ROI: The full name of ROI is Region of Interest. In deep learning, a common application is to use convolutional neural networks for object detection. When processing images or videos, the network needs to identify objects in the image, and usually, it is only interested in some specific areas of the image, not the entire image. This specific area can be called ROI.
[0040] Mask: The English name is mask. It uses a selected image, graphic or object to block the processed image (all or part) to control the area or process of image processing. The specific image or object used for covering is called a mask or template. The mask is a binary image composed of 0 and 1. When a mask is applied to a function, the 1-value area is processed, and the masked 0-value area is not included in the calculation. The image mask is defined by specifying data values, data ranges, finite or infinite values, areas of interest, and annotation files. Any combination of the above options can also be used as input to establish a mask. A mask is a template for an image filter. For example, in product quality inspection under machine vision scenarios, when extracting surface defects of products, an N*N matrix is used to filter the image pixels, and then the marks we need are highlighted. This matrix is a kind of mask.
[0041] Image segmentation: The technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. It is a key step from image processing to image analysis. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories, such as deep learning. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-overlapping regions. The image segmentation process is also a labeling process, that is, pixels belonging to the same region are assigned the same number.
[0042] Specifically, the implementation process of the method provided in the embodiment of the present application is as follows: Figure 1 As shown, the following steps are included:
[0043] Step 110 , determining the ROI area of the target image to be inspected for defects, and cropping the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image.
[0044] In some exemplary embodiments, the ROI region can be obtained by the following methods, including extraction using a rectangular region and extraction using a mask. Among them, extraction using a rectangular region may include the following methods: ① specifying the region of interest by defining the coordinates of the upper left corner and the width and height of the rectangle; ② using the cv::Rect class to represent a rectangular region, and specifying the region of interest by defining its upper left corner coordinates and the width and height; ③ using the cv::Mat::operator() function, by passing the rectangular region in parentheses, the corresponding sub-image can be obtained from the original image. Extraction using a mask includes the following methods: ① creating a binary mask image of the same size as the original image; ② defining the region of interest on the mask, setting the pixels of the region to white (255), and other positions to black (0); ③ applying the mask to the original image, only the positions where the corresponding pixels in the mask are white will be retained, and other positions will be suppressed. In addition, the ROI can also be extracted by manual selection or by programmatically generating a mask region. Manual selection usually allows users to directly select the region of interest on the image through a graphical user interface, while programmatically generating a mask may require writing code to automatically determine which pixels belong to the ROI. For example, masks can be automatically generated by polygon filling or by analyzing image features.
[0045] Taking into account that when performing defect detection on the original target image, defects in certain areas of the target image may be missed or misdetected during a single defect detection process due to the image being too large, in order to improve the efficiency and accuracy of defect detection of the defect detection model, the embodiment of the present application may crop the ROI area of the target image to obtain a specified number of small images, and then perform defect detection on the specified number of small images respectively through the defect detection model. Since each small image is much smaller than the original target image, the amount of data calculated by the defect detection model when it is input into the defect detection model at a single time is also much smaller, thereby effectively improving the efficiency and accuracy of defect detection of the defect detection model each time.
[0046] Optionally, the ROI area can be cropped by equal cropping or sliding window cropping. Specifically, the ROI area of the target image is cropped to obtain a specified number of small images and cropping coordinate information of each small image, including:
[0047] The ROI area of the target image is evenly divided and cropped to obtain a first specified number of small images and cropping coordinate information of each small image; or,
[0048] The ROI area of the target image is cropped by sliding window to obtain a second specified number of small images and cropping coordinate information of each small image.
[0049] Among them, as the name suggests, the average cropping is to evenly divide the ROI area of the target image into a first specified number of small images, each of which has the same size, that is, the length and width of each small image are the same, and there is no overlap of image areas between adjacent small images. Figure 3 A schematic diagram of equal division and cropping in the defect detection method provided in an embodiment of the present application, Figure 3 The ROI area of the left image in the figure is evenly divided and cropped into 8 small images in the right image, where the coordinates of the cropping starting point and the image width and height are {(x i ,y i ), i=[1,8]} and (w / 2,h / 4), in which case the first specified number is 8.
[0050] Sliding window cropping sets the stride and window size, and crops the image in the window every time a stride is moved. The sizes of the second specified number of small images obtained after sliding window cropping are also the same. The difference is that after sliding window cropping, there is overlap in image areas between adjacent small images. Compared with equal cropping, sliding window cropping can crop the target image into more small images. Figure 4 A schematic diagram of sliding window clipping in the defect detection method provided in an embodiment of the present application, Figure 4 The ROI area of the left image in the figure is cropped into 15 small images in the right image by sliding, and the second specified number is 15. It can be seen that although the sizes of the cropped small images are similar, the sliding window cropping can crop the target image into more small images compared with the equal-division cropping, that is, generally speaking, the second specified number is greater than the first specified number.
[0051] Step 120: Determine a first number of defect thumbnails from a specified number of thumbnails by using an image classification model.
[0052] Among them, the role of the image classification model is to classify defective images and non-defective images. In order to improve the classification efficiency of defective small images, the multi-threaded image classification mode can be enabled, that is, each thread calls an image classification model to achieve parallel image classification of a specified number of small images. For example, if the specified number is 100, 25 threads can be enabled, and each thread calls an image classification model, then the image classification of 25 small images can be processed in parallel at a time. After four rounds of image classification, the image classification of 100 small images can be completed, and the first number of defective small images among the 100 small images can be determined.
[0053] In some embodiments, the training process of the image classification model may include:
[0054] Acquire training sample data, where the training sample data includes multiple groups of small images and annotation information based on defect information in each small image, where each group of small images is obtained by cropping an ROI region based on a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information;
[0055] Based on the training sample data, a preset image classification model is trained to obtain an image classification model for classifying defect images and non-defect images.
[0056] In some exemplary embodiments, the defect information may include information such as defect category and defect location. The defect category is used to indicate the category to which the defect belongs, including scratches, breakages, bubbles, metal leakage, foreign matter, etc. The defect location is used to indicate the location information of the defect in the image, and in practical applications, the location information may be expressed in the form of coordinate information.
[0057] It should be understood that since the role of the image classification model is to identify whether an image has defects or not, during the image classification model training process, the above-mentioned annotation information may include both defective and non-defective information.
[0058] The process of obtaining training sample data may include:
[0059] Collect battery appearance sample data in industrial scenarios and use polygonal boxes to mark NG defects, such as Figure 1 As shown, the defective area in each image is marked with a polygonal box, and the image mask is saved to obtain a marked sample set;
[0060] Crop each image in the labeled sample set to obtain a specified number of small images corresponding to each image in the labeled sample set;
[0061] Determine the defect-free images and defective images in the specified number of small images corresponding to each image and the corresponding annotation information.
[0062] Figure 5 A schematic diagram of a defect atlas and a non-defect atlas in a defect detection method provided in an embodiment of the present application. Figure 5 The left image in the figure is a non-defective atlas, and the right image is a defective atlas. Each defective image corresponds to a labeling information, which includes labeling information on the defect position and defect category in the defective image. Among them, the non-defective atlas can be represented by an OK image, and the defective atlas can be represented by an NG image, that is, the labeling information may include OK for indicating that the image is a non-defective image and NG for indicating that the image is a defective image.
[0063] As an example, after obtaining the training sample data, the training parameters can be set to train the preset image classification network to obtain the image classification network. Among them, the image classification model is based on the convolutional neural network structure, for example, the Resnet18 network is used as the feature extraction network (CNN Blocks). In order to classify small defects more effectively, the embodiment of the present application uses the Adp_Max_Pooling operation to perform feature pooling operation. Specifically, the process of feature pooling operation can use maximum pooling to replace the average pooling in the network. Among them, average pooling is to take the average value of the feature points in the neighborhood. Small targets occupy fewer pixels and features and are smaller after being averaged; while maximum pooling is to take only the maximum value of the feature points in the neighborhood, which is conducive to obtaining the feature points of small targets, so it can more effectively classify small defects.
[0064] Figure 6 A schematic diagram of the structure of the image classification network in the defect detection method provided in the embodiment of the present application. When training the preset image classification model based on the training sample data, the preset image classification model constructed by the Resnet18 network can be trained, using the cross-entropy loss function, the SGD optimization algorithm, the initial learning rate of 0.01, the training size of [896,640], and the training step of 100,000 steps, until the image classification model converges. It can be seen that Figure 6 The input image is processed by CNN Blocks convolution to obtain a feature map, and then processed by Adp_MaxPooling pooling to obtain a feature map with reduced size. Finally, the pooled features are integrated and classified by FC to obtain the image classification result OK or NG.
[0065] Step 130: Determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model.
[0066] Among them, the role of the defect detection model is to identify the defect location and defect category in the input image to output the defect location and defect category of the input image.
[0067] In some exemplary embodiments, the training process of the defect detection model includes:
[0068] Acquire training sample data, where the training sample data includes multiple groups of small images and annotation information based on defect information in each small image, where each group of small images is obtained by cropping an ROI region based on a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information;
[0069] Based on the training sample data, a preset instance segmentation model is trained to obtain a defect detection model for detecting defect information.
[0070] As an example, after obtaining training sample data, training parameters can be set to train the instance segmentation network to obtain a defect detection model.
[0071] The instance segmentation network has a one-stage and a two-stage structure. For example, a one-stage YOLOv5-seg network can be used, which includes image preprocessing, feature extraction, feature fusion, prediction head (YOLO Head) and mask branch. Its network structure is as follows: Figure 7 As shown. The embodiment of the present application can adopt YOLOv5n-seg structure training to reduce the time consumption of reasoning. It should be noted that the defect detection network in the embodiment of the present application can be used as long as it is trained on the basis of a lightweight segmentation network, and is not limited to the YOLO5-Seg network. Due to the YOLOv5-seg instance segmentation network, there are 5 structures according to the depth and width of the network, namely YOLOv5n-seg, YOLOv5s-seg, YOLOv5m-seg, YOLOv5l-seg and YOLOv5x-seg, among which YOLOv5n-seg and YOLOv5x-seg are the minimum and maximum structures respectively. The embodiment of the present application adopts YOLOv5n-seg minimum structure training, which can effectively reduce the time consumption of reasoning.
[0072] Specifically, when training the YOLOv5n-seg instance segmentation network, the loss function is loss = lbox + lobj + lcls + lseg, where lbox is the regression loss, lobj is the confidence loss, lcls is the category loss, and lseg is the mask classification loss. The optimizer uses the SGD optimization algorithm, with an initial learning rate of 0.01, a training size of [960, 1280], and 200 epochs until the defect detection model converges.
[0073] Optionally, to improve defect detection efficiency, the defect information of each defect sub-image in the first number of defect sub-images is determined by using a defect detection model, including:
[0074] Calling multiple threads to determine defect information of each defect thumbnail in the first number of defect thumbnails through a defect detection model; or,
[0075] Batch inference is performed on the first number of defect thumbnails through the defect detection model to determine defect information of each defect thumbnail in the first number of defect thumbnails.
[0076] Specifically, batch reasoning means that the defect detection model uses a single thread to process multiple small defect images at one time.
[0077] Step 140 : Determine defect information of the target image based on defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails.
[0078] In order to facilitate viewing of defect information in the target image, such as the location of the defect in the target image and the defect category, and thus repair the product corresponding to the target image according to the defect information of the target image, the embodiment of the present application can also restore the defect information of each defect thumbnail in the first number of defect thumbnails to the original target image. Specifically, based on the defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails, the defect information of the target image is determined, including:
[0079] splicing the first number of defect thumbnails into the target image according to the cropping coordinate information of each defect thumbnail in the first number of defect thumbnails;
[0080] Determine defect information of the target image based on defect information of each defect sub-image in the first number of defect sub-images after splicing.
[0081] As an example, the defect information of each defect sub-image in the first number of defect sub-images after splicing may be displayed in a visual form in the target image, so as to determine the defect position and defect category of the target image. Figure 8 to Figure 12 This is a schematic diagram of the effect of implementing the defect detection method provided in the embodiment of the present application. It can be seen that Figure 8 to Figure 12 The original target image is cropped into 8 small images by equal cropping, and then the image classification network is used to classify the small images. Figure 8-Figure 12 The 8 small pictures in the image classification are determined respectively Figure 8-Figure 12 The defect images in the 8 small pictures in the figure are then detected by the defect detection network. Figure 8-Figure 12 Defect detection is performed on the defect image in the Figure 8-Figure 12 The defect information in each defect image in the image is annotated in each defect image. Figure 8-Figure 12 By stitching together the small pictures in Figure 8-Figure 12 Defect information of each image in .
[0082] By adopting the method provided in the embodiment of the present application, when performing defect detection on the target image, since the ROI area of the target image to be defect-detected can be determined, and the ROI area of the target image can be cropped, a specified number of small pictures and the cropping coordinate information of each small picture can be obtained; then, through the image classification model, a first number of defective small pictures are determined from the specified number of small pictures, and the small pictures without defects are filtered out, and only the small pictures with defects are retained; and through the defect detection model, the defect information of each defective small picture in the first number of defective small pictures is determined; based on the defect information and cropping coordinate information of each defective small picture in the first number of defective small pictures, the defect information of the target image can be determined. Since the original target image can be cropped into multiple small pictures, and the small pictures without defects are filtered out, only the defect detection model is allowed to detect the defect information of the defective small pictures, compared with the prior art, the defect detection efficiency and accuracy of the defect detection for the entire large picture can be effectively improved. Moreover, by adopting the method provided in the embodiment of the present application, a single defect detection model can also be compatible with image detection of different sizes of large, medium and small, while solving the problem of high time consumption of large-resolution image reasoning.
[0083] The present application also provides a defect detection device. The specific structural diagram of the device is as follows: Fig.13 As shown, it includes an image cropping module 1301, an image classification module 1302, a defect detection module 1303 and a defect determination module 1304. The functions of each module are as follows:
[0084] The image cropping module 1301 is used to determine the ROI area of the target image to be defect-detected, and to crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image;
[0085] An image classification module 1302 is used to determine a first number of defect thumbnails from the specified number of thumbnails by using an image classification model;
[0086] A defect detection module 1303 is used to determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model;
[0087] The defect determination module 1304 is used to determine the defect information of the target image based on the defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails.
[0088] Optionally, when the defect determination module 1304 determines the defect information of the target image based on the defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails, it is specifically used to:
[0089] splicing the first number of defect thumbnails into the target image according to the cropping coordinate information of each defect thumbnail in the first number of defect thumbnails;
[0090] Determine the defect information of the target image based on the defect information of each defect thumbnail in the first number of spliced defect thumbnails.
[0091] Optionally, when the image cropping module 1301 crops the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image, it is specifically used to:
[0092] The ROI area of the target image is evenly divided and cropped to obtain a first specified number of small images and cropping coordinate information of each small image; or,
[0093] The ROI area of the target image is cropped with a sliding window to obtain a second specified number of small images and cropping coordinate information of each small image.
[0094] Optionally, when the defect detection module 1303 determines the defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model, it is specifically used to:
[0095] Calling multiple threads to determine the defect information of each defect thumbnail in the first number of defect thumbnails through a defect detection model; or,
[0096] The defect detection model is used to perform batch inference on the first number of defect thumbnails to determine defect information of each defect thumbnail in the first number of defect thumbnails.
[0097] Optionally, the training process of the image classification model includes:
[0098] Acquire training sample data, where the training sample data includes multiple groups of thumbnails and annotation information based on defect information in each thumbnail, where each group of thumbnails is obtained by cropping an ROI region of a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information;
[0099] Based on the training sample data, a preset image classification model is trained to obtain an image classification model for classifying defect images and non-defect images.
[0100] Optionally, the training process of the defect detection model includes:
[0101] Acquire training sample data, where the training sample data includes multiple groups of thumbnails and annotation information based on defect information in each thumbnail, where each group of thumbnails is obtained by cropping an ROI region of a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information;
[0102] Based on the training sample data, a preset instance segmentation model is trained to obtain a defect detection model for detecting defect information.
[0103] By using the device provided by the embodiment of the present application, when performing defect detection on the target image, since the ROI area of the target image to be defect-detected can be determined, and the ROI area of the target image can be cropped, a specified number of small images and the cropping coordinate information of each small image can be obtained; then, through the image classification model, a first number of defective small images are determined from the specified number of small images, and the small images without defects are filtered out, and only the small images with defects are retained; and through the defect detection model, the defect information of each defective small image in the first number of defective small images is determined; based on the defect information and cropping coordinate information of each defective small image in the first number of defective small images, the defect information of the target image can be determined. Since the original target image can be cropped into multiple small images, and the small images without defects are filtered out, only the defect detection model is allowed to detect the defect information of the defective small images, compared with the prior art, the defect detection efficiency and accuracy of the defect detection for the entire large image can be effectively improved. Moreover, by using the method provided by the embodiment of the present application, a single defect detection model can also be compatible with image detection of different sizes of large, medium and small, while solving the problem of high time consumption of large-resolution image reasoning.
[0104] Fig.14 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Fig.14 As shown, the device includes: a memory 141 and a processor 142.
[0105] The memory 141 is used to store computer programs and can be configured to store various other data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, pictures, videos, etc.
[0106] The processor 142 is coupled to the memory 141 and is used to execute the computer program in the memory 141, so as to: determine the ROI area of the target image to be defect-detected, and crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image; determine a first number of defect small images from the specified number of small images through an image classification model; determine the defect information of each defect small image in the first number of defect small images through a defect detection model; determine the defect information of the target image based on the defect information and cropping coordinate information of each defect small image in the first number of defect small images.
[0107] By using the electronic device provided by the embodiment of the present application, when performing defect detection on the target image, since the ROI area of the target image to be defect-detected can be determined, and the ROI area of the target image can be cropped, a specified number of small images and the cropping coordinate information of each small image can be obtained; then, through the image classification model, a first number of defective small images are determined from the specified number of small images, and the defect-free small images are filtered out, and only the defective small images are retained; and through the defect detection model, the defect information of each defective small image in the first number of defective small images is determined; based on the defect information and cropping coordinate information of each defective small image in the first number of defective small images, the defect information of the target image can be determined. Since the original target image can be cropped into multiple small images, and the defect-free small images are filtered out, only the defect detection model is allowed to detect the defect information of the defective small images, compared with the prior art, the defect detection efficiency and accuracy of the defect detection for the entire large image can be effectively improved. Moreover, by using the method provided by the embodiment of the present application, a single defect detection model can also be compatible with image detection of different sizes of large, medium and small, while solving the problem of high time consumption of large-resolution image reasoning.
[0108] Further, if Fig.14 As shown, the electronic device also includes: a communication component 143, a display 144, a power component 145, an audio component 146 and other components. Fig.14 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 2 In addition, depending on the implementation form of the traffic playback device, Fig.14 The components in the dashed box are optional components, not mandatory components. For example, when the electronic device is implemented as a terminal device such as a smartphone, tablet computer or desktop computer, it may include Fig.14 Components in the dashed box; when the electronic device is implemented as a server-side device such as a conventional server, cloud server, data center or server array, it may not include Fig.14 Components within the dashed box.
[0109] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method embodiment.
[0110] Accordingly, the embodiment of the present application further provides a computer program product, including a computer program / instruction, which can implement each step that can be performed by the electronic device in the above method embodiment when the computer program / instruction is executed. Optionally, the computer program product, in addition to executing each step in the above method embodiment.
[0111] Above Fig.14The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may also include a near field communication (NFC) module, a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wideband (UWB) technology, a Bluetooth (BT) technology, etc.
[0112] Above Fig.14 The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0113] Above Fig.14 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0114] Above Fig.14 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.
[0115] Above Fig.14 The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.
[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0122] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0125] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A defect detection method, It is characterized in that include: Determine the ROI area of the target image to be defect-detected, and crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image; Determining a first number of defect thumbnails from the specified number of thumbnails by an image classification model; Determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model; Determine the defect information of the target image based on the defect information and cropping coordinate information of each defect thumbnail in the first number of defect thumbnails.
2. The method according to claim 1, It is characterized in that The determining the defect information of the target image based on the defect information and the cropping coordinate information of each defect thumbnail in the first number of defect thumbnails includes: splicing the first number of defect thumbnails into the target image according to the cropping coordinate information of each defect thumbnail in the first number of defect thumbnails; Determine the defect information of the target image based on the defect information of each defect thumbnail in the first number of spliced defect thumbnails.
3. The method according to claim 1, It is characterized in that The step of cropping the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image includes: The ROI area of the target image is evenly divided and cropped to obtain a first specified number of small images and cropping coordinate information of each small image; or, The ROI area of the target image is cropped with a sliding window to obtain a second specified number of small images and cropping coordinate information of each small image.
4. The method according to claim 1, It is characterized in that The step of determining the defect information of each defect thumbnail in the first number of defect thumbnails by using the defect detection model includes: Calling multiple threads to determine the defect information of each defect thumbnail in the first number of defect thumbnails through a defect detection model; or, The defect detection model is used to perform batch inference on the first number of defect thumbnails to determine defect information of each defect thumbnail in the first number of defect thumbnails.
5. The method according to claim 1, It is characterized in that The training process of the image classification model includes: Acquire training sample data, where the training sample data includes multiple groups of thumbnails and annotation information based on defect information in each thumbnail, where each group of thumbnails is obtained by cropping an ROI region of a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information; Based on the training sample data, a preset image classification model is trained to obtain an image classification model for classifying defect images and non-defect images.
6. The method according to claim 1, It is characterized in that The training process of the defect detection model includes: Acquire training sample data, where the training sample data includes multiple groups of thumbnails and annotation information based on defect information in each thumbnail, where each group of thumbnails is obtained by cropping an ROI region of a product appearance image in an industrial scenario, and each product appearance image carries annotation information of defect information; Based on the training sample data, a preset instance segmentation model is trained to obtain a defect detection model for detecting defect information.
7. A defect detection device, It is characterized in that include: An image cropping module is used to determine the ROI area of the target image to be defect-detected, and to crop the ROI area of the target image to obtain a specified number of small images and cropping coordinate information of each small image; An image classification module, configured to determine a first number of defect thumbnails from the specified number of thumbnails by using an image classification model; A defect detection module, used to determine defect information of each defect thumbnail in the first number of defect thumbnails by using a defect detection model; A defect determination module is used to determine the defect information of the target image based on the defect information and cropping coordinate information of each defect sub-image in the first number of defect sub-images.
8. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, It is characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.