A detection method, computer device, and storage medium
By using a combination method of backbone network and multiple branch networks in industrial image defect detection, the problem of defect classification with large morphological differences and similar morphological defect classification is solved, the detection accuracy and efficiency are improved, and the automation and accuracy of defect detection are achieved. Classification.
Patent Information
- Application Number
- CN201910459436.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-05-29
AI Technical Summary
In industrial image defect detection, defects with uneven morphological differences lead to low detection accuracy and low efficiency, especially defects with similar morphological shapes are difficult to accurately classify, which is prone to false inspection or false inspection, affecting product quality and customer interests.
A detection method is adopted to extract feature by inputting object pictures into defect detection network's backbone network to obtain feature data, and input it into multiple branch networks for detection. Each branch network specifically deals with one category of defects to avoid learning multiple learning at the same time. This type of defects can improve detection accuracy and efficiency.
It improves the accuracy and efficiency of defect detection, saves computing resources, realizes the accurate classification and distinction of defects with similar shapes, reduces labor costs, and realizes comprehensive automation of detection.
Smart Images

Figure CN112016575B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a detection method, a computer device, and a computer-readable storage medium. Background Art
[0002] With the continuous upgrading of industrial automation, higher and higher requirements are put forward for the application of artificial intelligence in the industrial field. In the field of industrial image defect detection, there are many forms of defects, and the morphological differences between different defects are uneven. For example, some defects have similar morphologies, while some defects have large morphological differences. This characteristic of industrial defects is a major problem faced by defect detection and classification.
[0003] As we all know, different forms of defects have different degrees of impact on the business, so customers have relatively high requirements for the accuracy of defect classification. If the defect category is incorrectly classified, it will cause losses in two aspects: first, a defect with a large impact will be mistakenly detected as a defect with a small impact, which will cause unexpected serious problems during the use of the product, causing serious losses to downstream customers and causing complaints from downstream customers; second, a defect with a small impact will be mistakenly detected as a defect with a large impact, resulting in low product pricing, and the product will be sold at a low price, causing customer revenue loss.
[0004] The applicant has found through research that when there are many defect shapes on the inspection object and the morphological differences between defects of different shapes are uneven, it is possible to better distinguish between defects with large differences, but classification errors are prone to occur between defects with similar morphological differences, resulting in problems of low detection accuracy and low detection efficiency. Summary of the invention
[0005] In view of the above problems, the present application is proposed to provide a detection method, a computer device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.
[0006] According to one aspect of the present application, a detection method is provided, comprising:
[0007] Input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data;
[0008] Inputting the characteristic data into a plurality of branch networks of the defect detection network;
[0009] According to the feature data, the multiple branch networks respectively detect different categories of defects on the object image and output defect detection results.
[0010] Optionally, the multiple branch networks are respectively trained using sample defect data of different categories labeled for object image samples.
[0011] Optionally, before inputting the object image into the backbone network of the defect detection network for feature extraction to obtain feature data, the method further includes:
[0012] Inputting an object image sample into the main network for feature extraction to obtain sample feature data, and outputting the sample feature data to the multiple branch networks;
[0013] The main network and multiple branch networks are trained using sample defect data of different categories labeled for the object image samples to obtain the defect detection network.
[0014] Optionally, the adopting of sample defect data of different categories labeled for the object image samples to train the trunk network and the plurality of branch networks to obtain the defect detection network comprises:
[0015] Comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data;
[0016] According to the sample difference data of the plurality of branch networks, the corresponding branch networks and the trunk network are corrected.
[0017] Optionally, the sample defect data includes subcategories of sample defects, and before comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data to obtain sample difference data of the plurality of branch networks, the method further includes:
[0018] The category corresponding to the sample defect data is determined according to the parent category to which the subcategory of the sample defect belongs.
[0019] Optionally, the sample defect data includes subcategories of sample defects, and before outputting the sample feature data to the plurality of branch networks, the method further includes:
[0020] According to the parent category to which the subcategory of the sample defect belongs, category information is added to the sample feature data at the feature layer of the defect detection network.
[0021] Optionally, the sample defect data includes at least one of the following: a location area of the sample defect, a subcategory of the sample defect, and pixel data of the sample defect.
[0022] Optionally, the defect detection result includes at least one of a location area of the defect and pixel data of the defect. After the plurality of branch networks respectively detect different categories of defects on the object image according to the feature data and output the defect detection result, the method further includes:
[0023] Determining specification information of the defect on the object image according to the location area and / or pixel data of the defect;
[0024] Provides the stated specification information.
[0025] Optionally, the defect detection result includes at least one of a location area of the defect and pixel data of the defect. After the plurality of branch networks respectively detect different categories of defects on the object image according to the feature data and output the defect detection result, the method further includes:
[0026] Determine an area adjacent to the defect on the object image according to the location area and / or pixel data of the defect;
[0027] comparing the defect with a surrounding area to obtain comparison information;
[0028] The comparative information is provided.
[0029] Optionally, the object picture includes at least one of the following: a picture of a polycrystalline silicon cell, a picture of a polycrystalline silicon module, a picture of a steel plate, a picture of a cloth, an X-ray picture of a tire, a picture of an electronic device, and a picture of food.
[0030] According to another aspect of the present application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements one or more of the above methods when executing the computer program.
[0031] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, one or more of the above methods are implemented.
[0032] According to an embodiment of the present application, an object image is input into a backbone network of a defect detection network for feature extraction to obtain feature data, and the feature data is input into multiple branch networks of the defect detection network. According to the feature data, the multiple branch networks respectively detect different categories of defects on the object image and output defect detection results, so that each branch of the neural network can detect a category of defects separately in a targeted manner, thereby avoiding the problem of low detection accuracy when the neural network learns multiple defect forms at the same time. On the other hand, it also consumes less computing resources than using multiple neural networks for detection, and improves the accuracy and efficiency of defect detection while consuming the same computing resources.
[0033] Furthermore, by adding category information to the sample feature data in the feature layer of the defect detection network according to the parent category to which the subcategory of the sample defect belongs, the overall accuracy of the defect detection network can be improved after adding the category information as classification supervision information, and the newly added multi-layer convolution can be directly pruned during network prediction, which will not have any impact on the efficiency of the algorithm.
[0034] Furthermore, by determining the specification information of the defect on the object image based on the location area and / or pixel data of the defect, the specification information is provided to facilitate subsequent repair or defect grading, thereby achieving full automation of detection, saving labor costs and improving detection efficiency.
[0035] Furthermore, by determining the adjacent area of the defect on the object image based on the location area and / or pixel data of the defect, and comparing the defect with the surrounding area, comparison information is obtained, and the comparison information is provided to facilitate subsequent repairs or defect grading, thereby achieving full automation of detection, saving labor costs, and improving detection efficiency.
[0036] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0038] Figure 1 A photograph of a cell sheet is shown;
[0039] Figure 2 A schematic diagram showing a multi-branch hierarchical network structure;
[0040] Figure 3 A schematic diagram of the detection process is shown;
[0041] Figure 4 A flow chart of a detection method embodiment according to Embodiment 1 of the present application is shown;
[0042] Figure 5 A flow chart of a detection method embodiment according to the second embodiment of the present application is shown;
[0043] Figure 6A structural block diagram of a detection device embodiment according to Embodiment 3 of the present application is shown;
[0044] Figure 7 An exemplary system is shown that can be used to implement the various embodiments described in this disclosure. DETAILED DESCRIPTION
[0045] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0046] In order to enable those skilled in the art to better understand the present application, the concepts involved in the present application are explained below:
[0047] The object image includes a visible light photo of the detection object, an X-ray (X-ray, Roentgen ray) photo of the detection object, an infrared photo of the detection object, etc., or any other applicable image, which is not limited in the embodiments of the present application.
[0048] For example, when detecting defects in solar cells, Figure 1 The photo of the cell shown in the figure shows that for solar cells, more than 20 types of defects need to be detected, including: "single crack", "cross crack", "missing corner", "black spot", "bright flake", "light and dark flake", etc. The morphological differences between different defects vary. For example, the two defects of "single crack" and "cross crack" are similar, while the difference between "single crack" and "bright flake" is relatively large.
[0049] To detect defects on an object image, a neural network can be used to learn defects on the object image to obtain a defect detection network that can detect defects. The defect detection network can be used to detect the position, shape, segmentation (defect area segmented from the background), etc. of the defect, or any other applicable detection, which is not limited in the embodiments of the present application.
[0050] For example, by using a supervised learning method, object image samples with the location and shape of sample defects marked are input, and the parameters in the convolutional neural network are continuously adjusted according to the location and shape of the marked sample defects until the required performance is achieved, thereby obtaining a defect detection network.
[0051] For the detection needs of defects of different forms, if a defect detection network is created specifically for each form of defect, when actually performing defect detection, using multiple defect detection networks to perform detection separately can accurately detect and clearly distinguish multiple defects. However, it is obvious that both the training process and the detection process will consume more computing resources and the efficiency is too low. For example, in the defect detection of solar cells, the defects are observed and analyzed, and it is found that the defects can be divided into three categories according to their morphological characteristics: linear defects, block defects, and whole-piece defects. If three models are used to perform instance segmentation on these three types of defects respectively, better results can be achieved than a single model, but the time consumption is also three times that of a single model.
[0052] To this end, the present application proposes a defect detection network, including a backbone network and multiple branch networks, wherein the backbone network is used to extract features from an object image and output the obtained feature data to multiple branch networks. The feature extraction process is shared, which is equivalent to sharing the front-layer backbone network of the three networks. Compared with performing feature extraction separately in multiple defect detection networks, it obviously saves the computing resources consumed by the feature extraction process. For example, the object image is input into the convolution layer of the convolutional neural network, and after feature extraction, a feature map, i.e., feature data, is obtained.
[0053] Defects can be divided into multiple categories, and each branch network processes defects of one category separately. For example, according to the principle of similar morphology, more than 20 types of defects in solar cells are divided into N categories. For example, two defects named "single crack" and "cross crack" are similar and can be classified into one category. Each branch network of the defect detection network processes defects of one category separately. Each branch network learns the features of several similar defects in this category, and then obtains the position, morphology, segmentation, etc. Each branch network can better detect and distinguish defects of this category.
[0054] In order to enable multiple branch networks to process defects of one category separately, it is necessary to use sample defect data of one category marked for the object image sample and train a branch network separately, that is, the object image samples marked with sample defect data of N categories correspond to N branch networks. The sample defect data includes the position, morphology, segmentation, etc. of the defects marked for the object image samples, or any other applicable data, which is not limited in the embodiments of the present application. During the training process, in each branch network, the parameters in the defect detection network are continuously adjusted according to the sample defect data of the corresponding category until the required performance is achieved, and finally a defect detection network including a backbone network and a branch network is obtained.
[0055] For the object image that needs to detect defects, after extracting the feature data, the feature data is input into multiple branch networks respectively. The multiple branch networks detect different types of defects on the object image according to the feature data and output defect detection results. The defect detection results correspond to the sample defect data marked during the training process. The defect detection results include but are not limited to the location, shape, segmentation, etc. of the defects. The training can be carried out according to actual needs, and the embodiments of the present application do not limit this.
[0056] In an optional embodiment of the present application, during the process of detecting the defect detection network, it is necessary to perform feature extraction on the object image samples, and the obtained feature data is recorded as sample feature data. The feature extraction of the object image samples is the same as the feature extraction of the object image during the detection process.
[0057] In an optional embodiment of the present application, when training the defect detection network, multiple rounds of iterations are required, and multiple branch networks output defect detection results in each round, which are recorded as sample defect detection results. The sample defect detection results of each branch network are compared with the sample defect data of the corresponding category to obtain the sample difference data of this branch network. The sample difference data is back-propagated in the neural network, and the branch network and the trunk network are corrected. After multiple rounds of iterations, when the output sample defect detection results are close enough to the marked sample defect data, the required performance is met. At this time, the corrected trunk network and branch network constitute a trained defect detection network.
[0058] In an optional embodiment of the present application, sample defects can be divided into multiple subcategories, and the multiple subcategories are divided, and the subcategories belong to parent categories. For example, there are 20 types of defects in solar cells, which refer to subcategories. According to the principle of similar morphology, the 20 subcategories of defects in solar cells are divided into N categories, which refer to parent categories. According to the parent category to which the subcategories of sample defects belong, the category corresponding to the sample defect data can be determined.
[0059] In an optional embodiment of the present application, when a convolutional neural network is usually used to detect targets in an image, the RPN (Region Proposal Network) layer will add binary supervision information based on whether it is a target, and the RCNN network (Regions with Convolutional Neural Networks) will then perform classification of specific categories. This hierarchical strategy can improve the detection accuracy. In order to affect the learning process of the backbone network, so that the feature layer can learn the characteristics of defects of various categories, and each branch network in the multiple branch networks can learn the characteristics of defects of their respective categories, the present application adds multiple layers of convolution to the feature layer before the multi-branch network to learn the classification of defect categories and add category information to the sample feature data. After adding the category information as classification supervision information, the overall accuracy of the defect detection network can be improved, and the newly added multi-layer convolution can be directly pruned during network prediction, which will not have any impact on the efficiency of the algorithm.
[0060] For example, Figure 2 The schematic diagram of the multi-branch hierarchical network structure shown in FIG. The input image is the input object picture, which is input to the backbone network for feature extraction. The backbone network outputs a feature map (i.e., feature data). At this layer, according to the parent category to which the subcategory of the sample defect belongs, category information (i.e., the category of the defect) is added to the sample feature data as supervision information, and then input to N branch networks respectively, and multiple branch networks respectively output N categories of defect detection results. In practical applications, the category information can be divided into N+1 categories, where N categories are the categories of defects and 1 category is the background of the defect in the picture.
[0061] In an optional embodiment of the present application, the sample defect data includes but is not limited to: the location area of the sample defect, the subcategory of the sample defect, and the pixel data of the sample defect. The location area is an area where the defect is located on the image, such as Figure 1 In the photo of the cell shown, the area formed by the boxes corresponding to defects such as "single crack", "cross crack", "missing corner", "black spot", "bright flake", "light and dark flake" etc. The sub-category is the specific classification of the defect, such as "single crack", "cross crack", "missing corner", "black spot", "bright flake", "light and dark flake", etc. The pixel data includes but is not limited to the pixels included in the defect itself, or the pixels at the edge of the defect, for example, the segmentation of the defect in the photo of the cell.
[0062] In an optional embodiment of the present application, the defect specification information includes the length, width, area, etc. of the defect, or any other applicable specification information, which is not limited in the present embodiment. For example, for a solar cell, the length of a "single crack" defect can be calculated based on the segmentation of the defect detected.
[0063] In an optional embodiment of the present application, the contrast information of the defect includes but is not limited to: the contrast between the defect and the surrounding area, the contrast between the defect and the entire image, etc., and the present embodiment does not limit this. For example, for a solar cell, based on the segmentation of the defect detected, the contrast between the "bright piece" defect and the surrounding area can be calculated.
[0064] In an optional embodiment of the present application, the object images include but are not limited to: images of polycrystalline silicon cells, images of polycrystalline silicon modules, images of steel plates, images of cloth, X-ray images of tires, images of electronic devices, images of food, etc., and the present embodiment of the application does not impose any restrictions on this.
[0065] According to an embodiment of the present application, in the field of industrial defect detection, when there are many defect forms of the detection object and the morphological differences between different defects are uneven, it is possible to better distinguish between defects with large differences, but it is easy to misclassify defects with similar morphological differences, resulting in low detection accuracy and low detection efficiency. Figure 3 The schematic diagram of the detection process shown in the figure shows that the present application provides a detection mechanism, which obtains feature data by inputting an object image into the main network of a defect detection network for feature extraction, and then inputting the feature data into multiple branch networks of the defect detection network. According to the feature data, multiple branch networks respectively detect different categories of defects on the object image, and output defect detection results, so that each branch of the neural network can detect a category of defects separately and specifically, avoiding the problem of low detection accuracy when the neural network learns multiple defect forms at the same time. On the other hand, it also consumes less computing resources than using multiple neural networks for detection, and improves the accuracy and efficiency of defect detection while consuming the same computing resources. The present application is applicable to but not limited to the above application scenarios.
[0066] Reference Figure 4 , shows a flow chart of a detection method embodiment according to Embodiment 1 of the present application, the method may specifically include the following steps:
[0067] Step 101, input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data.
[0068] In the embodiment of the present application, after the defect detection network is trained, defects can be detected. The object image is used as the input of the defect detection network and input into the backbone network, and the backbone network extracts features from the object image to obtain feature data.
[0069] For example, deep learning methods are used to segment industrial defects using a multi-branch network structure based on the fully convolutional network (FCN) structure to complete the automated detection of industrial defects. The image of the solar cell is input into the backbone network of the defect detection network to extract features and obtain a FeatureMap.
[0070] Step 102: input the feature data into a plurality of branch networks of the defect detection network.
[0071] In the embodiment of the present application, the feature data output by the backbone network is used as the input of multiple branch networks, that is, each branch network inputs the same feature data. For example, each category of defects corresponds to a branch network, and the feature data of the object image is input into each branch network respectively.
[0072] Step 103: According to the feature data, the multiple branch networks respectively detect different types of defects on the object image and output defect detection results.
[0073] In the embodiment of the present application, after the feature data is input into multiple branch networks, the multiple branch networks respectively detect different categories of defects on the object image, and each branch network respectively detects a category of defects on the object image and outputs defect detection results. For example, when performing instance segmentation on defects of solar cells, the output defect detection results include the location area, morphology, segmentation, etc. of the defects.
[0074] According to an embodiment of the present application, an object image is input into a backbone network of a defect detection network for feature extraction to obtain feature data, and the feature data is input into multiple branch networks of the defect detection network. According to the feature data, the multiple branch networks respectively detect different categories of defects on the object image and output defect detection results, so that each branch of the neural network can detect a category of defects separately in a targeted manner, thereby avoiding the problem of low detection accuracy when the neural network learns multiple defect forms at the same time. On the other hand, it also consumes less computing resources than using multiple neural networks for detection, and improves the accuracy and efficiency of defect detection while consuming the same computing resources.
[0075] Reference Figure 5 , shows a flow chart of a detection method embodiment according to Embodiment 2 of the present application, the method may specifically include the following steps:
[0076] Step 201: Input the object image sample into the backbone network for feature extraction to obtain sample feature data.
[0077] Step 202: adding category information to the sample feature data at the feature layer of the defect detection network according to the parent category to which the subcategory of the sample defect belongs.
[0078] Step 203: output the sample feature data to the multiple branch networks.
[0079] Step 204 , training the main network and multiple branch networks using sample defect data of different categories labeled for the object image samples to obtain the defect detection network.
[0080] In an embodiment of the present application, optionally, the main network and multiple branch networks are trained using sample defect data of different categories labeled for the object image samples to obtain an implementation method of the defect detection network may include:
[0081] Sub-step S1, comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively, to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data;
[0082] Sub-step S2: correcting corresponding branch networks and the trunk network according to the sample difference data of the plurality of branch networks.
[0083] The defect detection network needs to go through multiple rounds of iterations during the training process. At the end of each branch network, after each round of outputting the sample defect detection results, the sample defect detection results are compared with the sample defect data of the corresponding category to obtain the sample difference data. The sample difference data reflects the accuracy of the defect detection network detection. According to the sample difference data, the parameters of the branch network and the trunk network are modified until the required accuracy is achieved. Among them, when correcting the trunk network, the sample difference data of multiple branch networks can be added and averaged, and the average value can be used to correct the trunk network. Any applicable correction method can be used, and the embodiments of the present application are not limited to this.
[0084] In an embodiment of the present application, optionally, the sample defect data includes subcategories of sample defects, and before comparing the sample defect detection results output by the multiple branch networks with the corresponding sample defect data respectively to obtain the sample difference data of the multiple branch networks, it may also include: determining the category corresponding to the sample defect data according to the parent category to which the subcategory of the sample defect belongs. The sample defect data includes subcategories of sample defects, and during the training process of the defect detection network, a correspondence between the subcategories and the parent categories is set, and the category corresponding to the sample defect data can be determined according to the parent category to which the subcategory belongs.
[0085] Step 205: Input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data.
[0086] Step 206: input the feature data into a plurality of branch networks of the defect detection network.
[0087] Step 207: According to the feature data, the multiple branch networks respectively detect different types of defects on the object image and output defect detection results.
[0088] Step 208: Determine the specification information of the defect on the object image according to the position area and / or pixel data of the defect.
[0089] In the embodiment of the present application, the defect detection result includes at least one of the location area of the defect and the pixel data of the defect. According to the location area, the specification information of the defect on the object image can be calculated, for example, Figure 1 In the photo of the cell shown, the length, width, area, etc. of the boxes corresponding to defects such as "single crack", "cross crack", "missing corner", "black spot", "bright flake", "light and dark flake" can be calculated as a rough defect specification information. The specification information of the defect on the object image can also be calculated based on the pixel data. For example, based on the pixel data of "single crack", the length of the defect "single crack" can be calculated as a precise defect specification information.
[0090] According to actual needs, one or more specification information of defects on the object image can be determined, and calculations can be performed based on the location area and / or pixel data of the defects. The embodiments of the present application are not limited to this.
[0091] Step 209: Provide the specification information.
[0092] According to an embodiment of the present application, an object image sample is input into the backbone network for feature extraction to obtain sample feature data, and the sample feature data is output to the multiple branch networks. According to the parent category to which the subcategory of the sample defect belongs, category information is added to the sample feature data at the feature layer of the defect detection network. The backbone network and multiple branch networks are trained using sample defect data of different categories marked for the object image samples to obtain the defect detection network. The object image is input into the backbone network of the defect detection network for feature extraction to obtain feature data. The feature data is input into the multiple branch networks of the defect detection network. According to the feature data, the multiple branch networks respectively detect defects of different categories on the object image and output Output defect detection results, input the object image into the main network of the defect detection network for feature extraction to obtain feature data, input the feature data into multiple branch networks of the defect detection network, and according to the feature data, the multiple branch networks respectively detect different categories of defects on the object image, and output defect detection results, wherein the multiple branch networks are respectively trained with sample defect data of different categories marked for object image samples, so that each branch of the neural network can detect a category of defects separately and specifically, avoiding the problem of low detection accuracy when the neural network learns multiple defect forms at the same time, and on the other hand, it consumes less computing resources than using multiple neural networks for detection, and improves the accuracy and efficiency of defect detection while consuming the same computing resources.
[0093] Furthermore, by adding category information to the sample feature data in the feature layer of the defect detection network according to the parent category to which the subcategory of the sample defect belongs, the overall accuracy of the defect detection network can be improved after adding the category information as classification supervision information, and the newly added multi-layer convolution can be directly pruned during network prediction, which will not have any impact on the efficiency of the algorithm.
[0094] Furthermore, by determining the specification information of the defect on the object image based on the location area and / or pixel data of the defect, the specification information is provided to facilitate subsequent repair or defect grading, thereby achieving full automation of detection, saving labor costs and improving detection efficiency.
[0095] Furthermore, by determining the adjacent area of the defect on the object image based on the location area and / or pixel data of the defect, and comparing the defect with the surrounding area, comparison information is obtained, and the comparison information is provided to facilitate subsequent repairs or defect grading, thereby achieving full automation of detection, saving labor costs, and improving detection efficiency.
[0096] Reference Figure 6, shows a structural block diagram of a detection device embodiment according to Embodiment 3 of the present application, which may specifically include:
[0097] The feature extraction module 401 is used to input the object image into the backbone network of the defect detection network to extract features and obtain feature data;
[0098] A data input module 402, used to input the characteristic data into a plurality of branch networks of the defect detection network;
[0099] The detection module 403 is used to detect different types of defects on the object image by the multiple branch networks according to the feature data, and output defect detection results.
[0100] In the embodiment of the present application, optionally, the multiple branch networks are respectively trained using sample defect data of different categories labeled for object image samples.
[0101] In the embodiment of the present application, optionally, the device further includes:
[0102] A feature extraction module, configured to input a sample of the object image into the main network for feature extraction to obtain sample feature data before inputting the object image into the main network of the defect detection network for feature extraction to obtain feature data, and output the sample feature data to the multiple branch networks;
[0103] The training module is used to train the main network and multiple branch networks by using sample defect data of different categories marked for the object image samples to obtain the defect detection network.
[0104] In an embodiment of the present application, optionally, the training module includes:
[0105] A comparison submodule, used to compare the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively, to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data;
[0106] The correction submodule is used to correct the corresponding branch networks and the trunk network according to the sample difference data of the multiple branch networks.
[0107] In the embodiment of the present application, optionally, the sample defect data includes subcategories of sample defects, and the apparatus further includes:
[0108] A category determination module is used to determine the category corresponding to the sample defect data according to the parent category to which the subcategory of the sample defect belongs, before comparing the sample defect detection results output by the multiple branch networks with the corresponding sample defect data to obtain the sample difference data of the multiple branch networks.
[0109] In the embodiment of the present application, optionally, the sample defect data includes subcategories of sample defects, and the apparatus further includes:
[0110] An information adding module is used to add category information to the sample feature data at the feature layer of the defect detection network according to the parent category to which the subcategory of the sample defect belongs before outputting the sample feature data to the multiple branch networks.
[0111] In the embodiment of the present application, optionally, the sample defect data includes at least one of the following: a location area of the sample defect, a subcategory of the sample defect, and pixel data of the sample defect.
[0112] In the embodiment of the present application, optionally, the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and the device further includes:
[0113] a specification information determination module, configured to determine specification information of defects on the object image according to a location area and / or pixel data of the defects, after the plurality of branch networks respectively detect defects of different categories on the object image according to the feature data and output defect detection results;
[0114] The first providing module is used to provide the specification information.
[0115] In the embodiment of the present application, optionally, the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and the device further includes:
[0116] A region determination module, configured to determine a region adjacent to the defect on the object image according to a position region and / or pixel data of the defect, after the plurality of branch networks respectively detect defects of different categories on the object image according to the feature data and output defect detection results;
[0117] A comparison module, used to compare the defect with the surrounding area to obtain comparison information;
[0118] The second providing module is used to provide the comparison information.
[0119] In an embodiment of the present application, optionally, the object picture includes at least one of the following: a picture of a polycrystalline silicon cell, a picture of a polycrystalline silicon module, a picture of a steel plate, a picture of a cloth, an X-ray picture of a tire, a picture of an electronic device, and a picture of food.
[0120] According to an embodiment of the present application, an object image is input into a backbone network of a defect detection network for feature extraction to obtain feature data, and the feature data is input into multiple branch networks of the defect detection network. According to the feature data, the multiple branch networks respectively detect different categories of defects on the object image and output defect detection results, so that each branch of the neural network can detect a category of defects separately in a targeted manner, thereby avoiding the problem of low detection accuracy when the neural network learns multiple defect forms at the same time. On the other hand, it also consumes less computing resources than using multiple neural networks for detection, and improves the accuracy and efficiency of defect detection while consuming the same computing resources.
[0121] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0122] Embodiments of the present disclosure may be implemented as a system using any suitable hardware, firmware, software, or any combination thereof to perform the desired configuration. Figure 7 An exemplary system (or device) 500 that can be used to implement various embodiments described in the present disclosure is schematically illustrated.
[0123] For one embodiment, Figure 7 An exemplary system 500 is shown having one or more processors 502, a system control module (chip set) 504 coupled to at least one of the processor(s) 502, a system memory 506 coupled to the system control module 504, a non-volatile memory (NVM) / storage device 508 coupled to the system control module 504, one or more input / output devices 510 coupled to the system control module 504, and a network interface 512 coupled to the system control module 504.
[0124] Processor 502 may include one or more single-core or multi-core processors, and processor 502 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.) In some embodiments, system 500 can be used as a browser as described in the embodiments of the present application.
[0125] In some embodiments, the system 500 may include one or more computer-readable media (e.g., system memory 506 or NVM / storage device 508) having instructions and one or more processors 502 that are combined with the one or more computer-readable media and configured to execute the instructions to implement the modules to perform the actions described in the present disclosure.
[0126] For one embodiment, system control module 504 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 502 and / or any suitable device or component in communication with system control module 504 .
[0127] The system control module 504 may include a memory controller module to provide an interface to the system memory 506. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0128] The system memory 506 may be used, for example, to load and store data and / or instructions for the system 500. For one embodiment, the system memory 506 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 506 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0129] For one embodiment, the system control module 504 may include one or more input / output controllers to provide interfaces to the NVM / storage device 508 and the input / output device(s) 510 .
[0130] For example, NVM / storage 508 may be used to store data and / or instructions. NVM / storage 508 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0131] NVM / storage device 508 may include storage resources that are physically part of the device on which system 500 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 508 may be accessed via input / output device(s) 510 over a network.
[0132] The input / output device(s) 510 may provide an interface for the system 500 to communicate with any other appropriate device, and the input / output device 510 may include a communication component, an audio component, a sensor component, etc. The network interface 512 may provide an interface for the system 500 to communicate through one or more networks, and the system 500 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof for wireless communication.
[0133] For one embodiment, at least one of the processor(s) 502 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the system control module 504. For one embodiment, at least one of the processor(s) 502 may be packaged together with the logic of one or more controllers of the system control module 504 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 502 may be integrated on the same die with the logic of one or more controllers of the system control module 504. For one embodiment, at least one of the processor(s) 502 may be integrated on the same die with the logic of one or more controllers of the system control module 504 to form a system on chip (SoC).
[0134] In various embodiments, the system 500 may be, but is not limited to: a browser, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 500 may have more or fewer components and / or a different architecture. For example, in some embodiments, the system 500 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.
[0135] Wherein, if the display includes a touch panel, the display screen can be implemented as a touch screen display to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0136] An embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a terminal device, the terminal device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0137] In one example, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as in an embodiment of the present application when executing the computer program.
[0138] In one example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, one or more methods of the embodiments of the present application are implemented.
[0139] The embodiments of the present application disclose a detection method and device. Example 1 includes a detection method, including:
[0140] Input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data;
[0141] Inputting the characteristic data into a plurality of branch networks of the defect detection network;
[0142] According to the feature data, the multiple branch networks respectively detect different categories of defects on the object image and output defect detection results.
[0143] Example 2 may include the method described in Example 1, wherein the multiple branch networks are respectively trained using sample defect data of different categories labeled for object image samples.
[0144] Example 3 may include the method described in Example 1 and / or Example 2, wherein, before inputting the object image into the backbone network of the defect detection network for feature extraction to obtain feature data, the method further includes:
[0145] Inputting an object image sample into the main network for feature extraction to obtain sample feature data, and outputting the sample feature data to the multiple branch networks;
[0146] The main network and multiple branch networks are trained using sample defect data of different categories labeled for the object image samples to obtain the defect detection network.
[0147] Example 4 may include the method described in one or more of Examples 1 to 3, wherein the use of sample defect data of different categories labeled for the object image samples to train the trunk network and the plurality of branch networks to obtain the defect detection network includes:
[0148] Comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data;
[0149] According to the sample difference data of the plurality of branch networks, the corresponding branch networks and the trunk network are corrected.
[0150] Example 5 may include the method described in one or more of Examples 1 to 4, wherein the sample defect data includes subcategories of sample defects, and before comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data to obtain sample difference data of the plurality of branch networks, the method further includes:
[0151] The category corresponding to the sample defect data is determined according to the parent category to which the subcategory of the sample defect belongs.
[0152] Example 6 may include the method of one or more of Examples 1 to 5, wherein the sample defect data includes subcategories of sample defects, and before outputting the sample feature data to the plurality of branch networks, the method further includes:
[0153] According to the parent category to which the subcategory of the sample defect belongs, category information is added to the sample feature data at the feature layer of the defect detection network.
[0154] Example 7 may include the method of one or more of Examples 1 to 6, wherein the sample defect data includes at least one of the following: a location area of the sample defect, a subcategory of the sample defect, and pixel data of the sample defect.
[0155] Example 8 may include the method described in one or more of Examples 1 to 7, wherein the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and after the plurality of branch networks respectively detect different categories of defects on the object image according to the feature data and output the defect detection result, the method further includes:
[0156] Determining specification information of the defect on the object image according to the location area and / or pixel data of the defect;
[0157] Provides the stated specification information.
[0158] Example 9 may include the method described in one or more of Examples 1 to 8, wherein the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and after the plurality of branch networks respectively detect different categories of defects on the object image according to the feature data and output the defect detection result, the method further includes:
[0159] Determine an area adjacent to the defect on the object image according to the location area and / or pixel data of the defect;
[0160] comparing the defect with a surrounding area to obtain comparison information;
[0161] The comparative information is provided.
[0162] Example 10 may include one or more of the methods described in Examples 1-9, wherein the object image includes at least one of the following: an image of a polycrystalline silicon cell, an image of a polycrystalline silicon module, an image of a steel plate, an image of a cloth, an X-ray image of a tire, an image of an electronic device, and an image of food.
[0163] Example 11 includes a detection device, comprising:
[0164] A feature extraction module, which is used to input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data;
[0165] A data input module, used for inputting the characteristic data into a plurality of branch networks of the defect detection network;
[0166] A detection module is used to detect different types of defects on the object image by the multiple branch networks according to the feature data, and output defect detection results.
[0167] Example 12 may include the apparatus described in Example 11, wherein the plurality of branch networks are respectively trained using sample defect data of different categories labeled for object image samples.
[0168] Example 13 may include the apparatus of Example 11 and / or Example 12, wherein the apparatus further comprises:
[0169] A feature extraction module, configured to input a sample of the object image into the main network for feature extraction to obtain sample feature data before inputting the object image into the main network of the defect detection network for feature extraction to obtain feature data, and output the sample feature data to the multiple branch networks;
[0170] The training module is used to train the main network and multiple branch networks by using sample defect data of different categories marked for the object image samples to obtain the defect detection network.
[0171] Example 14 may include the apparatus of Example 11 and / or Example 13, wherein the training module comprises:
[0172] A comparison submodule, used to compare the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively, to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data;
[0173] The correction submodule is used to correct the corresponding branch networks and the trunk network according to the sample difference data of the multiple branch networks.
[0174] Example 15 may include the apparatus of Example 11 and / or Example 14, wherein the sample defect data includes a subcategory of sample defects, and the apparatus further comprises:
[0175] A category determination module is used to determine the category corresponding to the sample defect data according to the parent category to which the subcategory of the sample defect belongs, before comparing the sample defect detection results output by the multiple branch networks with the corresponding sample defect data to obtain the sample difference data of the multiple branch networks.
[0176] Example 16 may include the apparatus of Example 11 and / or Example 15, wherein the sample defect data includes a subcategory of sample defects, and the apparatus further comprises:
[0177] An information adding module is used to add category information to the sample feature data at the feature layer of the defect detection network according to the parent category to which the subcategory of the sample defect belongs before outputting the sample feature data to the multiple branch networks.
[0178] Example 17 may include the apparatus of Example 11 and / or Example 16, wherein the sample defect data includes at least one of the following: a location area of the sample defect, a subcategory of the sample defect, and pixel data of the sample defect.
[0179] Example 18 may include the apparatus of Example 11 and / or Example 17, wherein the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and the apparatus further includes:
[0180] a specification information determination module, configured to determine specification information of defects on the object image according to a location area and / or pixel data of the defects, after the plurality of branch networks respectively detect defects of different categories on the object image according to the feature data and output defect detection results;
[0181] The first providing module is used to provide the specification information.
[0182] Example 19 may include the apparatus of Example 11 and / or Example 18, wherein the defect detection result includes at least one of a location area of the defect and pixel data of the defect, and the apparatus further includes:
[0183] A region determination module, configured to determine a region adjacent to the defect on the object image according to a position region and / or pixel data of the defect, after the plurality of branch networks respectively detect defects of different categories on the object image according to the feature data and output defect detection results;
[0184] A comparison module, used to compare the defect with the surrounding area to obtain comparison information;
[0185] The second providing module is used to provide the comparison information.
[0186] Example 20 may include the apparatus described in Example 11 and / or Example 19, wherein the object image includes at least one of the following: an image of a polycrystalline silicon cell, an image of a polycrystalline silicon module, an image of a steel plate, an image of a cloth, an X-ray image of a tire, an image of an electronic device, and an image of food.
[0187] Example 21 includes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, one or more methods of Examples 1-10 are implemented.
[0188] Example 22 includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements one or more methods of Examples 1-10.
[0189] Although certain embodiments are for the purpose of illustration and description, various alternatives, and / or, equivalent implementations, or calculations to achieve the same purpose of the embodiment shown and described implementations do not depart from the scope of the present application. This application is intended to cover any modifications or changes of the embodiments discussed herein. Therefore, it is apparent that the embodiments described herein are limited only by the claims and their equivalents.
Claims
1. A detection method, It is characterized in that include: Input the object image into the backbone network of the defect detection network for feature extraction to obtain feature data; Inputting the characteristic data into a plurality of branch networks of the defect detection network; Each of the branch networks processes a category of defects independently, and the category of the defects is determined based on the principle of similar morphology; According to the feature data, the multiple branch networks respectively detect different types of defects on the object image and output defect detection results; the defect detection results include at least one of the location area of the defect and the pixel data of the defect; The branch network of the defect detection network is obtained by training sample defect data, and the sample defect data includes subcategories of sample defects; According to the parent category to which the subcategory of the sample defect belongs, category information is added to the sample feature data at the feature layer of the defect detection network.
2. The method according to claim 1, It is characterized in that The multiple branch networks are respectively trained using sample defect data of different categories labeled for object image samples.
3. The method according to claim 2, It is characterized in that Before inputting the object image into the backbone network of the defect detection network to extract features and obtain feature data, the method further includes: Inputting the object image sample into the main network to extract features, obtaining sample feature data, and outputting the sample feature data to the multiple branch networks; The main network and multiple branch networks are trained using sample defect data of different categories labeled for the object image samples to obtain the defect detection network.
4. The method according to claim 3, It is characterized in that The method of training the main network and the plurality of branch networks by using sample defect data of different categories marked for the object image samples to obtain the defect detection network comprises: Comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data respectively to obtain sample difference data of the plurality of branch networks, wherein one branch network corresponds to one category of sample defect data; According to the sample difference data of the plurality of branch networks, the corresponding branch networks and the trunk network are corrected.
5. The method according to claim 4, It is characterized in that The sample defect data includes subcategories of sample defects. Before comparing the sample defect detection results output by the plurality of branch networks with the corresponding sample defect data to obtain sample difference data of the plurality of branch networks, the method further includes: The category corresponding to the sample defect data is determined according to the parent category to which the subcategory of the sample defect belongs.
6. The method according to claim 2, It is characterized in that The sample defect data includes at least one of the following: a location area of the sample defect, a subcategory of the sample defect, and pixel data of the sample defect.
7. The method according to claim 1, It is characterized in that After the plurality of branch networks respectively detect different types of defects on the object image according to the feature data and output defect detection results, the method further includes: Determining specification information of the defect on the object image according to the location area and / or pixel data of the defect; Provides the stated specification information.
8. The method according to claim 1, It is characterized in that The defect detection result includes at least one of a location area of the defect and pixel data of the defect. After the plurality of branch networks respectively detect defects of different categories on the object image according to the feature data and output the defect detection result, the method further includes: Determine an area adjacent to the defect on the object image according to the location area and / or pixel data of the defect; comparing the defect with a surrounding area to obtain comparison information; The comparative information is provided.
9. The method according to claim 1, It is characterized in that The object pictures include at least one of the following: pictures of polycrystalline silicon cells, pictures of polycrystalline silicon modules, pictures of steel plates, pictures of cloth, X-ray pictures of tires, pictures of electronic equipment, and pictures of food.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to one or more of claims 1-9 is implemented.
11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to one or more of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Front vehicle parameter identification method based on multitask convolution nerve network
CN105975915A
Convolutional neural network-based vehicle attribute identification method and apparatus
CN106778583A
Product defect detection method, apparatus and system, server and storage medium
CN108564104A
Aerial photography road vehicle count method and device based on deep learning
CN108710875A
Data prediction method and device based on neural network and neural network system
CN109190646A
