Industrial part surface defect detection method and device

Abstract

The invention provides an industrial part surface defect detection method and device. The detection method comprises the following steps: acquiring image data of a to-be-detected industrial part; the image data is input into an optimized and improved Faster R-CNN model, the Faster R-CNN model is obtained after iterative training in a training stage and verification in a verification stage, the Faster R-CNN model comprises a Res2Net and FPN feature extraction network, a region candidate network and a GRoIE network, and a high-granularity image convolution calculation module is adopted in the Res2Net network; and determining the position and the defect type of the surface defect of the industrial part in the image data according to the output of the Faster R-CNN model so as to carry out detection work.

Description

Technical Field

[0001] The present invention relates to the field of parts detection, and in particular to a method and device for detecting surface defects of industrial parts. Background Art

[0002] In various industrial production processes, countless parts are involved. Inevitably, these parts may be affected by environmental corrosion, machine errors, human errors and other factors, resulting in surface defects such as scratches, corrosion, and cracks. These defects not only affect the appearance, but also cause the failure of key parts of industrial machines, resulting in catastrophic failures and losses, and bring hidden dangers to industrial production. Therefore, in industrial production, surface defect detection of industrial parts is an important part of parts quality control.

[0003] At present, surface defect detection is mainly divided into manual detection and automated detection according to the detection method. Manual detection has problems such as large workload, high cost, and unstable detection results, and is not suitable for large-scale industrial parts detection. Most of the automated detection is based on microwave, vibroacoustic modulation, light, magnetoelectric, wireless sensing and other technologies, and has problems such as small detection range and low detection accuracy, which cannot meet the needs of high-precision part surface defect detection.

[0004] Some existing solutions use machine vision to identify and detect images. These methods have the advantage of high detection accuracy and can detect local information in images more accurately. In recent years, they have been widely used in various industrial fields such as steel plates, printing, wood, rails, and textiles, saving a lot of labor costs.

[0005] However, for small defects and irregularly shaped defects of objects in industrial images, such as surface defects of industrial parts, the machine vision recognition methods in the prior art still have the problem of low detection accuracy.

[0006] In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of a method and device for detecting surface defects of industrial parts, which can be used to more accurately detect various defects on the surface of industrial parts, especially to accurately detect and identify tiny defects and defects of irregular shapes, accurately output the location and category of defects, improve the accuracy and reliability of industrial parts surface defect detection, thereby reducing the safety hazards caused by parts defects and providing a solid guarantee for the efficient and stable operation of the production line. Summary of the invention

[0007] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0008] In order to overcome the above-mentioned defects in the prior art, the present invention provides a method for detecting surface defects of industrial parts, which is used to detect flaws on the surface of industrial parts, and the flaws include corrosion, scratches, scratches and built-up edge. The detection method comprises: obtaining image data of industrial parts to be detected, and the flaws exist on the surface of some parts of the industrial parts to be detected; inputting the image data into an optimized and improved Faster R-CNN model, which is obtained after iterative training in a training phase and verification in a verification phase, and the Faster R-CNN model includes Res2Net and FPN feature extraction networks, a region candidate network and a GRoIE network, and a multi-scale backbone architecture is adopted in the Res2Net network; and determining the position and defect type of the surface defects of industrial parts in the image data according to the output of the Faster R-CNN model for detection.

[0009] In one embodiment, preferably, a multi-scale backbone architecture is adopted in the Res2Net network, which may include: in the Res2Net network, the bottleneck block in each convolution module is composed of a 1×1 convolution, a 3×3 convolution and a 1×1 convolution, wherein after the first 1×1 convolution, the feature map is divided into s scale dimensions, and the 3×3 convolution is performed according to the following formula:

[0010]

[0011] Among them, K represents a 3×3 convolution kernel, x represents the input feature map, y represents the output feature map, i represents the scale dimension of the feature map, and s represents the total number of scale dimensions into which the feature map is divided.

[0012] In one embodiment, preferably, the region candidate network adopts a multi-level cascade, each level includes an adaptive convolution module that dynamically generates an adaptive convolution kernel according to the detection target, and generates a candidate suggestion box based on the convolution calculation. The adaptive convolution module and the candidate suggestion box of the previous level are respectively used to bridge the image features of the next level and provide a regression bounding box.

[0013] In one embodiment, optionally, the region candidate network adopts a two-stage cascade.

[0014] In one embodiment, preferably, the image processing of the GRoIE network may include the following steps: performing maximum pooling of regions of interest of different sizes in the image through a region of interest pooler to obtain an image representation of uniform size; performing preliminary processing on the feature map to be aggregated so that each image feature is semantically associated with adjacent features; merging and aggregating a single RoI from each branch to fuse feature information of different receptive fields; and setting a convolution layer, a non-local operation layer and an attention layer to fuse global image features and focus on the feature information of the surface defects of the industrial part.

[0015] In one embodiment, preferably, the acquiring of the image data of the industrial part to be inspected may include: after acquiring the image data of the industrial part to be inspected, performing data enhancement preprocessing on the image data.

[0016] In one embodiment, preferably, the training phase of the Faster R-CNN model may include: obtaining an industrial part image data sample whose surface contains the defect, annotating and preprocessing the data sample to obtain a part surface defect data set; dividing the part surface defect data set into a training set and a validation set; and iteratively optimizing the Faster R-CNN model using the training set; the verification phase of the Faster R-CNN model may include: selecting a model evaluation index, and using the validation set to verify the performance of the Faster R-CNN model obtained in the training phase, so as to optimize the Faster R-CNN model for part defect detection.

[0017] In one embodiment, preferably, dividing the part surface defect dataset into a training set and a validation set may include: shuffling the preprocessed part surface defect dataset, splitting the part surface defect dataset using a three-fold cross-validation method to form three independent subsets, selecting two of the subsets as the training set each time and the remaining one as the validation set, thereby performing training and validation of the Faster R-CNN model.

[0018] Another aspect of the present invention also provides a device for detecting surface defects of industrial parts, the device comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the method for detecting surface defects of industrial parts as described in any one of the above items.

[0019] The present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for detecting surface defects of industrial parts as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0021] Figure 1 is a schematic diagram of a method flow chart of a method for detecting surface defects of industrial parts according to one aspect of the present invention;

[0022] Figure 2 is a schematic diagram of the structure of the Faster R-CNN model used in the present invention according to an embodiment of the present invention;

[0023] Figure 3A is a schematic diagram of the structure of the Res2Net and FPN feature extraction network in the Faster R-CNN model adopted by the present invention according to an embodiment of the present invention;

[0024] Figure 3B is a schematic diagram of the structure of the bottleneck block of the Res2Net network in the Faster R-CNN model adopted by the present invention according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of the structure of a two-stage cascaded region candidate network in the Faster R-CNN model adopted by the present invention according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of the structure of the GRoIE network in the Faster R-CNN model adopted by the present invention according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of the Faster R-CNN model training and use process according to an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of the Faster R-CNN model training and testing process according to an embodiment of the present invention;

[0029] Figure 8 to Figure 13 is a photograph of a comparison of the effects of surface defect prediction of industrial parts according to an embodiment of the present invention; and

[0030] Fig.14 The figure is a schematic diagram of the structure of a device for detecting surface defects of industrial parts according to another aspect of the present invention. DETAILED DESCRIPTION

[0031] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0032] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0034] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0035] In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a method and device for detecting surface defects of industrial parts, which are used to more accurately detect various defects on the surface of industrial parts, especially to accurately detect and identify tiny defects and defects with irregular shapes, accurately output the location and category of the defects, and improve the accuracy and reliability of industrial parts surface defect detection, thereby reducing the safety hazards caused by parts defects and providing a solid guarantee for the efficient and stable operation of the production line.

[0036] Figure 1The present invention is a flowchart of a method for detecting surface defects of industrial parts according to one aspect of the present invention.

[0037] Please refer to Figure 1 The method for detecting surface defects of industrial parts provided by the present invention is used to detect surface defects of industrial parts, wherein the surface defects include corrosion, scratches, scuffs and built-up edge. The detection method 100 may include:

[0038] Step 101: Acquire image data of an industrial part to be inspected, where the defect exists on the surface of some parts of the industrial part to be inspected;

[0039] Step 102: input the image data into an optimized and improved Faster R-CNN model, which is obtained after iterative training in a training phase and verification in a verification phase. The Faster R-CNN model includes a Res2Net and FPN feature extraction network, a region candidate network, and a GRoIE network. The Res2Net network adopts a multi-scale backbone architecture; and

[0040] Step 103: Determine the location and defect type of the surface defect of the industrial part in the image data according to the output of the Faster R-CNN model to perform detection.

[0041] In a preferred embodiment, in step 101 of the detection method 100 provided by the present invention, the acquisition of image data of the industrial part to be detected may further include: after acquiring the image data of the industrial part to be detected, the image data is preprocessed for data enhancement. For example, the acquired original image of the part may be flipped, scaled, saturation enhanced and weakened, etc. After preprocessing, the image data is input into the optimized and improved Faster R-CNN model of the present invention for part defect detection.

[0042] The present invention adopts an improved and optimized Faster R-CNN model to detect surface defects of industrial parts. The FasterR-CNN model includes Res2Net and FPN feature extraction networks, a region candidate network and a GRoIE network. Among them, Res2Net and FPN feature extraction networks are used to obtain feature information of part images; the region candidate network is used to obtain the position of surface defects of industrial parts in two-dimensional images for the next stage of screening; the GRoIE network is used to integrate the feature information extracted by the feature extraction module, so that the network pays more attention to defect feature information.

[0043] Figure 2 is a schematic diagram of the Faster R-CNN model structure adopted by the present invention, illustrated according to an embodiment of the present invention.

[0044] like Figure 2As shown, in Figure 2 In the feature extraction network part on the left, Res2Net is used for the backbone network. In particular, the present invention adopts a multi-scale backbone architecture to increase the receptive field, so that the feature information of part defects can be better captured, and the extracted feature information can be more detailed, thereby improving the detection accuracy. In the cascaded region candidate network (RPN), the cascade setting makes the suggestion box given by the network more accurate. The position of the suggestion box is gradually refined through multi-level RPN, so that the suggestion box is more closely aligned with the defect feature position. The GRoIE (Generic RoI Extraction) network is used to extract the features of the region of interest, extract the key feature information of all layers of the FPN network in the feature extraction network, and effectively utilize all the extracted defect feature information, so that the extracted information can be effectively utilized in the final output result. Finally, the Faster R-CNN model provided by the present invention classifies defects and regresses bounding boxes through a fully connected layer to achieve defect detection.

[0045] The optimized and improved Faster R-CNN model network adopted in the present invention has a more accurate detection effect on small defects and irregularly shaped defects, thereby improving the overall performance of the network and achieving high-precision detection of surface defects of industrial metal parts.

[0046] In a preferred embodiment, in the optimized and improved Faster R-CNN model network adopted by the present invention, a high-granularity image convolution calculation module is adopted in the Res2Net network, which is specifically described below in conjunction with FIG. 3 .

[0047] Figure 3A is a structural diagram of the Res2Net and FPN feature extraction network in the Faster R-CNN model adopted by the present invention, illustrated according to an embodiment of the present invention.

[0048] The method for detecting surface defects of industrial parts provided by the present invention uses the Res2Net and FPN feature extraction networks in the Faster R-CNN model to extract features of small defects on the surface of precision parts. Figure 3A As shown in the figure, Res2Net is a residual network. The Res2Net network is improved from the ResNet network and contains a total of 4 large convolutional layers. Each large convolutional layer consists of a different number of basic blocks (Basic Block) and bottleneck blocks (Bottleneck Block). Figure 3A The structure of one of the convolutional layers C2 is expanded, and it can be seen that it contains a basic block and two bottlenecks.

[0049] In the industrial parts surface defect detection method provided by the present invention, the bottleneck block of the backbone architecture in the Res2Net network is improved, and a multi-scale backbone architecture is adopted to form a more detailed residual structure, so that richer feature information can be extracted and the accuracy of small target defect detection can be improved. Figure 3B .

[0050] Figure 3B is a schematic diagram of the structure of the Res2Net network bottleneck block in the Faster R-CNN model adopted by the present invention, illustrated according to an embodiment of the present invention.

[0051] like Figure 3B As shown on the left, Figure 3A The bottleneck block in consists of a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution. Figure 3B The 3×3 convolution calculation module is further expanded on the right side, replacing the original 3×3 convolution with a multi-scale convolution structure, giving it a higher granularity and a more detailed receptive field, thereby being able to extract richer feature information and improve detection accuracy.

[0052] In a preferred embodiment, in the industrial parts surface defect detection method provided by the present invention, the Res2Net network adopts a multi-scale backbone architecture, which may include: in the Res2Net network, the bottleneck block in each layer of the convolution module consists of a 1×1 convolution, a 3×3 convolution and a 1×1 convolution, wherein after the first 1×1 convolution, the feature map is divided into s scale dimensions, and the 3×3 convolution is performed according to the following formula:

[0053]

[0054] Among them, K represents a 3×3 convolution kernel, x represents the input feature map, y represents the output feature map, i represents the scale dimension of the feature map, and s represents the total number of scale dimensions into which the feature is divided.

[0055] For example, Figure 3B As shown in the figure, in the bottleneck block of the Res2Net network backbone architecture, the feature map is divided into four scale dimensions after a 1×1 convolution. The first part x1 is directly passed to y1 without any processing; the second part x2 is divided into two branches after a 3×3 convolution, one branch continues to pass forward to y2, and the other branch is passed to x3. The third part x3 and the second branch of x2 are fused and passed through a 3×3 convolution, one branch is passed to y3, and the other branch is passed to x4. The fourth part x4 and the second branch of x3 are fused and passed to y4 through a 3×3 convolution.

[0056] Continue to refer Figure 3AIn the process of downsampling the C2, C3, C4 and C5 large convolutional layers of the Res2Net network, although the semantic information increases layer by layer, the image resolution decreases layer by layer, making it difficult to retain the underlying detail information, which is not conducive to small target defect detection. In order to avoid this problem, the FPN structure is added. It will fuse the high-resolution image of the bottom layer of Res2Net with the high-semantic information features obtained by upsampling through a horizontal connection according to the different receptive fields, generate a feature map with high semantic information and high resolution, and generate multiple feature maps of different scales in a similar way, which can not only retain the detail information, but also improve the expression ability of the semantic information, so as to better meet the detection needs of small target defects such as surface defects of industrial parts.

[0057] The following is a detailed description of the region candidate network used for part surface defect detection in the present invention.

[0058] In a preferred embodiment, in the industrial parts surface defect detection method provided by the present invention, the region candidate network adopts a multi-level cascade, each level includes an adaptive convolution module that dynamically generates an adaptive convolution kernel according to the detection target, and generates a candidate suggestion box based on the convolution calculation. The adaptive convolution module of the previous level and the candidate suggestion box are respectively used to bridge the image features of the next level and provide a regression bounding box.

[0059] The multi-level cascaded region proposal network (CRPN, Cascade RPN) adopts a multi-level cascaded RPN method to refine the proposed box step by step, which can give more accurate proposed candidate boxes for irregularly shaped defects. Each level of RPN contains an adaptive convolution module that can more accurately adapt to the size and shape of the target, making the predicted box more aligned with the target, which will help the network better grasp the specific location and shape of the defect.

[0060] Further, in one embodiment, optionally, the region candidate network in the present invention adopts a two-level cascade. Figure 4 . Figure 4 1 is a schematic diagram of the structure of a two-stage cascaded region candidate network in the Faster R-CNN model adopted by the present invention, illustrated according to an embodiment of the present invention.

[0061] like Figure 4As shown in the figure, the two-stage CRPN network structure contains two RPNs. The first RPN obtains feature information from the feature extraction network and preliminarily generates a set of proposal boxes. The preliminarily generated proposal boxes may include many false detections and overlapping areas. The second-stage RPN performs more detailed screening on these preliminarily generated proposal boxes, gradually improving the accuracy of the proposal boxes, thereby improving the detection capability of the entire network. In each RPN, an adaptive convolution module is included. The adaptive convolution module can dynamically generate an adaptive convolution kernel size according to the size and shape of the input feature map, thereby generating a more flexible proposal box, aligning the anchor box with the defect feature, so as to more accurately determine the defect position and shape of the surface of industrial parts. Compared with the single-stage RPN network, it can make up for the disadvantage of inaccurate detection when processing irregularly shaped and sized part defects by only performing a single proposal box generation, and can better adapt to various irregularly shaped defects, thereby improving the detection accuracy.

[0062] Compared with the RoI pooling (RoIPooling) adopted by the Faster R-CNN model in the prior art, the industrial parts surface defect detection method provided by the present invention adopts the GRoIE network to specifically utilize the multi-scale feature information in the previous network of the model, and effectively mine and utilize the multi-scale feature map information in Res2Net and FPN, thereby improving the performance of model detection in various complex environments and conditions where industrial parts defects are located.

[0063] The GRoIE structure uses non-local building blocks, which can comprehensively consider the feature information of all scales output by the feature extraction network when processing each region of interest (RoI) of the part image, so that the network can have richer information reference when processing defects of various scales and types, thereby improving the detection accuracy. In addition, the GRoIE network can pre-process and aggregate the regions of interest (RoI) of each scale separately, so that the network can learn and utilize the feature information of each scale at a deeper level, improving the richness of feature expression. At the same time, by introducing the attention mechanism, the network can automatically learn and extract the key information of defect features, exclude irrelevant or redundant information, and make the final feature expression more accurate and effective.

[0064] In a preferred embodiment, the industrial parts surface defect detection method provided by the present invention, the image processing of the GRoIE network includes the following four steps, so as to systematically process and fuse multi-scale feature information, which can be combined with reference Figure 5 .

[0065] Figure 5 is a structural diagram of a GRoIE network in a Faster R-CNN model adopted by the present invention, illustrated according to an embodiment of the present invention.

[0066] like Figure 5 As shown, in the GRoIE network adopted by the present invention, in the first stage, the regions of interest of different sizes in the feature map are pooled by the region of interest pooler to obtain a representation of uniform size; in the second stage, the feature map to be aggregated is preliminarily processed by the preprocessing module, so that each feature is semantically associated with the adjacent features; in the third stage, the single RoI from each branch is merged and aggregated by the aggregation module to achieve the fusion of feature information of different receptive fields; in the fourth stage, the convolution layer, the non-local layer, and the attention layer ( Figure 5 (not shown in the figure), taking all scales into account, enabling the network to learn and fuse global image features, enhancing the RoI information integration capability. At the same time, attention is paid to the defect feature information on the surface of industrial parts, eliminating useless feature information, so that the combined feature information finally returned is more accurate.

[0067] In a preferred embodiment, the industrial part surface defect detection method provided by the present invention, the training stage of the Faster R-CNN model may include: obtaining an industrial part image data sample whose surface contains the defect, annotating and preprocessing the data sample to obtain a part surface defect data set; dividing the part surface defect data set into a training set and a validation set; and using the training set to iteratively optimize the Faster R-CNN model.

[0068] The verification phase of the Faster R-CNN model may include: selecting a model evaluation index, using the verification set to verify the performance of the Faster R-CNN model obtained in the training phase, so as to optimize the Faster R-CNN model for part defect detection.

[0069] Figure 6 FIG. 4 is a schematic diagram of the Faster R-CNN model training and use process according to an embodiment of the present invention.

[0070] In this embodiment, in the industrial parts surface defect detection method provided by the present invention, the Faster R-CNN model training and use may include the following steps:

[0071] Step 601: In the process of detecting surface defects of parts, obtain industrial part surface defect data samples. For example, impeller part surface defect images can be collected from the production line of an industrial parts manufacturing plant. These images contain four common industrial surface defects: corrosion, scratches, scuffs, and built-up edge. These defects are small in size and irregular in shape, requiring high-precision detection. In this embodiment, a Sony A6400 camera is used to capture the video stream of each impeller part.

[0072] Step 602: Preprocess the collected samples of part surface defects to remove blurred and significantly unusable samples, annotate the defect locations and categories according to the acquired samples and part defect standards, and perform data enhancement on the data to increase the richness of the data samples.

[0073] For example, clear and suitable part images for model training are selected from the images taken by the camera. In this embodiment, 1478 images with a resolution of 1920×1080 are collected. These images can be annotated by senior engineers on the industrial production line using the LabelImg image annotation tool, including real bounding boxes and label categories. In the image collection process of this embodiment, since the parts have been photographed from different angles and lighting conditions, no enhancement is performed on this data set.

[0074] In another embodiment, the aluminum profile surface defect public data set provided by Ali Tianchi is used as the detection object. Since the number of pictures of each type in the original data set is relatively small, the data enhancement method is used to flip, scale, enhance and weaken the original pictures, so as to enrich the data set. After data enhancement, a total of 24,559 pictures are obtained. It is easy to understand that the detection effect of the part defect detection method proposed in the present invention can be further verified by using different data samples.

[0075] Step 603: divide the data set into a training set and a validation set to facilitate model training and effect verification.

[0076] For example, in a preferred embodiment, the part surface defect data set is divided into a training set and a validation set, which may include: shuffling the preprocessed part surface defect data set, dividing the part surface defect data set into three independent subsets by a three-fold cross validation method, selecting two of the subsets as the training set each time, and the remaining one as the validation set, so as to train and validate the Faster R-CNN model. This can not only maximize the use of experimental data, but also effectively avoid the overfitting problem.

[0077] Step 604: Determine the model evaluation index to facilitate the evaluation of the model effect.

[0078] For example, in one embodiment, the mean average precision (mAP) of three trainings is calculated and its average value is used as the final performance evaluation index of the model to more accurately evaluate the model performance. Since the detection of the defective area of ​​the part is determined by the overlap criterion of the true boundary Garea and the candidate boundary Carea, the overlap criterion is the intersection union (IOU). According to the COCO evaluation index, the acceptable threshold of IOU in this embodiment is 0.5, where the formula of IOU is:

[0079]

[0080] Among them, G area is the true boundary, C area is the candidate boundary and IOU is the intersection union.

[0081] The performance evaluation index uses the average precision AP (Average Precision), which mainly includes two values: precision and recall. The precision represents the proportion of samples that actually belong to the category in the samples predicted to be positive in different categories. The formula is:

[0082]

[0083] Among them, TP (True Positive) represents the number of samples that correctly identify the surface defect targets of industrial parts, FN (False Negative) represents the number of samples that do not identify the surface defect targets of industrial parts, and FP (False Positive) represents the number of samples that incorrectly identify the defect targets of parts.

[0084] The recall rate represents the ratio of the number of correctly detected targets to the total number of targets in the test set. The formula for the recall rate is:

[0085]

[0086] Among them, the total number of TP+FN defective samples;

[0087] The formula for average accuracy AP is:

[0088]

[0089] The model is evaluated by mAP, which represents the mean of the average accuracy of all categories in the model. The formula of mAP is:

[0090]

[0091] After the model evaluation index is determined, step 605 is then executed: the image feature information is obtained using the Res2Net network. In the present invention, the Res2Net network is improved and optimized, and a multi-scale backbone architecture is adopted in it, so that the network can obtain detailed feature information of the image, which is convenient for subsequent processing of the feature information.

[0092] Step 606: Use the improved multi-level cascaded region proposal network to obtain the part defect location. Use the cascaded RPN network to obtain a more accurate proposal box.

[0093] Step 607: Use the GRoIE network to extract features of the region of interest, integrate the collected image defect features more comprehensively, fuse the global features, increase the network's attention to the defect features, and ignore useless information.

[0094] Based on the above steps, the optimized and improved Faster R-CNN model in the present invention is trained, verified and iteratively optimized using data sets and model evaluation indicators.

[0095] Finally, step 608 is performed: using the optimized Faster R-CNN target detection network model to detect surface defects of industrial parts.

[0096] Figure 7 4 is a schematic diagram of the Faster R-CNN model training and testing process according to an embodiment of the present invention. Figure 7 The process of training, validating, and iteratively optimizing the Faster R-CNN model is further demonstrated.

[0097] The method for detecting surface defects of industrial parts provided by the present invention adopts an optimized and improved Faster R-CNN model, adopts a multi-scale backbone structure to improve the feature extraction network according to the characteristics of small size of part defects, sets a multi-level cascaded region candidate network according to the irregular shape characteristics of part defects, and adopts an adaptive adjustment suggestion box to meet the high-precision requirements of industrial part surface defect detection tasks, making it more suitable for small defect and defect detection tasks of different shapes and sizes, thereby achieving the technical effect of improving the accuracy and reliability of the target detection network for part surface defect detection.

[0098] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0099] The actual effect of the method for detecting surface defects of industrial parts provided by the present invention is demonstrated below with reference to examples.

[0100] Figure 8 to Figure 13 The figure is a photograph showing a comparison of the effects of surface defect prediction of industrial parts according to an embodiment of the present invention.

[0101] exist Figure 8 to Figure 13In the figure, the defect detection results of multiple photos of actual parts are shown, where the blue box is the manually annotated defect target box, the yellow box is the part defect anchor box output by image detection using the conventional Faster R-CNN model in the prior art, and the red box is the part defect anchor box output by image detection using the optimized and improved Faster R-CNN model in the present invention. The numbers of corresponding colors next to the boxes are the probability values ​​of specific defects shown in the figure when the corresponding model method is used to detect part defects, and the model evaluation indicators described above are used to obtain them. Figure 8 to Figure 13 It can be seen from the actual examples that the method for detecting surface defects of industrial parts provided by the present invention can more accurately and precisely identify the location and type of surface defects of industrial parts, effectively improving the accuracy and reliability of part surface defect detection.

[0102] At the same time, we also compared the actual performance of the optimized and improved Faster R-CNN model used in the present invention with the SSD algorithm and the YOLOX algorithm in the detection of surface defects of industrial parts. Please refer to the following table.

[0103] Table 1

[0104]

[0105] As shown in Table 1, the optimized and improved Faster R-CNN model used in the present invention is generally superior to the other two algorithms in the work of detecting surface defects of industrial parts, and its mAP index is 14.1% higher than SSD and 5.4% higher than YOLOX. For irregularly shaped difficult-to-detect defects, the method of the present invention has a detection accuracy of 20.8% higher than SSD for corrosion defects and 6.1% higher than YOLOX for corrosion defects, which further verifies the advantages of the method of the present invention for irregularly shaped difficult-to-detect defects. For small target defects with tiny sizes, the method of the present invention has a detection accuracy of 39% higher than SSD for built-up edge defects and 11.1% higher than YOLOX for built-up edge defects, which also verifies that the method of the present invention has a better detection effect for small defects of parts that are difficult to detect.

[0106] Table 2

[0107]

[0108] As shown in Table 2, the part defect detection method provided by the present invention using the optimized and improved Faster R-CNN model is 17.5% higher than the mAP index of SSD and 3.2% higher than the mAP index of YOLOX. For irregular defects such as scratches and jets and small target defects such as paint bubbles and dirty spots, the detection method provided by the present invention shows obvious advantages, which verifies the superior detection effect of the method of the present invention on the detection of small and irregular surface defects of parts, and further verifies that the cascaded RPN network is used in the present invention to adjust the suggestion box step by step and perform adaptive convolution to effectively improve the detection ability of part defects, so that the overall defect detection mAP of the detection method of the present invention finally reaches 95.2%, which further verifies that the detection method of industrial part surface defects provided by the present invention is not only highly accurate, but also has strong generalization ability, and has broad application prospects in metal part surface defect detection.

[0109] According to another aspect of the present invention, an embodiment of a device 1400 for detecting surface defects of industrial parts is also provided herein.

[0110] Fig.14 The figure is a schematic diagram of the structure of a device for detecting surface defects of industrial parts according to another aspect of the present invention.

[0111] like Fig.14 As shown, the above industrial part surface defect detection device 1400 provided in this embodiment may include a memory 1401 and a processor 1402 coupled to the memory 1401. The processor 1402 may be configured to implement any of the above industrial part surface defect detection methods.

[0112] According to another aspect of the present invention, an embodiment of a computer storage medium is also provided herein.

[0113] The computer storage medium stores a computer program, which, when executed by a processor, can implement the steps of any of the above-mentioned methods for detecting surface defects of industrial parts.

[0114] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0115] The processor described in this case can be implemented using electronic hardware, computer software or any combination thereof. Whether such processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processor presented in this disclosure, any part of the processor, or any combination of processors can be implemented with a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gated logic, a discrete hardware circuit, and other suitable processing components configured to perform various functions described throughout this disclosure. The functionality of the processor presented in this disclosure, any part of the processor, or any combination of processors can be implemented with software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0116] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0117] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the universal principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting surface defects of industrial parts, for detecting flaws on the surface of industrial parts, wherein the flaws include corrosion, scratches, scuffs and built-up edge, and the detection method comprises: Acquire image data of industrial parts to be inspected, where the defects exist on the surfaces of some parts of the industrial parts to be inspected; Inputting the image data into an optimized and improved Faster R-CNN model, wherein the Faster R-CNN model is obtained after iterative training in a training phase and verification in a verification phase, wherein the Faster R-CNN model includes a Res2Net and FPN feature extraction network, a region candidate network, and a GRoIE network, wherein a multi-scale backbone architecture is adopted in the Res2Net network; as well as The location and defect type of the surface defects of industrial parts in the image data are determined according to the output of the Faster R-CNN model for detection.

2. The detection method according to claim 1, characterized in that The Res2Net network adopts a multi-scale backbone architecture, including: In the Res2Net network, the bottleneck block in each convolutional module consists of a 1×1 convolution, a 3×3 convolution and a 1×1 convolution, where after the first 1×1 convolution, the feature map is divided into s scale dimensions, and the 3×3 convolution is performed according to the following formula: Among them, K represents a 3×3 convolution kernel, x represents the input feature map, y represents the output feature map, i represents the scale dimension of the feature map, and s represents the total number of scale dimensions into which the feature is divided.

3. The detection method according to claim 1, characterized in that The region candidate network adopts a multi-level cascade, each level includes an adaptive convolution module that dynamically generates an adaptive convolution kernel according to the detection target, and generates a candidate suggestion box based on the convolution calculation. The adaptive convolution module and the candidate suggestion box at the previous level are respectively used to bridge the image features of the next level and provide a regression bounding box.

4. The detection method according to claim 3, characterized in that The region proposal network adopts a two-stage cascade.

5. The detection method according to claim 1, characterized in that The image processing of the GRoIE network includes the following steps: The region of interest pooler performs maximum pooling on regions of interest of different sizes in the image to obtain an image representation of uniform size; Perform preliminary processing on the feature map to be aggregated so that each image feature is semantically associated with adjacent features; Merge and aggregate the single RoIs from each branch to fuse the feature information of different receptive fields; and Convolutional layers, non-local operation layers and attention layers are set to fuse global image features and focus on the characteristic information of surface defects of the industrial parts.

6. The detection method according to claim 1, characterized in that: The step of acquiring image data of the industrial part to be inspected comprises: After acquiring the image data of the industrial part to be inspected, the image data is preprocessed for data enhancement.

7. The detection method according to claim 1, characterized in that The training phase of the Faster R-CNN model includes: Acquire an image data sample of an industrial part whose surface contains the defect, and annotate and preprocess the data sample to obtain a part surface defect data set; Dividing the part surface defect dataset into a training set and a validation set; and Iteratively optimizing the Faster R-CNN model using the training set; The validation phase of the Faster R-CNN model includes: A model evaluation index is selected, and the validation set is used to verify the performance of the Faster R-CNN model obtained in the training phase, so as to optimize the Faster R-CNN model for part defect detection.

8. The detection method according to claim 7, characterized in that The step of dividing the part surface defect data set into a training set and a validation set includes: The preprocessed part surface defect dataset is shuffled, and the part surface defect dataset is divided into three independent subsets by a three-fold cross validation method. Two of the subsets are selected as the training sets each time, and the remaining one is selected as the validation set, so as to train and validate the Faster R-CNN model.

9. A device for detecting surface defects of industrial parts, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the method for detecting surface defects of industrial parts according to any one of claims 1 to 8.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting surface defects of industrial parts according to any one of claims 1 to 8 is implemented.

Images