Defect detection method and device, electronic equipment and storage medium

Abstract

The application discloses a defect detection method and device, electronic equipment and storage medium. The defect detection method comprises the following steps: acquiring a plurality of defect key points in an initial defect area of a to-be-detected image; performing aggregation processing on pixel points in a preset range corresponding to each defect key point according to feature information of the pixel points, to obtain an aggregation area corresponding to each defect key point; performing merging processing on the aggregation area corresponding to each defect key point and the initial defect area where each defect key point is located respectively, to obtain a candidate defect area; and performing splicing on two candidate defect areas with matched position information according to position information of each candidate defect area, to obtain a defect detection result. The above scheme can accurately detect defects in the to-be-detected image, and improves defect detection efficiency.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to a defect detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of industrial technology, the production efficiency of corresponding products has been significantly improved, and the requirements for product quality testing capabilities have become more stringent.

[0003] Currently, most product testing methods rely on manual inspection, such as defect detection for chemical fiber spindles. However, with the continuous advancement of industrial technology, traditional manual inspection methods are far from meeting the needs of subsequent production efficiency. Furthermore, the lack of uniformity in inspection standards can lead to production accidents such as missed or false detections, affecting product production efficiency and quality. Summary of the Invention

[0004] This application provides at least one defect detection method, apparatus, device, and computer-readable storage medium.

[0005] The first aspect of this application provides a defect detection method, comprising: acquiring a plurality of defect key points in an initial defect region of an image to be detected; aggregating the pixels in a preset range corresponding to each defect key point according to the feature information of the pixels in the preset range corresponding to each defect key point to obtain an aggregated region corresponding to each defect key point; merging the aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located to obtain a candidate defect region; and splicing two candidate defect regions whose position information matches according to the position information of each candidate defect region to obtain a defect detection result.

[0006] In one embodiment, before the step of aggregating the pixels in the preset range corresponding to each defect key point according to the feature information of the pixels in the preset range corresponding to each defect key point to obtain the aggregated region corresponding to each defect key point, the method further includes: performing feature detection on the image to be detected to obtain grayscale information and edge information of the image to be detected; constructing a self-attention heatmap of the image to be detected based on the grayscale information and the edge information, wherein the self-attention heatmap contains the heat value of each pixel in the image to be detected, and the heat value is used to characterize the attention level of each pixel in the image to be detected.

[0007] In one embodiment, the feature information of the pixel includes the grayscale information and the thermal value. The step of aggregating the pixels within the preset range corresponding to each defect key point based on the feature information of the pixels within the preset range corresponding to each defect key point to obtain an aggregated region corresponding to each defect key point includes: filtering the pixels within the preset range corresponding to each defect key point based on the grayscale information and thermal value of each defect key point to obtain filtered pixels; and aggregating the filtered pixels within the preset range corresponding to each defect key point to obtain an aggregated region corresponding to each defect key point.

[0008] In one embodiment, the step of constructing a self-attention heatmap of the image to be detected based on the grayscale information and the edge information includes: constructing a grayscale feature map of the image to be detected based on the grayscale information of the image to be detected and the filtered image to be detected; constructing an edge feature map of the image to be detected based on the edge information of the image to be detected; and fusing the grayscale feature map and the edge feature map to obtain a self-attention heatmap of the image to be detected.

[0009] In one embodiment, the step of stitching together two candidate defect regions whose location information matches, based on the location information of each candidate defect region, to obtain a defect detection result includes: traversing each candidate defect region; obtaining the direction matching degree and coordinate matching degree between each candidate defect region based on the direction information and point coordinate information contained in the location information of each candidate defect region; determining whether the direction matching degree between each candidate defect region satisfies a preset direction matching degree and whether the coordinate matching degree between each candidate defect region satisfies a preset coordinate matching degree; and stitching together two candidate defect regions whose direction matching degree satisfies the preset direction matching degree and whose coordinate matching degree satisfies the preset coordinate matching degree to obtain the defect detection result.

[0010] In one embodiment, the step of obtaining several defect key points in the initial defect region of the image to be detected includes: obtaining the image to be detected; inputting the image to be detected into a pre-trained target detection model to obtain the initial defect region output by the target detection model; and extracting defect key points in the initial defect region based on the image contrast between the initial defect region and the background region corresponding to the initial defect region, wherein the background region is the pixel region in the image to be detected other than the initial defect region.

[0011] In one embodiment, the step of obtaining several key defect points in the initial defect region of the image to be detected includes: obtaining the object to be detected in the image to be detected; determining the region to be detected in the image to be detected based on the object to be detected; cutting the region to be detected according to a preset sliding window to obtain several sub-images to be detected; performing target detection on each sub-image to obtain several key defect points in the initial defect region of each sub-image to be detected.

[0012] A second aspect of this application provides a defect detection device, comprising: an acquisition module for acquiring a plurality of defect key points in an initial defect region of an image to be detected; an aggregation module for aggregating pixels in a preset range corresponding to each defect key point according to feature information of pixels in a preset range corresponding to each defect key point, to obtain an aggregated region corresponding to each defect key point; a merging module for merging the aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located, respectively, to obtain a candidate defect region; and a stitching module for stitching two candidate defect regions whose position information matches according to position information of each candidate defect region, to obtain a defect detection result.

[0013] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described defect detection method.

[0014] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the aforementioned defect detection method.

[0015] The above scheme, based on the defect key points in the initial defect region of the image to be detected, aggregates the pixels in the preset range corresponding to the defect key points to obtain the aggregated region corresponding to the defect key points; merges the aggregated region and the initial defect region to obtain the candidate defect region in the image to be detected; and splices the two candidate defect regions with matching position information to obtain the defect detection result. This can accurately detect defects in the image to be detected and improve defect detection efficiency.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0018] Figure 1This is a flowchart illustrating an exemplary embodiment of the defect detection method of this application;

[0019] Figure 2 This is a schematic diagram illustrating the effect of neighborhood pixel aggregation processing in the defect detection method of this application;

[0020] Figure 3 This is a flowchart illustrating an exemplary embodiment of the defect detection method of this application before performing neighborhood pixel aggregation processing;

[0021] Figure 4 This is a flowchart illustrating the process of aggregating pixels within a preset range corresponding to each defect key point in the defect detection method of this application.

[0022] Figure 5 This is a schematic diagram of the process of obtaining a self-attention heatmap through image fusion in the defect detection method of this application;

[0023] Figure 6 This is a schematic diagram illustrating the effect of constructing grayscale feature maps and edge feature maps in the defect detection method of this application;

[0024] Figure 7 This is a flowchart illustrating the candidate defect region splicing process in the defect detection method of this application;

[0025] Figure 8 This is a schematic diagram illustrating the effect of the candidate defect region splicing process in the defect detection method of this application;

[0026] Figure 9 This is a flowchart illustrating the process of obtaining several key defect points in the initial defect region of the image to be detected in the defect detection method of this application.

[0027] Figure 10 This is a block diagram illustrating a defect detection apparatus according to an exemplary embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0029] Figure 12 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0032] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the defect detection method of this application. Specifically, it may include the following steps:

[0034] Step S110: Obtain several key defect points in the initial defect region of the image to be detected.

[0035] The initial defect region refers to the image region in the image to be detected that may contain defects. The image to be detected records the information of the object to be detected in the form of an image. It can be understood that the initial defect region of the image to be detected mentioned in this application actually refers to the region in the object to be detected that may contain defects. The defect key point refers to the feature point obtained by feature extraction of the initial defect region.

[0036] Specifically, feature extraction of the initial defect region of the image to be detected can obtain the key points of the defect in the initial defect region.

[0037] For example, a neural network model, such as a target detection model, can be used to perform feature detection on the image to be detected to obtain the initial defect region of the image to be detected. Then, feature extraction can be performed on the initial defect region to obtain the defect key points in the initial defect region.

[0038] Step S120: Based on the feature information of the pixels in the preset range corresponding to each defect key point, the pixels in the preset range corresponding to each defect key point are aggregated to obtain the aggregated region corresponding to each defect key point.

[0039] The feature information of a pixel includes information such as the pixel's grayscale, brightness, and intensity. The preset range refers to the pre-set image range, such as the neighborhood range of a defect key point.

[0040] For example, the preset range can be set to 4 neighborhoods or 8 neighborhoods; see the following for details. Figure 2 This is a schematic diagram illustrating the effect of aggregating pixels within a preset range corresponding to key defect points. Figure 2 The diagram illustrates the operation of aggregating pixels with a gray value of 120 or higher within a preset range corresponding to the defect key point under 4-neighbor and 8-neighbor conditions, respectively, to obtain the aggregated region corresponding to the defect key point. The aggregation operation can be implemented by a preset region growing algorithm, which will not be elaborated here.

[0041] Specifically, within a preset range corresponding to each defect key point, the pixels within the preset range corresponding to each defect key point are clustered based on the feature information of the pixels surrounding each defect key point to obtain the aggregated region corresponding to each defect key point.

[0042] Step S130: Merge the aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located to obtain the candidate defect region.

[0043] The aggregation region is obtained by aggregating the pixels within a preset range corresponding to the defect key points.

[0044] The initial defect region can be obtained based on a preset target detection model. By merging the aggregated region with the initial defect region, the error and false detection caused by the deep learning model can be effectively filtered out, thereby improving the accuracy and reliability of the defect detection results of the image to be detected and enhancing the stability of the defect detection method provided in this application.

[0045] Specifically, the aggregated region corresponding to each defect key point and the initial defect region are bitwise ORed to merge the aggregated region corresponding to each defect key point and the initial defect region to obtain the candidate defect region.

[0046] Step S140: Based on the location information of each candidate defect region, the two candidate defect regions with matching location information are spliced ​​together to obtain the defect detection result.

[0047] Location information refers to the direction and coordinates of the candidate defect region, such as the principal axis direction, endpoint coordinates, and center point coordinates of the candidate defect region.

[0048] Specifically, after traversing each candidate defect region, the defect detection result can be obtained by splicing two candidate defect regions whose position information matches each other based on the position information of each candidate defect region.

[0049] It should be noted that if the location information of two candidate defect regions matches, they can be considered to be different parts of the same defect to a certain extent. Therefore, it is necessary to splice the two candidate defect regions with matching location information. It can be understood that the remaining unspliced ​​candidate defect regions after splicing can be considered to be independent defects to a certain extent. Statistical analysis of the spliced ​​candidate defect regions and the unspliced ​​candidate defect regions can be used as the defect detection result for output.

[0050] As can be seen, this application aggregates the pixels in the preset range corresponding to the defect key points in the initial defect region of the image to be detected, based on the defect key points obtained, to obtain the aggregated region corresponding to the defect key points; the aggregated region and the initial defect region are merged to obtain the candidate defect region in the image to be detected, and the two candidate defect regions with matching position information are spliced ​​to obtain the defect detection result. Thus, the defects in the image to be detected can be accurately detected, and the defect detection efficiency can be improved.

[0051] Based on the above embodiments, the embodiments of this application adopt the following... Figure 3 The flowchart shown details the operations preceding the step of aggregating pixels within a preset range corresponding to each defect key point to obtain the aggregated region corresponding to each defect key point. Please refer to [link to flowchart documentation]. Figure 3 , Figure 3 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of the defect detection method prior to step 120. Specifically, the method of this embodiment includes the following steps:

[0052] Step 310: Perform feature detection on the image to be detected to obtain the grayscale information and edge information of the image to be detected.

[0053] Grayscale information refers to the grayscale value of each pixel in the image to be detected. Pixel values ​​in an image, such as boundaries where grayscale values ​​suddenly change, are called edges. Edges reflect the speed of image change, and the edge information here refers to the edge information in the image to be detected, which can be obtained by edge extraction. For example, image edge information is mainly concentrated in the high-frequency band, meaning that the image will produce abrupt gradient changes at the edges. This feature can be used to perform high-frequency filtering or image sharpening to obtain edge information; this is the process of edge detection. Therefore, image edge detection is essentially high-frequency filtering.

[0054] For example, the edge information of the image to be detected can be obtained by performing Gaussian smoothing and differentiation steps based on the Sobel operator.

[0055] Step 320: Construct a self-attention heatmap of the image to be detected based on grayscale information and edge information. The self-attention heatmap contains the heat value of each pixel in the image to be detected. The heat value is used to characterize the attention level of each pixel in the image to be detected.

[0056] Self-attention heatmaps are heatmaps of images to be detected, constructed based on a self-attention mechanism. This mechanism can quickly extract important features from sparse data and excels at capturing internal correlations within data or features. Heatmaps can characterize the attention level of an image, detect outliers, display differences between multiple variables, and detect any correlations between pixels. Specifically, the heat values ​​in the heatmap can be used to characterize the attention level of each pixel in the image to be detected.

[0057] Self-attention mechanisms and heatmaps can be used to identify pixels in the image under test that have high attention, i.e., pixels that may be located in defective parts of the image under test.

[0058] As can be seen, by performing weighted fusion calculations on the grayscale feature map and the edge feature map through the above steps to obtain a heat map, the heat map and the heat values ​​of the pixels can reflect which pixels in the image to be detected have higher attention, or which pixels have higher weight proportions. Analyzing these pixels whose heat values ​​are within the preset heat value range can greatly determine the pixels with defects in the image to be detected, thereby improving the accuracy of defect detection.

[0059] Based on the above embodiments, the embodiments of this application adopt the following... Figure 4 The flowchart shown details the specific steps of aggregating pixels within a preset range corresponding to each defect key point to obtain the aggregated region corresponding to each defect key point. Please refer to [link to flowchart documentation]. Figure 4 , Figure 4 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step 120 in the defect detection method. Specifically, the method of this embodiment includes the following steps:

[0060] Step 410: Based on the grayscale information and thermal value of each defect key point, the pixels within the preset range corresponding to each defect key point are filtered to obtain the filtered pixels.

[0061] For ease of explanation, the defects of the object to be detected in the image to be detected will be referred to as the defect object.

[0062] It is understandable that within the preset range corresponding to each defect key point, there may be pixels belonging to the defective object, and there may also be pixels that do not belong to the defective object. Therefore, the pixels within the preset range corresponding to each defect key point can be filtered. Specifically, the pixels within the preset range corresponding to each defect key point can be filtered based on the grayscale information and thermal value of each defect key point.

[0063] For example, the grayscale range of neighboring pixels of each defect key point is determined based on the grayscale information of each defect key point, and the thermal range of neighboring pixels of each defect key point is determined based on the thermal value of each defect key point. Pixels within a preset range corresponding to each defect key point, whose grayscale information and thermal value are both within the defect grayscale range, are selected as pixels of the defect object. The explanation of the neighborhood can be found in the foregoing embodiments and... Figure 2 The explanation will not be repeated here.

[0064] Step 420: Aggregate the filtered pixels within the preset range corresponding to each defect key point to obtain the aggregated region corresponding to each defect key point.

[0065] As can be seen from the above, the filtered pixels are very likely to be pixels belonging to the defective object. Therefore, by aggregating these pixels, we can obtain the approximate range where the defective object exists, which is the aggregation area corresponding to each defect key point.

[0066] For example, in addition to aggregating pixels in the neighborhood of the defect key point to achieve region growth based on the preset defect grayscale range and defect thermal range, the aggregation process of pixels can also be controlled by a preset image template. Specifically, an image template is constructed by selecting n×n grayscale information in the neighborhood of the defect key point, and the image template is matched with the grayscale information near the pixels to be aggregated, so that the aggregation stops when the aggregation region encounters intersections and inflection points.

[0067] Based on the above embodiments, this application adopts the following... Figure 5 The flowchart shown illustrates the process of obtaining a self-attention heatmap using image fusion in step 320. Specifically, the method of this embodiment includes the following steps:

[0068] Step 510: Construct a grayscale feature map of the image to be detected based on the grayscale information of the image to be detected and the filtered image to be detected.

[0069] The grayscale information of the image to be detected includes the grayscale values ​​of the pixels in the image, as explained in detail in the foregoing embodiments and will not be repeated here. The grayscale feature map refers to the grayscale information based on the image to be detected and the filtered image to be detected. For example, the grayscale variance information of the image to be detected is calculated, and a feature map is constructed based on the grayscale variance information. For information on the construction effect of the grayscale feature map, please refer to [reference needed]. Figure 6 The effect diagram.

[0070] Furthermore, the image to be detected can be filtered by mean filtering to reduce image noise in the filtered image and make the filtered image smoother.

[0071] Specifically, the image to be detected is subjected to mean filtering using a preset convolution kernel to obtain the filtered image to be detected, which is referred to as the smoothed image. The size of the convolution kernel needs to be larger than the pixel width of the defect object in the image to be detected. Based on the brightness relationship between the initial defect area and the background area in the image to be detected, the gray-level variance information of the image to be detected is obtained by subtracting and calculating the variance between the smoothed image and the image to be detected. The gray-level feature map of the image to be detected is constructed based on the gray-level variance information.

[0072] It should be noted that the background area mentioned above refers to the pixel area in the image to be inspected, excluding the initial defect area. It can be understood that the gray value of a pixel ranges from 0 to 255. The larger the gray value of a pixel, the whiter it appears in the image, while the smaller the gray value of a pixel, the darker it appears in the image. Therefore, the brightness relationship between the initial defect area and the background area can be characterized by gray information, i.e., gray value.

[0073] Step 520: Construct an edge feature map of the image to be detected based on the edge information of the image to be detected.

[0074] For example, edge information can be obtained by using the Sobel operator to extract edges from the image to be detected. Based on the obtained edge information, an edge feature map of the image to be detected can be constructed. For details on the construction effect of the edge feature map, please refer to [reference needed]. Figure 6 The effect diagram.

[0075] Specifically, the Sobel operator is used to extract the gradient information in the preset horizontal and preset vertical directions of the image to be detected. The edge amplitude map of each pixel in the image to be detected is calculated based on the obtained horizontal and vertical gradient information. The edge amplitude map is then subjected to mean filtering using the convolution kernel in step 510 to obtain the edge feature map.

[0076] Furthermore, the calculation process for the edge amplitude map Amp is as follows:

[0077]

[0078] dx=A*C

[0079] dy=B*C

[0080]

[0081] Wherein, matrix A is the Sobel operator for the preset horizontal direction, matrix B is the Sobel operator for the preset vertical direction, C is the image to be detected, dx is the horizontal gradient information of the image to be detected, and dy is the vertical gradient information of the image to be detected.

[0082] Step 530: The grayscale feature map and the edge feature map are fused to obtain the self-attention heatmap of the image to be detected.

[0083] Specifically, the grayscale feature map obtained in step 510 and the edge feature map obtained in step 520 are weighted and fused to obtain the self-attention heatmap of the image to be detected. The size of the heatmap value of each pixel reflects the attention (weight ratio) of the pixel position in the image to be detected, that is, it represents the probability that the pixel position belongs to the defective object.

[0084] Based on the above embodiments, this application adopts the following... Figure 7 The flowchart shown illustrates the process of splicing candidate defect regions in step S140. Specifically, the method of this embodiment includes the following steps:

[0085] Step 710: Traverse each candidate defect region and obtain the direction matching degree and coordinate matching degree between each candidate defect region based on the direction information and point coordinate information contained in the position information of each candidate defect region.

[0086] The directional information includes the principal axis direction of the candidate defect region. The principal axis direction is usually represented by the longer direction of the object. The definition of the principal axis has three main aspects: 1. From the perspective of projection, the object's width is minimized when projecting along the principal axis direction; 2. From a statistical perspective, the principal axis direction is the direction of the object's principal component. A linear transformation based on this principal component can remove the correlation between elements in a random vector; 3. From the perspective of texture analysis and spectrum analysis, for regular, elongated objects, the principal axis direction is perpendicular to the direction of the texture with the most concentrated energy on the spectrum map.

[0087] Therefore, methods for determining the principal axis direction include, but are not limited to, projection method, principal component analysis method and spectral texture analysis method, and are not limited here. Thus, the principal axis direction of the candidate defect region in this application can be obtained.

[0088] Point coordinate information includes the endpoint coordinates and center point coordinates of the candidate defect region. The endpoint coordinates can be used to determine the distance between the endpoints of two candidate defect regions. The center point coordinates can be used to connect the center points of two candidate defect regions. The slope of the line connecting the center points can be used to determine the similarity with the principal axis direction of the two candidate defect regions, and thus determine whether the two candidate defect regions can be spliced ​​together.

[0089] Specifically, after sorting each candidate defect region according to its principal axis length, the process is traversed. The directional matching degree between two candidate defect regions is calculated based on the principal axis direction of each candidate defect region. The coordinate matching degree between two candidate defect regions is calculated based on the endpoint coordinate information of each candidate defect region. The center points of two candidate defect regions are connected based on the center point coordinate information of each candidate defect region. The slope of the line connecting the center points is calculated, and the matching degree between the slope and the slope of the two candidate defect regions is determined.

[0090] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating the effect of the candidate defect region splicing process, combined with... Figure 8 For example, when the principal axis directions of candidate defect region 1 and candidate defect region 2 are k1 and k2 respectively, the absolute value of the difference between k1 and k2, i.e., |k1-k2|, can be used to represent the directional matching degree between the two candidate defect regions. The larger the absolute value of the difference between k1 and k2, the greater the directional difference between the two candidate defect regions; the smaller the absolute value of the difference between k1 and k2, the smaller the directional difference between the two candidate defect regions. Similarly, when judging the slope matching degree between the line connecting the center points and the principal axis directions of the two candidate defect regions, if the slope of the line connecting the center points is k... c Then it can be determined according to |k1-k c | and |k2-k c | indicates the slope matching degree between two candidate defect regions.

[0091] It should also be noted that the coordinate matching degree of two candidate defect regions can be represented by the distance between the endpoints of the two candidate defect regions. Specifically, it can be calculated from the coordinate information of the two endpoints. It can be understood that if the endpoints of two candidate defect regions are too close, it can be considered to a certain extent that the two candidate defect regions belong to the same defect object. Therefore, the two candidate defect regions can be spliced ​​together.

[0092] Step 720: Determine whether the directional matching degree between each candidate defect region meets the preset directional matching degree and whether the coordinate matching degree between each candidate defect region meets the preset coordinate matching degree.

[0093] The preset direction matching degree is used to determine whether the directions of two candidate defect regions are similar based on the direction matching degree between them; the preset coordinate matching degree is used to determine whether the distance between two candidate defect regions is close based on the direction matching degree between them. It is easy to understand that if there are two candidate defect regions whose direction matching degree meets the preset direction matching degree and whose coordinate matching degree meets the preset coordinate matching degree, then the two candidate defect regions are considered to have similar directions and close endpoints, that is, the two candidate defect regions are considered to belong to the same defect object, and the two candidate defect regions should be spliced ​​together.

[0094] It should also be noted that the slope matching degree of the line connecting the center points of two candidate defect regions is also to verify whether the directions of the two candidate defect regions are similar. On the basis of verifying the main axis direction, the slope matching degree of the line connecting the center points is used to verify the direction of the two candidate defect regions, avoiding the merging of two or more parallel candidate defect regions with similar endpoints belonging to different tripwires. This can enhance the credibility of the data and enhance the robustness of the verification process.

[0095] Step 730: Two candidate defect regions whose orientation matching degree and coordinate matching degree both satisfy the preset orientation matching degree are spliced ​​together to obtain the defect detection result.

[0096] The defect detection results include the length and number of defects.

[0097] Using steps 710 and 720 as an example, the preset direction matching degree is Thr, and the preset coordinate matching degree is Dis. After calculating the direction and coordinate information between each candidate defect region, if the direction matching degree of the principal axis direction of two candidate defect regions is |k1-k2|, and the coordinate matching degree of endpoint i and endpoint j is D... ij and the slope matching degree are respectively |k1-k c |、|k2-k c Then determine whether these two candidate defect regions simultaneously satisfy |k1-k2|≤Thr,D ij <Dis、|k1-k c |≤Thr and|k2-k c If |≤Thr; then the two candidate defect regions are considered to be similar in direction and close in distance, and are highly likely to belong to the same defect object. In this case, the two candidate defect regions are spliced ​​together to obtain a single defect object, and the defect length of the defect object is recalculated. If not, the two candidate defect regions are considered not to belong to the same defect object, and there is no need to splice them together. Understandably, in this case, the two candidate defect regions are considered to be two different defect objects.

[0098] Based on the above embodiments, this application adopts the following... Figure 9 The flowchart shown illustrates the step S110 of acquiring several key defect points in the initial defect region of the image to be detected. Specifically, the method of this embodiment includes the following steps:

[0099] Step S910: Obtain the image to be detected;

[0100] Step S920: Input the image to be detected into the pre-trained target detection model to obtain the initial defect region output by the target detection model;

[0101] Step S930: Based on the image contrast between the initial defect region and the background region corresponding to the initial defect region, extract the defect key points in the initial defect region. The background region is the pixel region in the image to be detected other than the initial defect region.

[0102] For example, a pre-trained object detection model can use YOLOv4 as the backbone network to extract features from the image to be detected, thereby obtaining semantic information from the image. The backbone network mainly consists of several standard convolutional layers, max pooling layers, and fully connected layers.

[0103] In addition, the target detection model also includes a neck layer, which is used to extract network features more fully, fuse features at different scales, and perform detection on feature maps at multiple scales, thereby improving the network's detection performance for small targets.

[0104] It should also be noted that when training the target detection model, the training samples can be labeled with multiple overlapping segments according to the preset small targets, rather than labeling the sample data as a whole. For example, multiple overlapping but not included small annotation boxes can be made for the tripwire defect. Each annotation box contains a part of the tripwire defect, so that the output of the supervised learning detection model can contain multiple detection boxes, which improves the accuracy of target detection. Based on the image contrast between the defect area and the background area, the darkest pixel position on the diagonal of each detection box is calculated as the defect key point, and the gray value of the point is recorded.

[0105] Based on the above embodiments, the step of obtaining several key defect points in the initial defect region of the image to be detected in step S110 will be described. Specifically, the method of this embodiment includes the following steps:

[0106] The target object in the image to be detected is obtained, and the target region in the image to be detected is determined based on the target object.

[0107] To save computing resources and improve the efficiency of defect detection, before performing defect detection on the image to be detected, the detection area can be divided in the image based on the object to be detected in the image.

[0108] For example, in the detection scenario of chemical fiber spindles, if the outer side of the surface area of ​​the spindle is known to be circular, a circle-finding tool can be used to search for the boundary points of the surface of the chemical fiber spindle within a certain range and perform circular fitting to obtain the center and radius of the surface area of ​​the spindle. Then, the position information of the circumscribed rectangle of the surface area of ​​the spindle can be obtained from the center and radius. This circumscribed rectangle is the surface area of ​​the spindle, which is also the area to be detected in the image to be detected.

[0109] The region to be detected is segmented according to a preset sliding window to obtain several sub-images to be detected.

[0110] To make defect detection results more accurate, the image to be detected can be segmented into smaller sub-images. Detecting these sub-images improves the accuracy of the defect detection method.

[0111] For example, a preset sliding window is used to capture images of the acquired area to be detected by overlapping sliding windows. For instance, the image data of the surface area of ​​the silk spindle is cut by a sliding window with a window size of 416×416×3 and a step size of 400×400×3 to obtain multiple sub-images of the area to be detected with a size of 416×416×3.

[0112] Target detection is performed on each sub-image to be detected, and several key defect points in the initial defect region of each sub-image to be detected are obtained.

[0113] As shown above, by inputting the sub-images to be detected into the pre-trained target detection model, the model can obtain multiple detection boxes for each defect during inference and obtain multiple defect key points.

[0114] It should also be noted that before training the object detection model, the image to be detected can be segmented using a preset sliding window to obtain sub-images to be detected; multiple overlapping annotations are made on the defect objects in each sub-image to be detected so that the object detection model can generate multiple detection boxes for each defect object during model inference; and preprocessing techniques such as angle rotation, flipping, and random brightness changes are performed on each sub-image to be detected and its corresponding annotations to achieve data augmentation, thereby obtaining the training dataset for training the object detection model.

[0115] To further explain, in the process of training the object detection model, in addition to building the YOLOv4 backbone network and the neck layer, a loss function also needs to be set. The loss function of the object detection model provided in this application can be derived from the category prediction loss FL(…). t ) and bounding box location prediction loss LOSS CIOU It consists of two parts, as detailed below:

[0116] loss = FL(p) t )+ coord LOSS CIOU

[0117] Where, λ coord λ is the adjustment factor for the location prediction loss, which can be adjusted according to the importance placed on the accuracy of the bounding box location in the actual application of the target detection model. coord When λ < 1, the model prioritizes the accuracy of the predicted bounding box's category; coord When the value is greater than 1, the model places more emphasis on the accuracy of the predicted box location.

[0118] For example, the category prediction loss function FL(p t The Focal loss function can be used to effectively address the imbalance between positive and negative samples and the imbalance in the loss contribution of hard samples, which have low frequency and low model prediction scores. Assuming the network's output class is p, and the corresponding labeled class is y, the confidence level of the model predicting the correct class is:

[0119]

[0120] The cross-entropy loss function for binary classification can be expressed as:

[0121] CE(p,y)=E(p t ) = -log(p t )

[0122] The model's classification loss function is as follows:

[0123] FL(p t )=-(1- t ) y log(p t )

[0124] In the formula, y is an adjustment factor that controls the decay rate of the simple sample weights. The balance coefficient (1-p) t This reduces the contribution of a large number of simple samples to the overall loss, thus allowing the model to focus more on a smaller number of difficult samples.

[0125] Location prediction loss function LOSS CIOU The loss function used is based on the Intersection over Union (IOU) ratio of the predicted and labeled boxes. The IOU is the ratio of the area intersection to the area union of the predicted and labeled boxes. Assume the model's predicted boxes are b = {x, y, w, h}, and the labeled boxes are b... gt ={x gt ,y gt ,w gt ,h gtThe parameters represent the x-coordinate and y-coordinate of the top-left corner of the rectangle, as well as the width and height of the rectangle. The loss is defined as follows:

[0126] LOSS CIOU =1-CIOU

[0127]

[0128]

[0129]

[0130] The intersection-union function CIOU considers not only the overlap area between the predicted and calibration boxes, but also their center distance and aspect ratio, making the model's predicted boxes fit the calibration boxes more closely. In the formula, ρ 2 (b,b gt ) represents the distance between the center points of the model prediction box and the calibration box, and c represents the diagonal distance of the smallest rectangle that can simultaneously contain both the prediction box and the calibration box.

[0131] As described above, the established object detection network framework is trained using the training dataset. Stochastic gradient descent (SGD) can be used for model optimization. The initial learning rate is set to 0.01. The learning rate is warmed up for the first 1000 generations, and then a cosine annealing learning rate update strategy is adopted. The model iterates for a total of 10,000 generations.

[0132] It should be noted that the above-mentioned defect detection method can be applied to the inspection of chemical fiber spindles or yarn spools, etc. Therefore, this application does not limit the applicable scenarios for the defect detection method.

[0133] It should be further noted that the entity executing the defect detection method can be a defect detection device. For example, the defect detection method can be executed by a terminal device, a server, or other processing equipment. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the defect detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0134] Figure 10 This is a block diagram illustrating a defect detection apparatus according to an exemplary embodiment of this application. Figure 10As shown, the exemplary defect detection device 1000 includes: an acquisition module 1010, an aggregation module 1020, a merging module 1030, and a splicing module 1040. Specifically:

[0135] The acquisition module 1010 is used to acquire several key defect points in the initial defect region of the image to be detected.

[0136] The aggregation module 1020 is used to aggregate the pixels in the preset range corresponding to each defect key point according to the feature information of the pixels in the preset range corresponding to each defect key point, so as to obtain the aggregated region corresponding to each defect key point.

[0137] The merging module 1030 is used to merge the aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located to obtain the candidate defect region.

[0138] The splicing module 1040 is used to splice two candidate defect regions with matching location information based on the location information of each candidate defect region to obtain the defect detection result.

[0139] In this exemplary defect detection device, based on the defect key points in the initial defect region of the acquired image to be detected, the pixels in the preset range corresponding to the defect key points are aggregated to obtain the aggregated region corresponding to the defect key points; the aggregated region and the initial defect region are merged to obtain the candidate defect region in the image to be detected; the two candidate defect regions with matching position information are spliced ​​together to obtain the defect detection result, thereby accurately detecting defects in the image to be detected and improving defect detection efficiency.

[0140] The functions of each module can be found in the embodiment of the defect detection method, and will not be repeated here.

[0141] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 1100 includes a memory 1110 and a processor 1120. The processor 1120 is used to execute program instructions stored in the memory 1110 to implement the steps in any of the above-described defect detection method embodiments. In a specific implementation scenario, the electronic device 1100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 1100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0142] Specifically, processor 1120 controls itself and memory 1110 to implement the steps in any of the defect detection method embodiments described above. Processor 1120 may also be referred to as a CPU (Central Processing Unit). Processor 1120 may be an integrated circuit chip with signal processing capabilities. Processor 1120 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 1120 may be implemented using integrated circuit chips.

[0143] The above scheme, based on the defect key points in the initial defect region of the image to be detected, aggregates the pixels in the preset range corresponding to the defect key points to obtain the aggregated region corresponding to the defect key points; merges the aggregated region and the initial defect region to obtain the candidate defect region in the image to be detected; and splices the two candidate defect regions with matching position information to obtain the defect detection result. This can accurately detect defects in the image to be detected and improve defect detection efficiency.

[0144] Please see Figure 12 , Figure 12 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 1200 stores program instructions 1210 that can be executed by a processor. The program instructions 1210 are used to implement the steps in any of the above-described defect detection method embodiments.

[0145] The above scheme, based on the defect key points in the initial defect region of the image to be detected, aggregates the pixels in the preset range corresponding to the defect key points to obtain the aggregated region corresponding to the defect key points; merges the aggregated region and the initial defect region to obtain the candidate defect region in the image to be detected; and splices the two candidate defect regions with matching position information to obtain the defect detection result. This can accurately detect defects in the image to be detected and improve defect detection efficiency.

[0146] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0147] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A defect detection method, characterized in that, The method includes: Obtain several key defect points in the initial defect region of the image to be detected; Feature detection is performed on the image to be detected to obtain grayscale information and edge information of the image to be detected; a self-attention heatmap of the image to be detected is constructed based on the grayscale information and the edge information, the self-attention heatmap contains the heat value of each pixel in the image to be detected, and the heat value is used to characterize the attention level of each pixel in the image to be detected; Based on the feature information of pixels within a preset range corresponding to each defect key point, the pixels within the preset range corresponding to each defect key point are aggregated to obtain an aggregated region corresponding to each defect key point; the feature information of the pixels includes the grayscale information and the thermal value; The step of aggregating the pixels within a preset range corresponding to each defect key point based on the feature information of the pixels within that preset range to obtain an aggregated region corresponding to each defect key point includes: filtering the pixels within a preset range corresponding to each defect key point based on the grayscale information and thermal value of each defect key point to obtain filtered pixels; and aggregating the filtered pixels within the preset range corresponding to each defect key point to obtain an aggregated region corresponding to each defect key point. The aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located are merged to obtain a candidate defect region. Based on the location information of each candidate defect region, the two candidate defect regions with matching location information are spliced ​​together to obtain the defect detection result.

2. The method according to claim 1, characterized in that, The step of constructing a self-attention heatmap of the image to be detected based on the grayscale information and the edge information includes: Based on the grayscale information of the image to be detected and the filtered image to be detected, a grayscale feature map of the image to be detected is constructed. Construct an edge feature map of the image to be detected based on the edge information of the image to be detected; The grayscale feature map and the edge feature map are fused to obtain the self-attention heatmap of the image to be detected.

3. The method according to claim 1, characterized in that, The step of stitching together two candidate defect regions whose location information matches, based on the location information of each candidate defect region, to obtain the defect detection result includes: Traverse each candidate defect region, and obtain the direction matching degree and coordinate matching degree between each candidate defect region based on the direction information and point coordinate information contained in the position information of each candidate defect region; Determine whether the directional matching degree between each candidate defect region meets the preset directional matching degree and whether the coordinate matching degree between each candidate defect region meets the preset coordinate matching degree. The defect detection result is obtained by splicing together two candidate defect regions whose direction matching degree satisfies the preset direction matching degree and whose coordinate matching degree satisfies the preset coordinate matching degree.

4. The method according to claim 1, characterized in that, The step of acquiring several key defect points in the initial defect region of the image to be detected includes: Acquire the image to be detected; The image to be detected is input into a pre-trained target detection model to obtain the initial defect region output by the target detection model; Based on the image contrast between the initial defect region and the background region corresponding to the initial defect region, defect key points are extracted in the initial defect region, and the background region is the pixel region in the image to be detected other than the initial defect region.

5. The method according to claim 1, characterized in that, The step of acquiring several key defect points in the initial defect region of the image to be detected includes: Obtain the object to be detected in the image to be detected, and determine the region to be detected in the image to be detected based on the object to be detected; The region to be detected is segmented according to a preset sliding window to obtain several sub-images to be detected; Target detection is performed on each sub-image to be detected, and several key defect points in the initial defect region of each sub-image to be detected are obtained.

6. A defect detection device, characterized in that, include: The acquisition module is used to acquire several key defect points in the initial defect region of the image to be detected; Feature detection is performed on the image to be detected to obtain grayscale information and edge information of the image to be detected; a self-attention heatmap of the image to be detected is constructed based on the grayscale information and the edge information, the self-attention heatmap contains the heat value of each pixel in the image to be detected, and the heat value is used to characterize the attention level of each pixel in the image to be detected; An aggregation module is used to aggregate pixels within a preset range corresponding to each defect key point based on the feature information of the pixels within that range, to obtain an aggregated region corresponding to each defect key point. The feature information of the pixels includes grayscale information and thermal values. The step of aggregating pixels within a preset range corresponding to each defect key point based on the feature information of the pixels within that range to obtain an aggregated region corresponding to each defect key point includes: filtering pixels within a preset range corresponding to each defect key point based on the grayscale information and thermal values ​​of each defect key point to obtain filtered pixels; and aggregating the filtered pixels within the preset range corresponding to each defect key point to obtain an aggregated region corresponding to each defect key point. The merging module is used to merge the aggregated region corresponding to each defect key point and the initial defect region where each defect key point is located to obtain a candidate defect region. The splicing module is used to splice two candidate defect regions that match the location information of each candidate defect region to obtain the defect detection result.

7. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images