Sample generation method, defect detection method, electronic device, storage medium and program product

Abstract

The invention provides a sample generation method, a defect detection method, electronic equipment, a storage medium and a program product, which can be applied to the technical fields of image processing, defect detection, industrial quality inspection and the like. The sample generation method comprises the steps that a sample image and a label are acquired, and the label is used for identifying the defect position and the defect type of at least one sample defect in the sample image; according to the respective defect position of the at least one sample defect, adjusting the sizes of a plurality of first image blocks obtained based on sample image division to obtain a plurality of second image blocks; and according to the respective image quality evaluation values of the plurality of second image blocks, determining at least one target image block with a defect type in the plurality of second image blocks, and taking the plurality of second image blocks and the at least one target image block corresponding to each defect type as samples.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of image processing, defect detection, industrial quality inspection, etc. More specifically, the present disclosure relates to a sample generation method, a defect detection method, an electronic device, a storage medium, and a program product. Background Art

[0002] In industrial production scenarios, in order to control the quality of products, it is necessary to perform defect detection on the products. Defect detection refers to the detection of defects on the surface of products. For example, defects such as spots, pits, scratches, color differences, and defects on the surface of products are detected. Summary of the Invention

[0003] In view of this, the present disclosure provides a sample generation method, a defect detection method, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of the present disclosure, there is provided a sample generation method, including: obtaining a sample image and a label, where the label is used to identify the defect positions and defect types of at least one sample defect in the sample image; adjusting the sizes of a plurality of first image blocks obtained by dividing the sample image according to the defect positions of the at least one sample defect to obtain a plurality of second image blocks; and determining at least one target image block having the defect type from the plurality of second image blocks according to the image quality evaluation values of the plurality of second image blocks, and using the plurality of second image blocks and at least one target image block corresponding to each defect type as samples.

[0005] According to one aspect of the present disclosure, there is provided a defect detection method, including: inputting an image to be detected into a defect detection model to obtain defect detection information, where the defect detection model is trained using samples generated by the sample generation method, the image to be detected is an image of a product produced by a target process flow, and the defect detection information includes the defect positions and defect types of at least one target defect; and screening the at least one target defect according to a constraint rule to obtain a defect detection result, where the constraint rule is used to constrain the defect distribution of products produced by the target process flow.

[0006] According to another aspect of the present disclosure, there is provided an electronic device, including: one or more processors; a memory for storing one or more instructions, where when the one or more instructions are executed by the one or more processors, the one or more processors are caused to implement the method as described in the present disclosure.

[0007] According to another aspect of the present disclosure, there is provided a computer-readable storage medium having executable instructions stored thereon, and when the executable instructions are executed by a processor, the processor is caused to implement the method as described in the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a computer program product, the computer program product including computer-executable instructions, and the computer-executable instructions are used to implement the method as described in the present disclosure when executed. Description of the Drawings

[0009] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0010] Figure 1 Schematically showing a system architecture to which a sample generation method and a defect detection method can be applied according to an embodiment of the present disclosure;

[0011] Figure 2 Schematically showing a flowchart of a sample generation method according to an embodiment of the present disclosure;

[0012] Figure 3A Schematically showing an example diagram of a process of adjusting the sizes of a plurality of first image blocks obtained by dividing a sample image according to the defect positions of at least one sample defect to obtain a plurality of second image blocks;

[0013] Figure 3B Schematically showing an example diagram of a process of adjusting the size of an image block to be adjusted according to the defect position to obtain an adjusted image block;

[0014] Figure 4 Schematically showing an example diagram of a process of determining the number of target image blocks according to an embodiment of the present disclosure;

[0015] Figure 5 Schematically showing an example diagram of a process of determining at least one target image block having a defect type among a plurality of second image blocks according to the image quality evaluation values of the plurality of second image blocks according to an embodiment of the present disclosure;

[0016] Figure 6 Schematically showing a flowchart of a defect detection method according to an embodiment of the present disclosure;

[0017] Figure 7 Schematically showing an example diagram of a defect detection process according to an embodiment of the present disclosure;

[0018] Figure 8 Schematically showing an example diagram of an optimization process of a defect detection model according to an embodiment of the present disclosure;

[0019] Figure 9 Schematically shows an exemplary schematic diagram of a defect detection process according to another embodiment of the present disclosure;

[0020] Figure 10 Schematically shows a block diagram of a sample generation device according to an embodiment of the present disclosure;

[0021] Figure 11 Schematically shows a block diagram of a defect detection device according to an embodiment of the present disclosure; and

[0022] Figure 12 Schematically shows a block diagram of an electronic device suitable for implementing a sample generation method and a defect detection method according to an embodiment of the present disclosure. Detailed implementation manners

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

[0024] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0027] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties. Moreover, for the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, etc., all comply with relevant laws, regulations, and standards, adopt necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or reject.

[0028] In one example, the defect detection method may include a defect detection method based on traditional image processing and a defect detection method based on deep learning. The defect detection method based on traditional image processing refers to a method that uses technologies such as edge detection, morphological processing, and texture analysis to detect defects existing on the product surface. The defect detection method based on deep learning refers to a method that uses a deep neural network to detect defects existing on the product surface.

[0029] However, since the defect detection method based on traditional image processing requires rich professional knowledge and parameter adjustment; the defect detection method based on deep learning needs to build a model based on a large number of defect samples, and it is difficult to obtain images with defect labels. Moreover, due to the diverse surface morphologies and numerous defect types of the product, the lack of defect samples makes it difficult to apply the model.

[0030] In another example, defects can be manually created on a real product. Although this method is simple and direct, and the generated defects are relatively close to the actual situation, however, this method causes permanent damage to the product and is costly for high-value products; and since the defects of this method are generated by uncontrolled factors, they may not fully conform to the actual situation. In addition, defects can also be manually simulated on real images, but this method requires operators to have high skills and has a slow generation speed.

[0031] In yet another example, the number of samples can be increased through data augmentation and defect generation, but this method cannot simulate the shape and complex texture of defects. In addition, the dependence of the algorithm on samples can be reduced, but this method requires different algorithm designs for different scenarios and it is difficult to have generality.

[0032] To this end, the present disclosure provides a sample generation method, a defect detection method, an electronic device, a storage medium, and a program product, which can be applied to technical fields such as image processing, defect detection, and industrial quality inspection. The sample generation method includes: obtaining a sample image and a label, where the label is used to identify the defect positions and defect types of at least one sample defect in the sample image; adjusting the sizes of a plurality of first image blocks obtained by dividing the sample image according to the defect positions of at least one sample defect respectively to obtain a plurality of second image blocks; and determining at least one target image block with a defect type among the plurality of second image blocks according to the image quality evaluation values of the plurality of second image blocks respectively, and using the plurality of second image blocks and at least one target image block corresponding to each defect type as samples.

[0033] Figure 1 Schematically shows a system architecture to which the sample generation method and the defect detection method according to the embodiments of the present disclosure can be applied. It should be noted that Figure 1 The illustration is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] As Figure 1 shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between different devices.

[0035] It should be noted that the sample generation method and the defect detection method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the sample generation device and the defect detection device provided by the embodiments of the present disclosure can generally be arranged in the server 105.

[0036] Alternatively, the sample generation method and the defect detection method provided by the embodiments of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the sample generation device and the defect detection device provided by the embodiments of the present disclosure can also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0037] It should be understood that Figure 1 the numbers of the terminal devices, the network, and the server in

[0038] It should be noted that the serial numbers of the respective operations in the following methods are only used as representations of the operations for description purposes and should not be regarded as indicating the execution order of the respective operations. Unless explicitly stated, the method does not need to be executed exactly in the order shown.

[0039] The above has described the system architecture to which the sample generation method and the defect detection method provided by the present disclosure can be applied. Next, taking Figure 2 as an example, the sample generation process of the present disclosure will be further described.

[0040] Figure 2 FIG. schematically shows a flowchart of a sample generation method according to an embodiment of the present disclosure.

[0041] As Figure 2 shown, the sample generation method 200 includes operations S210 to S230.

[0042] In operation S210, a sample image and a label are obtained, where the label is used to identify the defect positions and defect types of at least one sample defect in the sample image respectively.

[0043] In operation S220, according to the defect positions of at least one sample defect respectively, the sizes of a plurality of first image blocks obtained by dividing the sample image are adjusted to obtain a plurality of second image blocks.

[0044] In operation S230, according to the image quality evaluation values of the plurality of second image blocks respectively, at least one target image block having a defect type is determined among the plurality of second image blocks, and the plurality of second image blocks and at least one target image block corresponding to each defect type are used as samples.

[0045] The sample image is an entire unprocessed image, and the sample image may include at least one sample defect. A sample defect refers to a feature or area in the sample image that does not conform to the normal standard or expectation. In one example, in the industrial production field, the sample image may be an image for detecting product surface defects, and the sample defect may be an object obtained by performing defect detection on the sample image.

[0046] The label can be used to identify the defect positions and defect types of at least one sample defect respectively. The defect position may refer to the specific position information of the sample defect in the sample image, for example, it can be represented by coordinates, a region range, etc. The defect type refers to the description and classification of the nature and type of the sample defect. For example, the defect type may include water droplets, water bubbles, holes, cracks, bulges, stains, and scratches, etc.

[0047] The specific form of the label can be configured according to business requirements and is not limited herein. For example, the label can be text, a table, etc. For example, taking the sample image as an image of a circuit board product, the circuit board product may include Defect 1 and Defect 2, then the label can be "Defect 1: The defect location is (10, 20) to (15, 25), and the defect type is a scratch; Defect 2: The defect location is (30, 40) to (30, 50), and the defect type is a stain".

[0048] After obtaining the sample image, the sample image can be divided to obtain a plurality of first image blocks. The first image block can refer to a plurality of image regions obtained by correspondingly performing network block on the sample image and the label. These image regions can have the same size or different sizes, which is not limited herein. The specific size can be determined according to the resolution of the sample image. For example, taking the resolution of the sample image as 100*100 as an example, in the case of dividing the sample image with a size of 10*10, 10 rows * 10 columns of first image blocks can be obtained.

[0049] After obtaining a plurality of first image blocks, the sizes of the plurality of first image blocks can be adjusted according to the defect location of each sample defect, so as to avoid the same sample defect being divided into different image blocks, obtain a plurality of adjusted image blocks, and determine the adjusted image block including the sample defect as the second image block. The second image block is an image region obtained by adjusting the size of the first image block according to the defect location of at least one sample defect on the basis of the first image block.

[0050] In one example, at least two first image blocks including the same sample defect can be determined from the plurality of first image blocks according to the defect location of the sample defect. By adjusting the sizes of the at least two first image blocks, at least two adjusted image blocks are obtained, so that the sample defect is located in one adjusted image block, and the other adjusted image blocks except the adjusted image block including the sample defect no longer include the sample defect.

[0051] For example, taking the resolution of the sample image as 100*100, dividing the sample image into 10 rows * 10 columns of first image patches with a size of 10*10, and the labels being "Defect 1: The defect location is (12, 12) to (18, 28), and the defect type is scratch; Defect 2: The defect location is (22, 32) to (28, 38), and the defect type is stain" as an example: For Defect 1, it can be determined that the first image patches including Defect 1 are the first image patch located in the 2nd row and 2nd column and the first image patch located in the 3rd row and 2nd column. Therefore, the sizes of these two first image patches can be adjusted so that Defect 1 is only located in one adjusted image patch; for Defect 2, it can be determined that the first image patch including Defect 2 is the first image patch located in the 4th row and 3rd column. Since Defect 2 is already only located in one image patch, there is no need to adjust the size of this first image patch.

[0052] After obtaining multiple second image patches, image quality assessment (IQA) can be performed on the multiple second image patches to obtain the image quality assessment values of each of the multiple second image patches. The specific image quality assessment method can be configured according to actual business requirements and is not limited here. For example, the image quality assessment method can include the image quality assessment method based on no-reference image quality assessment (NR-IQA) metrics, the image quality assessment method based on full-reference image quality assessment (FR-IQA) metrics, and the image quality assessment method based on reduced-reference image quality assessment (RR-IQA) metrics, etc.

[0053] After obtaining the image quality assessment values of each of the multiple second image patches, at least one target image patch with a defect type can be determined. The target image patch can refer to an image patch with relatively high image quality that can better reflect the characteristics of the sample defect. For example, if the image quality assessment values of three second image patches are 80, 90, and 70 in sequence, then the second image patch with an image quality assessment value of 90 can be selected as the target image patch. On this basis, at least one target image patch with different defect types can be used as a sample.

[0054] In one example, each second image block has a type of defect, and multiple second image blocks can have multiple types of defects. For multiple second image blocks with the same type of defect, at least one target image block with the type of defect can be determined from the multiple second image blocks according to the respective image quality evaluation values of each second image block. For different types of defects, the above operations are repeatedly performed until at least one target image block corresponding to each type of defect is obtained. On this basis, multiple second image blocks and at least one target image block corresponding to each type of defect can be used as samples, so that the number of image blocks with different types of defects in the samples is balanced. The balanced number means that the number of image blocks with different types of defects is similar.

[0055] For example, among 12 second image blocks, 3 second image blocks have defect type 1, 4 second image blocks have defect type 2, and 5 second image blocks have defect type 3. For defect type 1, 2 target image blocks with defect type 1 can be determined from the 3 second image blocks according to the respective image quality evaluation values of the 3 second image blocks; for defect type 2, 1 target image block with defect type 2 can be determined from the 4 second image blocks according to the respective image quality evaluation values of the 4 second image blocks.

[0056] On this basis, the original 3 second image blocks with defect type 1 and the determined 2 target image blocks with defect type 1, the original 4 second image blocks with defect type 2 and the determined 1 target image block with defect type 2, and the original 5 second image blocks with defect type 3 can be used as samples. The samples can be used for subsequent training of the defect detection model. It should be noted that the sample generation method provided in the present disclosure can be applied to various fields and is not limited here. For example, in the field of product production, the sample image can be an image of the produced product, and the product can be, for example, a screen. Alternatively, in the field of medical analysis, the sample image can be a medical image. Alternatively, in the field of intelligent transportation, the sample image can be a road marking image. Alternatively, in the field of character recognition, the sample image can be a rare character or a handwritten text image.

[0057] According to an embodiment of the present disclosure, an adaptive block division mechanism for obtaining a second image block by adjusting the size of a first image block obtained by dividing a sample image can reasonably divide the sample image according to the defect positions of sample defects, realizing refined processing of the image blocks, ensuring that the obtained second image block can more effectively focus on the sample defects, and being beneficial to improving the rationality of subsequent sample generation. On this basis, by screening out target image blocks with defect types according to the image quality evaluation value to construct samples, the image blocks including sample defects can be made richer, which helps to better utilize small sample defect resources and makes the defined boundaries between different sample defects clearer, helping to enhance the generalization of the defect detection model trained using the samples subsequently, and further helping to improve the efficiency and accuracy of defect detection.

[0058] The above describes the sample production process provided by the present disclosure. Next, in combination with Figure 3A and Figure 3B , an exemplary description will be given of the process of adjusting the size of the first image block to obtain the second image block.

[0059] Figure 3A FIG. schematically shows an example diagram of a process of adjusting the sizes of a plurality of first image blocks obtained by dividing a sample image according to the defect positions of at least one sample defect respectively to obtain a plurality of second image blocks according to an embodiment of the present disclosure.

[0060] As Figure 3A shown, in 300A, after obtaining the sample image 301 and the label, the sample image 301 and the label can be divided correspondingly to obtain a plurality of first image blocks 302. For example, the sample image 301 can be divided into 5 rows * 5 columns of first image blocks 302. In one example, for each sample defect, at least two image blocks to be adjusted can be determined among the plurality of first image blocks 302. On this basis, according to the defect position, the sizes of each image block to be adjusted can be adjusted respectively so that the sample defect is located in an adjusted image block, and the adjusted image block including the sample defect is used as the second image block.

[0061] Taking the sample image 301 including sample defects 303, 304, 305, 306, and 307 as an example, the process of determining at least two image blocks to be adjusted for each sample defect respectively and adjusting the image blocks to be adjusted to obtain adjusted image blocks will be described below.

[0062] It should be noted that the sample defect 303 and the sample defect 307 are defects of the same defect type, the sample defect 304 and the sample defect 306 are defects of the same defect type, and the defect types of the sample defect 303, the sample defect 304, and the sample defect 305 are different from each other.

[0063] For the sample defect 303, according to the defect position of the sample defect 303 in the label, the image block to be adjusted corresponding to the sample defect 303 can be determined as the first image block located in the second row and the second column. For the sample defect 307, according to the defect position of the sample defect 307 in the label, the image block to be adjusted corresponding to the sample defect 307 can be determined as the first image block located in the fifth row and the third column.

[0064] Since the image block to be adjusted corresponding to the sample defect 303 and the image block to be adjusted corresponding to the sample defect 307 only include one first image block, there is no need to perform size adjustment anymore, and the image block to be adjusted can be directly determined as the second image block.

[0065] For the sample defect 304, according to the defect position of the sample defect 304 in the label, the image blocks to be adjusted corresponding to the sample defect 304 can be determined as the first image block located in the second row and the third column and the first image block located in the third row and the third column. In this case, the size of the first image block in the second row and the third column can be reduced, and the size of the first image block in the third row and the third column can be enlarged, so that the sample defect 304 is located in an adjusted image block (i.e., the third row and the third column), and this image block can be used as the second image block.

[0066] For the sample defect 306, according to the defect position of the sample defect 306 in the label, the image blocks to be adjusted corresponding to the sample defect 306 can be determined as the first image block located in the fourth row and the first column and the first image block located in the fifth row and the first column. In this case, the size of the first image block in the fourth row and the first column can be reduced, and the size of the first image block in the fifth row and the first column can be enlarged, so that the sample defect 306 is located in an adjusted image block (i.e., the fifth row and the first column), and this image block can be used as the second image block.

[0067] For the sample defect 305, according to the defect position of the sample defect 305 in the label, the image blocks to be adjusted corresponding to the sample defect 305 can be determined as the first image block located in the third row and the fourth column and the first image block located in the third row and the fifth column. In this case, the size of the first image block in the third row and the fourth column can be reduced, and the size of the first image block in the third row and the fifth column can be enlarged. So that the sample defect 305 is located in an adjusted image block (i.e., the third row and the fifth column), and this image block can be used as the second image block.

[0068] According to an embodiment of the present disclosure, by determining at least two to-be-adjusted image blocks containing sample defects among a plurality of first image blocks according to the defect position, comprehensive coverage of the area with sample defects is ensured. On this basis, by adjusting the size of each to-be-adjusted image block according to the defect position, the sample defect can be completely and accurately contained in an adjusted image block to form a second image block. This adjustment method can avoid the influence of cross-block labels, not only improving the adaptability of the image block to the defect area, but also enhancing the pertinence and representativeness of the image block, and a high-quality second image block containing the sample defect can be obtained, which is equivalent to expanding the dataset and improving the small-sample problem.

[0069] Figure 3B Schematically shows an example diagram of adjusting the size of a to-be-adjusted image block according to the defect position to obtain an adjusted image block according to an embodiment of the present disclosure.

[0070] As Figure 3B shown, in 300B, taking the sample defect 306 as an example, the process of adjusting the size of the to-be-adjusted image block according to the defect position to obtain an adjusted image block is illustrated. The to-be-adjusted image blocks corresponding to the sample defect 306 are the first image block located in the 4th row and 1st column of the sample image 301 and the first image block located in the 5th row and 1st column of the sample image 301.

[0071] The plurality of first image blocks can be obtained by dividing the sample image based on a preset size. The preset size refers to the size of the image block preset when dividing the sample image. In one example, the preset size can be determined according to the resolution of the sample image. For example, the preset size can be w*h, and w and h can be the same or different, which is not limited herein. The defect position can include at least two edge point coordinates. For example, the defect position of the sample defect 306 can include the edge point coordinates (x1, y1) of point A1 and the edge point coordinates (x2, y2) of point A2.

[0072] In one example, the defect size can be determined based on at least two edge point coordinates, and the to-be-adjusted size for each to-be-adjusted image block can be determined according to the preset size and the defect size. The defect size refers to the size of the defect calculated based on the edge point coordinates of the defect position. The to-be-adjusted size refers to the size used to adjust the size of the image block determined according to the defect size and the preset size, with the aim of enabling the sample defect to be completely located in an adjusted image block.

[0073] Specifically, the dimension to be adjusted can be determined according to the adjustment direction. For example, the defect dimension of the sample defect 306 can be determined as (x2 - x1) * (y2 - y1). Since the adjustment direction is the vertical direction, the preset dimension in the vertical direction and the defect dimension in the vertical direction can be used to determine the dimension to be adjusted, and the dimension to be adjusted d = (y2 - y1 - h) is obtained.

[0074] After obtaining the dimension to be adjusted, the first image block located at the 4th row and 1st column of the sample image 301 can be reduced by the dimension to be adjusted, and the first image block located at the 5th row and 1st column of the sample image 301 can be enlarged by the dimension to be adjusted, so that the sample defect 306 is located in an adjusted image block 308.

[0075] According to the embodiments of the present disclosure, by using the coordinates of at least two edge points of the defect position and the preset dimension for dividing the first image block to determine the defect dimension, and then calculating the dimension to be adjusted based on the preset dimension and the defect dimension, it can be ensured that the size of each image block to be adjusted matches the actual size of the sample defect, so that the sample defect can be completely and accurately contained in an adjusted image block, avoiding the problem of loss or incompleteness of defect information caused by unreasonable image block division, improving the adaptability and coverage accuracy of the image block to the area of the sample defect, realizing the precise adjustment of the image block size, helping to improve the representativeness of the subsequent generated samples, and further helping to improve the accuracy of defect detection.

[0076] The above describes the process of how to adjust the size of the first image block to obtain the second image block provided by the present disclosure. Next, in combination with Figure 4 , an exemplary description will be given of how to determine the number of target image blocks.

[0077] Figure 4 FIG. shows an exemplary schematic diagram of the process for determining the number of target image blocks according to an embodiment of the present disclosure.

[0078] In one example, each second image block has one type of defect, and multiple second image blocks can have multiple types of defects. For example, for 10 second image blocks, 1 second image block can have defect type 1, 2 second image blocks can have defect type 2, and 7 second image blocks can have defect type 3.

[0079] Determine at least one first defect type and at least one second defect type according to the number of second image blocks with different defect types. The first defect type may refer to the defect type with a higher occurrence frequency or quantity, and the second defect type may refer to the defect type with an occurrence frequency or quantity lower than that of the first defect type. The number of second image blocks with the first defect type among multiple second image blocks is M, and the number of second image blocks with the second defect type is N. Both M and N are positive integers and M > N. For example, the defect type with a larger number of second image blocks with the defect type among multiple second image blocks can be determined as the first defect type (i.e., the majority class), and the defect type with a smaller number of second image blocks with the defect type among multiple second image blocks can be determined as the second defect type (i.e., the minority class).

[0080] As Figure 4 shown, in 400, taking the sample defects of a sample image including three defect types as an example, the process of determining the number of target image blocks is described.

[0081] The sample image includes sample defects 401, 402, and 406 belonging to defect type 1, sample defects 403 and 405 belonging to defect type 2, and sample defect 404 belonging to defect type 3. It can be seen that the number of second image blocks with defect type 1 is 3, the number of second image blocks with defect type 2 is 2, and the number of second image blocks with defect type 3 is 1. Therefore, defect type 1 can be determined as the first defect type 407, and defect types 2 and 3 can be determined as the second defect type 408.

[0082] After determining the first defect type 407 and the second defect type 408, determine whether there is data imbalance according to the number of second image blocks with the first defect type 407 and the number of second image blocks with the second defect type 408. Data imbalance refers to the existence of defects with a small quantity. The determination method of the imbalance coefficient is shown in the following formula (1).

[0083]

[0084] where n l is the number of second image blocks with the first defect type 407, and n s is the number of second image blocks with the second defect type 408. Define d th as the maximum threshold of the defect type imbalance degree. In the case of d < d th , it indicates the existence of data imbalance.

[0085] After determining the existence of data imbalance, for each second defect type 408, the number of synthetic samples to be generated for the minority class can be determined according to the number of second image patches with the first defect type 407 and the number of second image patches with the second defect type 408, that is, the first number of target image patches for the second defect type 408. For example, the number of target image patches can be the difference between the number of second image patches with the first defect type 407 and the number of second image patches with the second defect type 408. The determination method of the first number can be shown as the following formula (2).

[0086] G=(n l -n s )×β (2);

[0087] Where β∈[0, 1] is a parameter used to specify the required balance level after generating synthetic data, and G represents the first number. In the case of β = 1, it means creating a completely balanced data set after the generalization process. At this time, d th = 1.

[0088] For example, since the number of second image patches with defect type 1 is 3 and the number of second image patches with defect type 2 is 2, it can be determined that the number of target image patches with defect type 2 is 1. Alternatively, since the number of second image patches with defect type 1 is 3 and the number of second image patches with defect type 1 is 1, it can be determined that the number of target image patches with defect type 1 is 2.

[0089] According to the embodiments of the present disclosure, by analyzing the number of second image patches with different defect types to determine the first defect type as the majority class and the second defect type as the minority class, the distribution characteristics of defect types in the samples can be effectively identified, realizing the classification of defect types. On this basis, by determining the first number of target image patches for the second defect type according to the number of second image patches with the first defect type and the number of second image patches with the second defect type, the number of target image patches can be dynamically adjusted, ensuring the representativeness of various defect types in the samples, avoiding the problem of sample imbalance caused by the difference in the number of defect types, and thus helping to improve the generalization ability of the model and the comprehensiveness and accuracy of defect detection.

[0090] The above describes how to determine the number of target image patches provided by the present disclosure. Next, in combination with Figure 5 , an exemplary description will be given of how to determine at least one target image patch with a defect type among multiple second image patches.

[0091] Figure 5Schematically illustrated is an example schematic diagram of a process of determining at least one target image block with a defect type among a plurality of second image blocks according to the image quality evaluation values of the respective second image blocks according to an embodiment of the present disclosure.

[0092] As Figure 5 shown, in 500, for at least two second image blocks having the same second defect type, the image quality evaluation differences between each two second image blocks can be determined to obtain a plurality of image quality evaluation differences.

[0093] For example, the second image blocks having the same second defect type include the second image block 501, the second image block 502, and the second image block 503. The image quality evaluation difference 504 between the second image block 501 and the second image block 502 can be calculated, the image quality evaluation difference 505 between the second image block 502 and the second image block 503 can be calculated, and the image quality evaluation difference 506 between the second image block 501 and the second image block 503 can be calculated.

[0094] The image quality evaluation difference refers to a numerical value used to measure the quality of the second image block. The image quality evaluation difference can be configured according to actual business requirements and is not limited herein. The image quality evaluation difference can be obtained by evaluating each two second image blocks based on image quality evaluation metrics. In one example, the image quality evaluation metrics can include at least one of the following: brightness metric, contrast metric, structure metric, color metric, noise metric, and sharpness metric.

[0095] The brightness metric can be obtained by calculating the average gray value of all pixels in the image and is used to measure the average brightness level of the image. The contrast metric can be obtained by calculating the difference between the maximum gray value and the minimum gray value of the image and is used to measure the difference degree between the bright and dark regions in the image. The structure metric can be evaluated by methods such as structural similarity and is used to measure the integrity of the structural information in the image. The color metric can be obtained by calculating color accuracy, saturation, or white balance error and is used to measure the color quality of the image. The noise metric can be obtained by calculating the signal-to-noise ratio or noise power of the image and is used to measure the noise level in the image. The sharpness metric can be obtained by calculating the gradient information or frequency distribution of the image and is used to measure the sharpness of the image.

[0096] The specific evaluation method can be configured according to actual business requirements and is not limited herein. For example, the evaluation method can include at least one of the following: full-reference image quality evaluation, weak-reference quality evaluation, and no-reference image quality evaluation.

[0097] Full-reference image quality assessment may refer to the need for a reference image as a reference to evaluate the quality of an image by comparing the differences between the reference image and the image to be evaluated. Weak-reference quality assessment may refer to the need for a reference image as a reference to evaluate the quality of an image by obtaining partial feature information from the reference image and based on the differences between the partial feature information and the image to be evaluated. For example, full-reference image quality assessment metrics and weak-reference quality assessment metrics may include at least one of the following: Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM).

[0098] No-reference image quality assessment may refer to the need to analyze the characteristics of the image to be evaluated itself without a reference image as a reference. For example, image quality evaluation metrics may include at least one of the following: contrast metric, sharpness metric, noise metric, brightness metric, and color metric. For example, the contrast can be evaluated by the variance of the entire image. Sharpness-related metrics may include sharpness and false color. For example, an edge detection operator can be used to evaluate sharpness. Alternatively, the false color can be evaluated by the UV difference in the texture area.

[0099] Noise-related metrics may include dynamic noise and static noise. For example, the variance of the luminance component can be used to evaluate dynamic noise. Alternatively, the variance of the 18-degree gray wall surface part in the image can be used to evaluate static noise. Brightness-related metrics may include brightness, AE convergence, local overexposure or underexposure. For example, the average brightness of the entire image can be used to evaluate brightness. Alternatively, the AE convergence can be evaluated based on the change in brightness values within a preset time period. Alternatively, local overexposure or underexposure can be evaluated by counting the number of pixels with too high or too low heights. Color-related metrics may include white balance and saturation. For example, the chromaticity value of the entire image can be used to evaluate white balance. Alternatively, the white balance can be evaluated based on the mean saturation of each pixel in the entire image.

[0100] After obtaining multiple image quality evaluation differences, for each second image block, the mean of the multiple image quality evaluation differences corresponding to the second image block can be determined, and this mean is determined as the image quality evaluation value of the second image block. The image quality evaluation value refers to the mean of the multiple image quality evaluation values corresponding to each second image block and can be used to measure the comprehensive difference in the quality of the image block.

[0101] For example, taking the image quality evaluation difference 504 as 90, the image quality evaluation difference 505 as 80, and the image quality evaluation difference 506 as 70 as an example, the image quality evaluation differences related to the second image block 501 are the image quality evaluation difference 504 and the image quality evaluation difference 506, and the average of the two is 80. Therefore, the image quality evaluation value of the second image block 501 is 80; the image quality evaluation differences related to the second image block 502 are the image quality evaluation difference 504 and the image quality evaluation difference 505, and the average of the two is 85. Therefore, the image quality evaluation value of the second image block 502 is 85; the image quality evaluation differences related to the second image block 503 are the image quality evaluation difference 505 and the image quality evaluation difference 506, and the average of the two is 75. Therefore, the image quality evaluation value of the second image block 503 is 75.

[0102] After obtaining the respective image quality evaluation values of each second image block, the multiple image quality evaluation values can be sorted according to the magnitude relationship of the image quality evaluation values to obtain the sorted image quality evaluation values. On this basis, the second image blocks corresponding to the first number of the top image quality evaluation values in the sorted image quality evaluation values can be determined as the target image blocks. Samples are generated based on the multiple second image blocks and the first number of target image blocks with each second defect type.

[0103] For example, if the image quality evaluation values are arranged from largest to smallest, the first number of the top image quality evaluation values can be the larger image quality evaluation values. By using the target image blocks corresponding to these larger image quality evaluation values as new data to supplement the samples, the images of sample defects can be made more abundant, and it is beneficial to enhance the generalization of the model.

[0104] Continuing with the example where the image quality evaluation value of the second image block 501 is 80, the image quality evaluation value of the second image block 502 is 85, and the image quality evaluation value of the second image block 503 is 75, the sorting can be performed according to the magnitude relationship of the image quality evaluation values. Since 85 > 80 > 75, the second image blocks can be arranged as the second image block 502, the second image block 501, and the second image block 503. On this basis, taking the first number as 2 as an example, the second image block 502 and the second image block 501 can be determined as the target image blocks.

[0105] After determining the target image blocks, oversampling can be performed on the second image block 502 and the second image block 501 to increase the number of minority class samples, so that the number of image blocks with the second defect type is close to the number of image blocks with the first defect type. For example, the target image blocks can be repeated, that is, two second image blocks 502, two second image blocks 501, and the second image block 503 are used as the training samples for this second defect type.

[0106] According to an embodiment of the present disclosure, for a second image block having a second defect type, by calculating the difference in image quality evaluation between at least two second image blocks and taking the average value thereof as the image quality evaluation value, a quantitative evaluation of the quality of each second image block is achieved. This not only considers the objective quality index of the second image block but also avoids the limitation of a single index by means of the average difference, making the evaluation result more comprehensive and reliable. By sorting the second image blocks based on the image quality evaluation value and selecting the first number of image blocks with higher rankings as the target image blocks, high-quality image blocks can be screened, ensuring the diversity and representativeness of the defect types in the sample and avoiding interference from low-quality image blocks in subsequent analysis. On this basis, by combining the number of target image blocks of the first defect type and the second defect type, the generated sample can better balance the distribution of different defect types, improve the representativeness and quality of the sample, and provide high-quality and balanced data support for the training of subsequent defect detection models.

[0107] In a specific example, taking the structural similarity index as the image quality evaluation index, the process of determining the difference in image quality evaluation between at least two second image blocks having the second defect type is described.

[0108] After obtaining at least two second image blocks, based on the sliding window method, each of the at least two second image blocks can be divided to obtain multiple sub-image blocks of each second image block. The sliding window method refers to moving a window with a fixed or variable width on the image and processing the image area within the window. The width of the window can be configured according to actual business requirements and is not limited herein. In one example, the width of the window can be configured according to the size of the second image block.

[0109] For example, if the second image block is 100*100 pixels and the width of the window is 10*10 pixels, the window can start moving from the upper left corner of the second image block, move 10 pixels to the right each time, move down 10 pixels when reaching the end of the row, and start from the left again, thereby covering the entire second image block, and multiple non-overlapping sub-image blocks can be obtained.

[0110] Alternatively, if the second image block is 100*100 pixels and the width of the window is 10*10 pixels, the window can start moving from the upper left corner of the second image block, move 5 pixels to the right each time, move down 5 pixels when reaching the end of the row, and start from the left again, thereby covering the entire second image block, and multiple overlapping sub-image blocks can be obtained.

[0111] After obtaining multiple sub-image blocks of each second image block, for every two second image blocks, based on an image quality evaluation metric, the sub-image quality evaluation differences of the sub-image blocks located at the same position of the two second image blocks can be calculated respectively, and the sub-image quality evaluation differences of each of multiple positions can be obtained. On this basis, the mean value of the sub-image quality evaluation differences of each of the multiple positions can be determined, and this mean value can be used as the image quality evaluation difference between the two second image blocks.

[0112] For example, the structural similarity index can include a luminance index, a contrast index, and a structure index, and the image quality evaluation difference between every two second image blocks can be calculated using the following formulas (4) to (6).

[0113]

[0114] SSIM(x, y) = [l(x, y) α ·c(x, y) β ·s(x, y) γ (6);

[0115] where x represents the data of each sub-image block in the first second image block, y represents the data of each sub-image block in the second second image block, μ x represents the mean of x, μ y represents the mean of y, σ x represents the variance of x, σ y represents the variance of y, σ xy represents the covariance between x and y, c 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 , c 3 =c 2 / 2 represents three constants used to avoid a denominator of 0, k 1 is 0.01, k 2 is 0.03, L represents the range of image pixel values, i.e., 2 B -1, l(x, y) represents the similarity between x and y in terms of luminance, c(x, y) represents the similarity between x and y in terms of contrast, s(x, y) represents the similarity between x and y in terms of structure, SSIM(x, y) represents the image quality evaluation difference between x and y, and α, β, and γ are three constants.

[0116] In the case where α = β = γ = 1, the image quality evaluation difference between every two second image blocks can be calculated using the following formula (7).

[0117]

[0118] It should be noted that the value range of the image quality evaluation difference is [-1, 1]. When the two second image blocks are exactly the same, the image quality evaluation difference is 1.

[0119] According to the embodiments of the present disclosure, since the sliding window method can finely divide the second image block to generate multiple sub-image blocks, realizing the refined processing of the local features of the image, it can capture the detailed information in the image and provide a richer data basis for subsequent quality evaluation. On this basis, by comparing the sub-image blocks at the same position in different second image blocks and using the image quality evaluation index to determine the image quality evaluation difference, the quality advantages and disadvantages between different image blocks can be effectively identified, providing a basis for screening high-quality image blocks.

[0120] The above are only exemplary embodiments, but not limited thereto. Other sample generation methods known in the art may also be included as long as the distribution of different defect types can be balanced.

[0121] The sample generation method provided by the present disclosure has been described above. Next, taking Figure 6 as an example, the defect detection process of another embodiment of the present disclosure will be further described.

[0122] Figure 6 The flowchart of the defect detection method according to the embodiments of the present disclosure is schematically shown.

[0123] As Figure 6 shown, the defect detection method 600 includes operations S610 to S620.

[0124] In operation S610, the image to be detected is input into the defect detection model to obtain defect detection information. Among them, the defect detection model is trained using the samples generated by the sample generation method, the image to be detected is an image of a product produced by the target process flow, and the defect detection information includes the defect positions and defect types of at least one target defect respectively.

[0125] In operation S620, according to the constraint rules, at least one target defect is screened to obtain a defect detection result. Among them, the constraint rules are used to constrain the defect distribution of the products produced by the target process flow.

[0126] The image to be detected is the entire unprocessed image. The image to be detected and the sample image used to produce the sample are images of products produced using the same target process flow. After generating the sample using the sample generation method, the initial model can be trained using the sample to obtain a defect detection model for defect detection of the images of products produced by the target process flow. The initial model can be configured according to actual business requirements, which are not limited herein. For example, the initial model can include at least one of the following: Convolutional Neural Network (CNN), Autoencoder, Generative Adversarial Network (GAN), and Long Short-Term Memory (LSTM), etc.

[0127] After obtaining the image to be detected, the image to be detected can be input into the defect detection model to obtain defect detection information. The defect detection information can include the defect positions and defect types of at least one target defect respectively. The target defect refers to a feature or area in the image to be detected that does not meet the normal standard or expectation. The defect position can refer to the specific position information of the target defect in the image to be detected. For example, it can be represented by coordinates, area range, etc. The defect type refers to the description and classification of the nature and type of the target defect. For example, the defect type can include water droplets, water bubbles, holes, cracks, bulges, stains, and scratches, etc.

[0128] After obtaining the defect detection information, at least one target defect in the defect detection information can be screened based on the constraint rules to obtain the defect detection result. The constraint rules can be rules configured according to the process requirements or quality standards of the target process flow for constraining the defect distribution of products produced by the target process flow. The specific content of the constraint rules can be configured according to actual business requirements, which are not limited herein. For example, the constraint rules can be that no defects are allowed in certain areas, or the size of the defects cannot exceed a certain threshold, etc.

[0129] According to an embodiment of the present disclosure, by inputting an image to be detected into a defect detection model, since the defect detection model is trained using samples generated by a sample generation method, and the samples balance minority class samples and majority class samples by combining image adaptive block adjustment and image quality evaluation, the defect position and defect type information of the product can be quickly and accurately obtained, improving the accuracy of defect detection. On this basis, by combining constraint rules to screen the detected defects, since the constraint rules are optimized for the defect distribution characteristics of the product under the target process flow, abnormal detection results that do not conform to the actual process logic can be eliminated, thereby accurately outputting the final defect detection result. This dual mechanism of combining model detection and rule screening not only improves the accuracy of defect detection, but also makes the defect detection result closer to the actual process flow requirements, enhances the adaptability of the defect detection method to different process flows and product types, and improves the accuracy and reliability of defect detection.

[0130] In an embodiment of the present disclosure, after obtaining defect detection information, at least one target defect included in the defect detection information can be screened according to constraint rules to obtain a defect detection result. The following will take Figure 7 as an example to give an example of a defect detection process.

[0131] Figure 7 Schematically shows an example schematic diagram of a defect detection process according to an embodiment of the present disclosure.

[0132] As Figure 7 shown, in 700, after obtaining the image 701 to be detected, the image 701 to be detected can be input into the defect detection model 702 to obtain defect detection information 703. The defect detection information 703 includes the defect position 703_1 and defect type 703_2 of each of at least one target defect. After obtaining the defect detection information 703, at least one target defect included in the defect detection information 703 can be screened according to the constraint rule 704 to obtain a defect detection result 705.

[0133] The constraint rule 704 is used to screen and constrain the defect detection information 703 to ensure that the obtained defect detection result 705 meets specific process requirements. In one example, the constraint rule 704 can be configured according to constraint items for the target process flow. The constraint item refers to a specific parameter or condition used to define the constraint rule. For example, the constraint item can include at least one of the following: product model, defect type, defect area, defect number, defect type priority, and confidence level.

[0134] The product model refers to the specific product type or specification. The defect type refers to the type of defect. For example, the defect type can include water droplets, water bubbles, holes, cracks, bulges, stains, scratches, etc. The defect area refers to the size of the defect. The defect number refers to the number of defects allowed in the product. The defect type priority refers to the ranking of defect types according to the severity of the impact of the defect on product quality, which characterizes the degree of influence of the defect type on the product. The confidence level refers to the reliability degree of the defect detection result.

[0135] For different target process flows, corresponding constraint items can be selected and configured according to the specific requirements of the target process flow to form a constraint rule 704. On this basis, the defect detection information 703 output by the defect detection model 702 can be compared with the constraint rule 704 to screen out the target defects that meet the constraint rule 704 and obtain the defect detection result 705.

[0136] In one example, the constraint rules formed according to the product model and defect type can be: The allowed defect types for product model A are stains and scratches, and the allowed defect type for product model B is cracks.

[0137] In one example, the constraint rules formed according to the product model, defect type, and defect area can be: For product model A, when the defect type is stain and the defect area is greater than 5mm 2 it is P1, when less than 5mm 2 it is P3. When the defect type is scratch and the defect area is greater than 3mm 2 it is P1, when less than 3mm 2 it is P3. For product model B, when the allowed defect type is crack and the defect area is greater than 5mm 2 it is P1, when less than 5mm 2 it is P3.

[0138] In one example, the constraint rules formed according to the product model and defect number can be: The allowed defect number for product model A is 2, and the allowed defect number for product model B is 1. The constraint rules formed according to the defect type priority can be: The priority of scratch is higher than that of crack. The constraint rules formed according to the confidence level can be: Target defects with a confidence level higher than 95% are considered accurate.

[0139] According to the embodiments of the present disclosure, by combining the constraint rules with the constraint items of the target process flow, since the constraint items cover multiple dimensions such as product model, defect type, defect area, defect number, defect type priority, and confidence level, the introduction of these constraint items enables the defect detection process to adapt to the requirements of different process flows, flexibly adjust the judgment criteria for defects, and achieve precise screening and optimization of the defect detection results, thereby significantly improving the accuracy and reliability of defect detection.

[0140] In an embodiment of the present disclosure, during the defect detection process, the defect detection model can also be optimized. The following will take Figure 8 as an example to further illustrate the optimization process of the defect detection model.

[0141] Figure 8 FIG. schematically shows an example diagram of the optimization process of the defect detection model according to an embodiment of the present disclosure.

[0142] As Figure 8 shown, in 800, after inputting the image to be detected into the defect detection model 801 to obtain the defect detection information 802, the defect detection information 802 can be screened according to the constraint rule 803 to obtain the defect detection result 804. After obtaining the defect detection result 804, the defect detection result 804 can be evaluated to optimize the defect detection model 801 to adapt to the production line situation. For example, the defect annotation information 805 for the image to be detected can be obtained, and the defect detection model 801 can be optimized based on the difference between the defect detection result 804 and the defect annotation information 805.

[0143] The defect detection information 802 can include the defect positions 8021 and defect types 8022 of at least one target defect respectively. The defect annotation information 805 can include the annotation positions and annotation types of at least one annotated defect respectively. After obtaining the defect annotation information 805, the difference 806 between the defect detection result 804 and the defect annotation information 805 can be determined, and this difference 806 can be used to characterize the ability of the defect detection model 801 to detect defects.

[0144] After obtaining the defect annotation information 805, the confusion matrix of the defect detection result 804 and the defect annotation information 805 can be determined, and the difference 806 can be determined according to the confusion matrix. The specific determination method of the difference 806 can be configured according to actual business requirements and is not limited herein. For example, the determination method of the difference 806 can include at least one of the following: accuracy, recall, and precision, etc.

[0145] After obtaining the difference 806, it can be determined whether the difference 806 meets the predetermined performance conditions. The predetermined performance conditions can be configured according to actual business requirements and are not limited herein. For example, the predetermined performance conditions can include at least one of the following: the accuracy is not less than 90%, the recall is not less than 85%, and the precision is not less than 85%. When the difference 806 meets the predetermined performance conditions, there is no need to optimize the defect detection model 801, that is, this defect detection model can continue to be used in the defect detection task for the target process flow.

[0146] In one example, when the difference 806 does not meet the predetermined performance conditions, the model parameters of the defect detection model 801 can be adjusted according to the difference 806 to obtain an adjusted defect detection model. For example, the hyperparameters of the model, such as the learning rate, regularization coefficient, etc., can be adjusted according to the performance indicators of the confusion matrix to improve the generalization ability of the model. Alternatively, the network structure can be optimized according to the performance of the model on specific defect types, such as adding convolutional layers, adjusting the size of convolutional kernels, etc.

[0147] In another example, when the difference 806 does not meet the predetermined performance conditions, the samples can also be sorted according to the misclassification cases shown in the confusion matrix to increase the balance of positive and negative samples and supplement the missing defect type samples.

[0148] In yet another example, when the difference 806 does not meet the predetermined performance conditions, image preprocessing steps, such as denoising, normalization, etc., can be added to improve the quality of the input image; and the post-processing steps can also be optimized, such as adjusting the confidence threshold, to reduce false detections.

[0149] According to the embodiments of the present disclosure, since the defect annotation information provides the position and type of the annotated defect, the difference is obtained by comparing the defect annotation information with the defect detection result, which can intuitively reflect the detection ability of the model in actual applications. On this basis, by adjusting the parameters of the defect detection model according to the difference, the dynamic optimization and improvement of the performance of the defect detection model are realized. This feedback-based optimization mechanism can not only quickly adapt to the changes in different process flows and products, but also continuously learn and optimize during continuous operation, thereby improving the accuracy and reliability of defect detection.

[0150] In the embodiments of the present disclosure, during the defect detection process, the constraint rules can also be adjusted. The following will take Figure 9 as an example to further illustrate the adjustment process of the constraint rules.

[0151] Figure 9 Schematically shows an example schematic diagram of the defect detection process according to another embodiment of the present disclosure.

[0152] As Figure 9As shown, in 900, the new products produced by the target process flow can be monitored regularly to handle exceptions in a timely manner. In one example, at least one defect detection result in a preset time period 901 can be monitored to obtain a monitoring result 902. The preset time period 901 refers to a preset time range that can be used to monitor the defect detection results. For example, the preset time period can be 8 hours. The monitoring result 902 refers to the result obtained by analyzing the defect detection results within the preset time period 901, and can characterize the actual distribution of the defects of the products produced by the target process flow.

[0153] After obtaining the monitoring result 902, an operation S910 can be performed. In operation S910, does the monitoring result 902 meet the preset defect distribution condition? The preset defect distribution condition refers to a preset defect distribution standard used to determine whether the monitoring result 902 meets the expected quality requirements. The preset defect distribution condition can be configured according to actual business needs and is not limited herein. For example, the preset defect distribution condition can be that the number of scratches does not exceed 5, the number of holes does not exceed 2, and the number of cracks does not exceed 1.

[0154] If so, there is no need to adjust the constraint rule 903, and the current constraint rule 903 and the defect detection model can be applied to the defect detection task for the target process flow. If not, the constraint rule 903 can be adjusted according to the monitoring result 902 to obtain an adjusted constraint rule 904. The adjusted constraint rule 904 can better adapt to the actual production situation. For example, when the monitoring result 902 indicates that the number of holes often exceeds the preset defect distribution during actual production, the allowable value of the number of holes can be increased from 2 to 3.

[0155] According to the embodiments of the present disclosure, by monitoring the defect detection results within a preset time period, since the monitoring results can intuitively reflect the actual distribution of the defects of the products produced by the target process flow, real-time and accurate feedback is provided for defect detection. On this basis, by dynamically adjusting the constraint rules according to the monitoring results, the intelligent optimization and dynamic management of the defect detection process are realized. It can not only effectively cope with the process fluctuations or defect distribution changes that may occur during the production process, but also improve the accuracy and adaptability of defect detection. By continuously optimizing the constraint rules, it can better meet the requirements of different production stages and product types, and further improve the efficiency and accuracy of defect detection.

[0156] The above are only exemplary embodiments, but are not limited thereto. Other known defect detection methods in the art can also be included as long as they can improve the accuracy and reliability of defect detection.

[0157] Based on the above sample generation method 200, the present invention also provides a sample generation device. The following will be combined withFigure 10 Describe the device in detail.

[0158] Figure 10 A block diagram of a sample generation device according to an embodiment of the present disclosure is schematically shown.

[0159] As Figure 10 shown, the sample generation device 1000 may include an acquisition module 1010, an adjustment module 1020, and a generation module 1030.

[0160] The acquisition module 1010 is configured to acquire a sample image and a label, where the label is used to identify the defect position and defect type of at least one sample defect in the sample image.

[0161] The adjustment module 1020 is configured to adjust the sizes of a plurality of first image blocks obtained by dividing the sample image according to the defect positions of at least one sample defect, to obtain a plurality of second image blocks.

[0162] The generation module 1030 is configured to determine at least one target image block having a defect type among the plurality of second image blocks according to the image quality evaluation values of the plurality of second image blocks, and use the plurality of second image blocks and at least one target image block corresponding to each defect type as samples.

[0163] Based on the above defect detection method 600, the present invention also provides a defect detection device. The following will be combined with Figure 11 Describe the device in detail.

[0164] Figure 11 A block diagram of a defect detection device according to an embodiment of the present disclosure is schematically shown.

[0165] As Figure 11 shown, the defect detection device 1100 may include an input module 1110 and a screening module 1120.

[0166] The input module 1110 is configured to input an image to be detected into a defect detection model to obtain defect detection information, where the defect detection model is trained with samples generated by the sample generation device 1100, the image to be detected is an image of a product produced through a target process flow, and the defect detection information includes the defect position and defect type of at least one target defect.

[0167] The screening module 1120 is configured to screen at least one target defect according to a constraint rule to obtain a defect detection result, where the constraint rule is used to constrain the defect distribution of products produced through a target process flow.

[0168] Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present disclosure, or at least some functions of any one or more of them, may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present disclosure may be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present disclosure may be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or may be implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present disclosure may be at least partially implemented as a computer program module, and when the computer program module is run, corresponding functions may be executed.

[0169] It should be noted that the sample generation device part in the embodiments of the present disclosure corresponds to the sample generation method part in the embodiments of the present disclosure. For the description of the sample generation device part, please refer to the sample generation method part specifically, and details will not be repeated here. The defect detection device part in the embodiments of the present disclosure corresponds to the defect detection method part in the embodiments of the present disclosure. For the description of the defect detection device part, please refer to the defect detection method part specifically, and details will not be repeated here.

[0170] Figure 12 A block diagram of an electronic device suitable for implementing the sample generation method and the defect detection method according to embodiments of the present disclosure is schematically shown. Figure 12 The shown electronic device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0171] As Figure 12 As shown, the computer electronic device 1200 according to embodiments of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1209 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application-specific integrated circuit (ASIC)), and so on. The processor 1201 may also include on-board memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of the present disclosure.

[0172] In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204.

[0173] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, and the input / output (I / O) interface 1205 is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the input / output (I / O) interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output (I / O) interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1210 as needed so that a computer program read therefrom is installed into the storage portion 1208 as needed.

[0174] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, a sample generation method and a defect detection method according to embodiments of the present disclosure are implemented.

[0175] In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device.

[0176] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program codes for executing the methods provided in the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program codes are used to cause the electronic device to implement the sample generation method and the defect detection method provided in the embodiments of the present disclosure.

[0177] When the computer program is executed by the processor 1201, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0178] According to embodiments of the present disclosure, program code for executing the computer programs provided by the embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0179] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in a different order than that marked in the accompanying drawings.

[0180] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A sample generation method, comprising: Acquire a sample image and a label, wherein the label is used to identify a defect position and a defect type of at least one sample defect in the sample image; According to the defect position of each of the at least one sample defect, adjusting the sizes of the plurality of first image blocks obtained based on the division of the sample image to obtain a plurality of second image blocks; and According to the image quality evaluation values ​​of the plurality of second image blocks, at least one target image block having the defect type is determined among the plurality of second image blocks, and the plurality of second image blocks and at least one target image block corresponding to each of the defect types are used as samples.

2. The method according to claim 1, wherein: The step of adjusting the sizes of the plurality of first image blocks obtained by dividing the sample image according to the defect position of each of the at least one sample defects to obtain the plurality of second image blocks comprises: For each of the sample defects, According to the defect position, determining at least two image blocks to be adjusted among the plurality of first image blocks, wherein the image blocks to be adjusted are first image blocks including the sample defect; and According to the defect position, the size of each of the image blocks to be adjusted is adjusted respectively, so that the sample defect is located in an adjusted image block, and the adjusted image block including the sample defect is used as the second image block.

3. The method according to claim 2, wherein: The plurality of first image blocks are obtained by dividing the sample image based on a preset size, and the defect position includes at least two edge point coordinates; The adjusting the size of each of the image blocks to be adjusted according to the defect position comprises: Determining a size to be adjusted for each of the image blocks to be adjusted according to the preset size and the defect size determined based on the coordinates of the at least two edge points, wherein the size to be adjusted is used to make the sample defect located in one of the adjusted image blocks; and According to the size to be adjusted, each of the image blocks to be adjusted is adjusted respectively to obtain the adjusted image blocks.

4. The method according to any one of claims 1 to 3, further comprising, before determining at least one target image block for the defect type among the plurality of second image blocks according to the image quality evaluation values ​​of the plurality of second image blocks: At least one first defect type and at least one second defect type are determined according to the number of second image blocks having different defect types, wherein: The number of second image blocks having the first defect type in the plurality of second image blocks is M, and the number of second image blocks having the second defect type is N, where both M and N are positive integers and M>N; as well as For each of the second defect types, According to M and N, a first number of target image blocks having the second defect type is determined.

5. The method according to claim 4, wherein: The determining, according to the image quality evaluation values ​​of the plurality of second image blocks respectively, at least one target image block for the defect type from the plurality of second image blocks comprises: Determine an image quality evaluation difference between at least two of the second image blocks having the second defect type, and determine an average of a plurality of image quality evaluation differences corresponding to each of the second image blocks as an image quality evaluation value of the second image block; A second image block corresponding to a first number of image quality assessment values ​​among the sorted image quality assessment values ​​is determined as the target image block.

6. The method according to claim 5, wherein: The determining of the image quality evaluation difference between at least two second image blocks having the second defect type comprises: Based on a sliding window method, at least two of the second image blocks are divided respectively to obtain a plurality of sub-image blocks of each of the second image blocks; and Based on the image quality evaluation index, the image quality evaluation difference is determined according to the sub-image blocks located at the same position in at least two of the second image blocks.

7. The method according to claim 6, wherein: The image quality evaluation index includes at least one of the following: a brightness index, a contrast index, a structure index, a color index, a noise index and a clarity index.

8. A defect detection method, comprising: Inputting the image to be detected into a defect detection model to obtain defect detection information, wherein the defect detection model is trained using the sample generated by the method described in claims 1 to 7, the image to be detected is an image of a product produced by a target process flow, and the defect detection information includes a defect position and a defect type of at least one target defect; and According to the constraint rules, the at least one target defect is screened to obtain a defect detection result, wherein the constraint rules are used to constrain the defect distribution of the product produced by the target process flow.

9. The method according to claim 8, wherein: The constraint rules are obtained according to the constraint item configuration for the target process flow; The constraint items include at least one of the following: product model, defect type, defect area, defect number, defect type priority and confidence, and the defect type priority represents the degree of influence of the defect type on the product.

10. The method according to claim 8 or 9, further comprising, after screening the at least one target defect according to the constraint rule to obtain the defect detection result: Monitor at least one defect detection result in a predetermined period to obtain a monitoring result, wherein: The monitoring results represent the actual distribution of defects of the product produced by the target process flow; as well as In response to the monitoring result not satisfying a predetermined defect distribution condition, the constraint rule is adjusted according to the monitoring result to obtain an adjusted constraint rule.

11. The method according to any one of claims 8 to 10, further comprising: Determining a difference between the defect detection result and defect annotation information of the image to be detected, wherein the defect annotation information includes an annotation position and an annotation type of at least one respective annotated defect, and the difference represents an ability of the defect detection model to detect defects; and In the case that the difference does not satisfy a predetermined performance condition, the model parameters of the defect detection model are adjusted according to the difference to obtain an adjusted defect detection model.

12. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images