Defect Detection and Sample Generation Methods, Apparatuses, Devices, and Storage Media

Abstract

The present application provides a defect detection and sample generation method, apparatus, device, and storage medium. The method includes: determining a first product defect area marked in a first sample image and its corresponding defect category; obtaining a method for generating elastic transformation parameters corresponding to the defect category, where the method for generating elastic transformation parameters is pre-configured based on the defect morphological features corresponding to the defect category; generating target elastic transformation parameters corresponding to the first product defect area according to the method for generating elastic transformation parameters, and performing elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area. By configuring different methods for generating elastic transformation parameters based on the presented defect morphological features for different defect categories, the elastic transformation results of product defects corresponding to each defect category can be made more consistent with the defect morphological features corresponding to that defect category, and high-quality defect sample images can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, apparatus, device, and storage medium for defect detection and sample generation. Background Art

[0002] Industrial defect detection is a challenging task, and its application scenarios cover various application fields such as heavy industry and light industry. For example, industries such as steel, electronics, and textiles all have the need to detect surface defects of products. In the surface defect detection task, due to the various shapes and positions of surface defects, it is difficult to detect. Conventional surface defect detection schemes generally use visible light imaging to collect images of the items to be detected and use deep neural networks for defect detection.

[0003] In the industrial surface defect detection task, the number of samples containing defect information that can be collected is generally small. Directly using a deep neural network to train a model based on the originally collected sample data is likely to cause overfitting of the neural network model, resulting in poor generalization performance of the model. Summary of the Invention

[0004] Embodiments of the present invention provide a method, apparatus, device, and storage medium for defect detection and sample generation to generate more realistic defect samples, thereby expanding the number of samples used for model training.

[0005] In a first aspect, an embodiment of the present invention provides a sample generation method, the method including:

[0006] Determine a first product defect area marked in a first sample image and a defect category corresponding to the first product defect area;

[0007] Obtain an elastic transformation parameter generation method corresponding to the defect category, where the elastic transformation parameter generation method is preconfigured based on defect morphological features corresponding to the defect category;

[0008] Generate a target elastic transformation parameter corresponding to the first product defect area according to the elastic transformation parameter generation method;

[0009] Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameter to obtain a second sample image including a second product defect area.

[0010] In a second aspect, an embodiment of the present invention provides a sample generation apparatus, the apparatus including:

[0011] A determination module, configured to determine a first product defect area marked in a first sample image and a defect category corresponding to the first product defect area;

[0012] An acquisition module, configured to acquire an elastic transformation parameter generation method corresponding to the defect category, where the elastic transformation parameter generation method is pre-configured based on the defect morphological features corresponding to the defect category;

[0013] A processing module, configured to generate target elastic transformation parameters corresponding to the first product defect area according to the elastic transformation parameter generation method, and perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor can at least implement the sample generation method as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which an executable code is stored, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the sample generation method as described in the first aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a sample generation method, the method including:

[0017] Receiving a defect category input by a user and defect morphological feature analysis items corresponding to the defect category;

[0018] Performing defect morphological feature analysis on a plurality of sample images corresponding to the defect category according to the defect morphological feature analysis items to display a defect morphological feature analysis result;

[0019] Receiving an elastic transformation parameter generation method corresponding to the defect category determined by the user according to the defect morphological feature analysis result;

[0020] Receiving a first sample image marked with a first product defect area, where the first product defect area corresponds to the defect category;

[0021] Generating target elastic transformation parameters corresponding to the first product defect area according to the elastic transformation parameter generation method;

[0022] Performing elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area.

[0023] In a sixth aspect, an embodiment of the present invention provides a defect detection method, including:

[0024] Obtain a first sample image from a training sample set corresponding to an industrial product with surface defects. The first sample image is marked with a first product defect area and a defect category corresponding to the first product defect area;

[0025] Obtain a method for generating elastic transformation parameters corresponding to the defect category, where the method for generating elastic transformation parameters is pre-configured based on the defect morphological features corresponding to the defect category;

[0026] Generate target elastic transformation parameters corresponding to the first product defect area according to the method for generating elastic transformation parameters;

[0027] Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image containing a second product defect area, and add the second sample image to the training sample set;

[0028] Based on the defect detection model trained using the training sample set, perform defect detection on the industrial product to be detected.

[0029] An embodiment of the present invention provides a solution for realizing sample image enhancement based on elastic transformation, thereby expanding the sample set. In the application scenario of product defect detection, statistical analysis of the corresponding defect morphological features can be pre-conducted for different defect categories, and then a method for generating elastic transformation parameters corresponding to the corresponding defect category can be pre-configured based on the defect morphological features corresponding to each defect category. In this way, the methods for generating elastic transformation parameters corresponding to each defect category are not completely the same. Based on this, during the model training process, for any first sample image that has been collected, the product defect area and the corresponding defect category included therein will be marked in the first sample image. For the first product defect area included therein, a method for generating elastic transformation parameters corresponding to the defect category can be obtained based on the defect category corresponding to the first product defect area, and then the target elastic transformation parameters corresponding to the first product defect area can be generated according to the obtained method for generating elastic transformation parameters, so as to perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image containing a second product defect area (i.e., the transformed result of the first product defect area), and subsequently, the second sample image can be used for model training.

[0030] In the solution provided by the embodiment of the present invention, by configuring different methods for generating elastic transformation parameters based on the defect morphological features presented by different defect categories, the elastic transformation results of the product defect areas corresponding to any defect category can be made more in line with the defect morphological features corresponding to the defect category, and more realistic and diverse defect sample images can be obtained. Description of the Drawings

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a flowchart of a sample generation method provided by an embodiment of the present invention;

[0033] Figure 2 It is an operation schematic diagram of a configuration interface provided by an embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of an offset matrix provided by an embodiment of the present invention;

[0035] Figure 4 It is an application schematic diagram of a sample generation method provided by an embodiment of the present invention;

[0036] Figure 5 It is a flowchart of a sample generation method provided by an embodiment of the present invention;

[0037] Figure 6 It is a schematic diagram of the structure of a sample generation device provided by an embodiment of the present invention;

[0038] Figure 7 For Figure 6 It is a schematic diagram of the structure of an electronic device corresponding to the sample generation device shown in the embodiment. Detailed Embodiments

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0040] The following will describe in detail some embodiments of the present invention with reference to the drawings. Without conflict between the embodiments, the embodiments and the features in the embodiments can be combined with each other. In addition, the step timings in the following method embodiments are only examples, not strictly limited.

[0041] The sample generation method provided by the embodiments of the present invention can be executed by an electronic device, which can be a server or a user terminal. The server can be a physical server or a virtual server (virtual machine) in the cloud.

[0042] The solution provided by the embodiments of the present invention can be applied to the scenario of product surface defect detection, such as detecting defects like scratches, pits, and bulges on the surface of lithium batteries. In traditional detection solutions, visible light imaging is performed on a large number of products to obtain sample images, including positive example sample images without defects and negative example sample images with defects. Then, based on the captured sample images, a neural network model is trained to obtain a model for defect detection (referred to as a defect detection model).

[0043] However, there are some problems in this traditional model training solution as follows:

[0044] First, the surface defect morphologies are diverse. Especially for non-rigid surface defects, there is a large degree of freedom in morphology, and the difference degree of the same type of defects is very large. If the model is directly trained based on the original sample images, the model can only learn a very small part of the defect morphologies, which is not conducive to the generalization performance of the model. Second, in the surface defect detection task, the number of sample images with severe defects is generally small. Directly training the model based on the original sample data is also likely to cause overfitting of the model, resulting in insufficient final generalization performance.

[0045] Based on this, the embodiments of the present invention propose a defect detection data enhancement solution based on adjustable elastic transformation. According to the unique morphological characteristics of different types of defects, the transformation space of the elastic transformation of the corresponding annotation rectangle region (i.e., the marked product defect region) can be adjusted by itself to cover as many defect morphologies as possible, providing more diverse training samples for the defect detection module, and finally improving the accuracy and generalization ability of the defect detection model.

[0046] The solution proposed by the embodiments of the present invention is mainly used in the training stage of the model to enhance the data of the input training sample images. The implementation process of this solution will be described below in combination with some embodiments.

[0047] Figure 1 is a flowchart of a sample generation method provided by the embodiments of the present invention. As Figure 1 shown, the method includes the following steps:

[0048] 101. Determine the first product defect region marked in the first sample image and the defect category corresponding to the first product defect region.

[0049] 102. Obtain the elastic transformation parameter generation method corresponding to the defect category, and the elastic transformation parameter generation method is pre-configured based on the defect morphological features corresponding to the defect category.

[0050] 103. Generate the target elastic transformation parameters corresponding to the first product defect area according to the elastic transformation parameter generation method.

[0051] 104. Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image containing the second product defect area.

[0052] To complete the product surface defect detection task, it is necessary to train a defect detection model. In order to obtain a defect detection model with better performance, it is necessary to generate training sample images with good quantity and quality. In practical applications, for a certain product, the defect detection model can be trained by collecting a large number of positive sample images and negative sample images. Among them, a positive sample image refers to an image obtained by photographing a product without defects, that is, there is no product defect area in the positive sample image; on the contrary, a negative sample image can be obtained by photographing a product with defects, that is, the negative sample image contains a product defect area.

[0053] However, in fact, the number of directly collectable negative sample images is relatively limited. Therefore, in the embodiments of the present invention, by adopting the method of elastic transformation, on the basis of the already collected negative sample images, the number of negative sample images is expanded, and the quality of the newly generated negative sample images is ensured.

[0054] In fact, the defects existing on the surface of a certain product can be divided into different categories, such as scratch defects, pit defects, bulge defects, and so on. And the morphological features presented by different defect categories are often different.

[0055] For example, scratch defects may present scratch curves in various directions such as horizontal, vertical, and inclined directions, and the fluctuation degree of the scratch curves is also diverse. For example, some are similar to straight lines, some have a small curvature, and some have a large curvature.

[0056] For another example, pit defects may present shape features such as circular and elliptical, and the radius is often within a certain value range.

[0057] For another example, bulge defects may present a long strip shape and are distributed horizontally or vertically along a certain edge of the product.

[0058] In order to obtain a defect detection model with better performance, it is necessary for the defect detection model to have good recognition ability for various possible defect categories. Therefore, during the model training stage, a large number of sample images corresponding to each defect category need to be collected. Based on this, when using the elastic transformation method to generate new sample images, for each defect category, a large number of corresponding training sample images can be generated specifically. And this "specificity" is mainly achieved by combining the defect morphological features presented by a certain defect category to generate corresponding elastic transformation parameters, and then performing elastic transformation on the existing sample images corresponding to this defect category based on the generated elastic transformation parameters.

[0059] In elastic transformation, the relevant parameters involved include: horizontal offset matrix, vertical offset matrix, Gaussian kernel corresponding to the horizontal offset matrix and horizontal amplitude control factor, Gaussian kernel corresponding to the vertical offset matrix and vertical amplitude control factor.

[0060] Since the defect morphological features of different defect categories are different, the generation methods or value determination results of the above parameters corresponding to different defect categories will be different.

[0061] In practical applications, users can first annotate the product defect areas and corresponding defect categories contained in the collected negative example sample images. Based on the annotation results of numerous negative example sample images, a corresponding sample set can be selected for each defect category. For example, if a certain defect category X is annotated in a sample image, then this sample image will be used as one of the sample sets corresponding to defect category X. After that, for the sample set corresponding to each defect category, the defect morphological features are analyzed to obtain the defect morphological features corresponding to each defect category. Furthermore, users can configure the corresponding elastic transformation parameter generation method for the analyzed defect morphological features of each defect category.

[0062] In practical applications, users can configure corresponding defect morphological feature analysis items for each defect category. The analysis items configured for different defect categories may be the same or different. After that, based on the analysis items configured by the user, the defect morphological features of each sample image in the sample set corresponding to the current defect category are analyzed to obtain the analysis result, that is, the specific defect morphological features, and the analysis result is output to the user so that the user can configure the corresponding elastic transformation parameter generation method based on this analysis result.

[0063] Combined with Figure 2 Exemplarily illustrate the configuration process of the elastic transformation parameter generation method. As Figure 2 shown in, in a certain configuration interface of the computing device, there may be a defect category selection box, and users can select any one of the multiple configured defect categories from it. In Figure 2In the example, assume that the user selects the scratch defect category.

[0064] After that, based on the defect category selection result of the user, the analysis item selection interface can be displayed. The interface contains multiple analysis items set, and the user selects the required analysis items. For example, it includes direction, length, width, fluctuation degree, shape, etc. For scratch defects, in Figure 2 It is assumed that the analysis items selected by the user include orientation, length, width, and degree of fluctuation. Based on the user's category selection result, the computing device can filter out sample images containing scratch defects based on the annotation information in several sample images that have been stored, wherein the annotation information includes the defect category and the defect location (i.e., the product defect area). Then, based on the user's above-mentioned analysis item configuration results, the computing device can perform corresponding morphological feature analysis on each product defect area corresponding to the scratch category in each filtered sample image, summarize the analysis results, and display the analysis results on the interface.

[0065] Among them, when the user configures the analysis items of orientation, length, width, and fluctuation degree, these analysis items are measured for the defects in the marked product defect area (referring to the rectangular marked box area corresponding to the scratch category) in each sample image screened out, and the measurement results can be expressed as: horizontal scratches, length value, width value, sharp fluctuation in value, and gentle fluctuation in horizontal direction. Among them, such as orientation and fluctuation degree, several feature description results and their corresponding value ranges can be pre-set. For example, when the angle between the orientation and the horizontal coordinate axis is less than the preset value, it is considered that the orientation is horizontal; for another example, if the change in amplitude in the vertical direction exceeds the set threshold, it is considered that the vertical direction fluctuates violently.

[0066] After analyzing the defect morphological characteristics of each sample image of the same category, the defect morphological characteristics presented by these sample images can be summarized. In simple terms, the summary is to group and summarize the measurement results corresponding to each of these sample images, and group the measurement results with insignificant differences in all or part of the analysis items into one group. In this way, users can know the different defect morphological characteristics presented under the same defect category (such as scratch defects). Then, the corresponding elastic transformation parameter generation method is configured based on the statistical defect morphological characteristics.

[0067] like Figure 2 As shown in , it is assumed that some of the collected sample images containing scratches present scratches in different directions such as horizontal, vertical, and oblique, and the curvature of the scratches varies. Based on the summary results of the scratch morphological features in each sample image, the user can know the morphological features presented by the scratch-type defects, and thus, the elastic transformation parameter generation method corresponding to the scratch-type defects can be configured in a targeted manner according to the statistical results.

[0068] When configuring the elastic transformation parameter generation method, the effects of different parameters need to be considered. Generally speaking, by controlling the values of the horizontal (lateral) offset matrix and the vertical (longitudinal) offset matrix, the direction of elastic transformation can be controlled. For example, in an extreme case: setting the vertical offset matrix to all zeros will result in the subsequent elastic transformation not occurring in the vertical direction. By controlling the standard deviation of the Gaussian kernel corresponding to the horizontal or vertical offset matrix, the degree of change of adjacent elements in the corresponding offset matrix can be controlled. By controlling the amplitude control factor corresponding to the horizontal or vertical offset matrix, the overall offset degree of the elastic transformation can be controlled.

[0069] Therefore, for the defect morphology features corresponding to different defect categories respectively, the generation methods of the horizontal offset matrix and the vertical offset matrix can be configured, the value ranges of the Gaussian kernel and the horizontal amplitude control factor corresponding to the horizontal offset matrix can be configured, and the value ranges of the Gaussian kernel and the vertical amplitude control factor corresponding to the vertical offset matrix can be configured.

[0070] Generally speaking, for a certain defect category, when its corresponding defect morphology features include the orientation of the defect, and / or, the amplitude fluctuation degree of the defect in the horizontal and vertical directions, the generation methods of the horizontal offset matrix and the vertical offset matrix can be determined according to the orientation of the defect, and / or, the amplitude fluctuation degree of the defect in the horizontal and vertical directions. When the defect morphology feature includes the size of the defect, the value ranges of the Gaussian kernel corresponding to the horizontal offset matrix and the Gaussian kernel corresponding to the vertical offset matrix can be determined according to the size of the defect. And when the defect morphology feature includes the amplitude fluctuation degree of the defect in the horizontal and vertical directions, the value ranges of the horizontal amplitude control factor and the vertical amplitude control factor can also be determined according to the amplitude fluctuation degree of the defect in the horizontal and vertical directions.

[0071] Optionally, the user can input the elastic transformation parameter generation method corresponding to each defect category respectively based on the statistical results of the defect morphology features corresponding to each defect category. Or the elastic transformation parameter generation methods corresponding to different defect morphology features can be pre-configured, and after obtaining the defect morphology features corresponding to each defect category, the elastic transformation parameter generation method corresponding to each defect category can be determined based on this pre-configured information.

[0072] For example, for the morphology features manifested as circular or oval, it is determined that the values of the horizontal offset matrix and the vertical offset matrix can be determined in a random manner. The so-called random manner means that the values in the matrix are random numbers uniformly distributed in the interval [-1, +1]; and it is determined that the value of the standard deviation of the Gaussian kernel corresponding to each offset matrix is greater than a set threshold, so that the smoothing range is larger.

[0073] For another example, for the morphological features characterized by horizontal scratches with a relatively large vertical fluctuation amplitude of the scratches, it can be determined that the values of the horizontal offset matrix are set to all 0, and the values of the vertical offset matrix are determined randomly; it can also be determined that the horizontal amplitude control factor corresponding to the horizontal offset matrix is less than the set threshold, while the vertical amplitude control factor corresponding to the vertical offset matrix is greater than the set threshold; in addition, when the width of the scratch is small (i.e., the scratch is thin), it can also be determined that the standard deviation of the Gaussian kernel corresponding to the horizontal offset matrix is less than the set threshold.

[0074] Optionally, when the defect morphological features corresponding to a certain defect category are diverse, the elastic transformation parameter generation method can be configured for each group of defect morphological features, or a unified elastic transformation parameter generation method corresponding to the defect category can be configured by comprehensive consideration.

[0075] For example, in the scratch defect exemplified above, it may be found from the statistical results that there are three groups of scratch morphological features with horizontal, vertical, and inclined orientations. Optionally, for the scratch morphological features with a horizontal orientation, it can be determined that the values of the horizontal offset matrix are set to all 0 or randomly determined within a smaller range such as [-0.5, 0.5], and the values of the vertical offset matrix are determined randomly. It can also be determined that the horizontal amplitude control factor corresponding to the horizontal offset matrix is less than the set threshold, while the vertical amplitude control factor corresponding to the vertical offset matrix is greater than the set threshold. Optionally, for the scratch morphological features with a vertical orientation, it can be determined that the values of the vertical offset matrix are set to all 0, and the values of the horizontal offset matrix are determined randomly. It can also be determined that the horizontal amplitude control factor corresponding to the vertical offset matrix is less than the set threshold, while the vertical amplitude control factor corresponding to the horizontal offset matrix is greater than the set threshold. Optionally, for the scratch morphological features with an inclined orientation, it can be determined that the values of the horizontal and vertical offset matrices are determined randomly, and it can also be determined that the horizontal amplitude control factor corresponding to the horizontal offset matrix and the vertical amplitude control factor corresponding to the vertical offset matrix are the set thresholds.

[0076] In addition to the configuration of the differential elastic transformation parameter generation method for different scratch morphological features exemplified above, a unified elastic transformation generation method can also be configured for scratch-type defects. For example, it is determined to use a random method to determine the values of the horizontal and vertical offset matrices, and the horizontal amplitude control factor corresponding to the horizontal offset matrix and the vertical amplitude control factor corresponding to the vertical offset matrix are determined as set thresholds (such as 2W). The Gaussian kernel corresponding to the horizontal offset matrix and the Gaussian kernel corresponding to the vertical offset matrix are determined as set thresholds (such as a mean of 0 and a standard deviation of 0.2W). Another example is for bulge-type defects. Suppose the morphological features presented are: long strip shape, little distortion in the horizontal direction, but at the edges, there are some large distortions in the vertical direction. At this time, the initial value of the horizontal offset matrix can be set to be small (such as all 0 or close to 0), and the value of the vertical offset matrix is determined by a random method; or, the amplitude control factor corresponding to the horizontal offset matrix can be set to be small, and the amplitude control factor corresponding to the vertical offset matrix can be set to be large. Here, W is the width of the horizontal and vertical offset matrices.

[0077] Based on the elastic transformation parameter generation methods set for different defect categories according to the defect morphological features presented by the user in combination with different defect categories, elastic transformation of the sample images can be performed during the model training phase.

[0078] Specifically, for a currently input sample image, referred to as the first sample image, based on the product defect area marked in the first sample image and the corresponding defect category, it can be known which defect category of defect is included in the first sample image and the location area of the defect. The product defect area is often marked by a rectangular box. There may be more than one marked rectangular box in the first sample image. Here, taking a selected product defect area (referred to as the first product defect area) as an example, based on the defect category annotation information corresponding to the first product defect area, the elastic transformation parameter generation method corresponding to the defect category can be determined from the already stored configuration information, where the configuration information is the elastic transformation parameter generation method configured for different defect categories as described above.

[0079] After that, according to the determined elastic transformation parameter generation method, generate the target elastic transformation parameters corresponding to the first product defect area. Then, perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image containing the transformed product defect area (referred to as the second product defect area). Among them, since the elastic transformation parameter generation method corresponding to this defect category is configured based on the defect morphological features corresponding to the defect category, the second product defect area obtained by transforming the first product defect area with the target elastic transformation parameters presents morphological features similar to those of the first product defect area, thus realizing the expansion of the sample images corresponding to this defect category and ensuring the quality of the generated new sample images: having the defect morphological features corresponding to this defect category.

[0080] The specific execution process of elastic transformation can be implemented by referring to existing related technologies and will not be elaborated in this article. Only a simple description of the implementation process is provided:

[0081] Suppose the first product defect area is represented as a picture X with a size of W*H, where W is the width and H is the height. First, a displacement field needs to be generated to deform the picture X, and this displacement field is the above-mentioned horizontal offset matrix and vertical offset matrix. Among them, the width and height dimensions of the offset matrix are also W and H. It should be noted that offset matrices need to be generated for the x direction (horizontal direction) and y direction (vertical direction) respectively, so a horizontal offset matrix and a vertical offset matrix need to be generated. The role of the offset matrix is to generate the basic (initial) position offset coefficients for the elastic transformation operation. The reason for saying it is the basic offset coefficient is that further smoothing and scaling processing need to be performed on this offset matrix later. The values in the horizontal offset matrix and vertical offset matrix correspond one by one to the pixel points in the picture X. For example, the value in the first row and first column of the offset matrix corresponds to the pixel point in the first row and first column of the image X, and is used to represent how much displacement the pixel point in the first row and first column of the image X needs to be moved horizontally and vertically. The coordinate system of the image X is established with the upper left corner as the origin, the horizontal direction as the x-axis, and the numerical direction as the y-axis, for example.

[0082] By default, each value in the offset matrix is a random number uniformly distributed in the interval [-1, +1], that is, the values are determined randomly, as shown in Figure 3 However, as mentioned above, for the picture X, the values of the horizontal and vertical offset matrices corresponding to the picture X need to be determined according to the determined offset matrix generation method corresponding to its defect category.

[0083] After that, Gaussian filtering needs to be performed on the initially generated horizontal offset matrix and vertical offset matrix. The role of Gaussian filtering is to smooth and scale the amplitude of the above-generated offset matrix. The purpose of smoothing is to avoid the situation where the final elastic transformation result does not match the shape of the defect in Picture X due to the obvious difference in the displacement degree of adjacent pixel points in Picture X. The purpose of amplitude scaling is to make the final overall pixel offset within a suitable range.

[0084] In fact, the above smoothing effect is achieved by performing convolution calculations using a Gaussian kernel, that is, using the set Gaussian kernel corresponding to the horizontal offset matrix to perform convolution processing on the horizontal offset matrix, and using the Gaussian kernel corresponding to the vertical offset matrix to perform convolution processing on the vertical offset matrix. The Gaussian kernel can be expressed as a Gaussian function with a mean of 0 and a standard deviation of sigma. The default value of sigma can be set, such as 0.2W. For Picture X, the corresponding method for generating elastic transformation parameters for its defect category will be configured with a corresponding method for determining the value of sigma, such as using the default value (i.e., the set threshold), or a certain value range greater than or less than the default value.

[0085] The above scaling effect is achieved by using the amplitude control factor alpha, that is, multiplying the convolved horizontal offset matrix by the set horizontal amplitude control factor corresponding to the horizontal offset matrix to achieve scaling processing of each value therein, and multiplying the convolved vertical offset matrix by the set vertical amplitude control factor corresponding to the vertical offset matrix to achieve scaling processing of each value therein. The default value of the amplitude control factor can be set, such as 2W. For Picture X, the corresponding method for generating elastic transformation parameters for its defect category will be configured with corresponding methods for determining the values of the horizontal and vertical amplitude control factors, such as using the default value (i.e., the set threshold), or a certain value range greater than or less than the default value.

[0086] Finally, the horizontal offset matrix and vertical offset matrix after the above smoothing and scaling processing are applied to the input Picture X to obtain Picture Y (i.e., the second product defect area) after elastic transformation processing.

[0087] For ease of understanding the execution process of the above solution, combined with Figure 4 exemplary illustration. In Figure 4In this case, assume that the product to be defect-detected is a battery. For the first battery image captured of the battery surface, a rectangular box marked as a scratch-type defect is included in the first battery image, and there is a scratch distributed horizontally within the rectangular box. Based on the annotation information of the rectangular box, when determining that the corresponding defect category is a scratch-type defect, determine the corresponding elastic transformation parameter generation method from the already stored configuration information. Assume that the horizontal offset matrix is set to all 0 values, and the values of the vertical offset matrix are randomly determined within the range of [-1, 1]; the Gaussian kernel corresponding to the horizontal offset matrix is (0, 0.1W), the Gaussian kernel corresponding to the vertical offset matrix is (0, 0.2W), the amplitude control factor corresponding to the horizontal offset matrix is 1W, and the amplitude control factor corresponding to the vertical offset matrix is 2W, where W is the width of the above rectangular box. Based on the determined elastic transformation parameter generation method, generate the corresponding elastic transformation parameters, and use the generated elastic transformation parameters to perform elastic transformation processing on the annotated rectangular box area to obtain the second battery image. As Figure 4 shown in, the second battery image includes a scratch generated after elastic transformation, and the morphological characteristics of this scratch are similar to those of the scratch in the first battery image.

[0088] As described above, the sample generation method provided by the present invention can be executed in the cloud. Several computing nodes can be deployed in the cloud, and each computing node has processing resources such as computing and storage. In the cloud, a service can be organized to be provided by multiple computing nodes. Of course, a single computing node can also provide one or more services. The way the cloud provides this service can be to provide an external service interface, and users call this service interface to use the corresponding service. The service interface includes forms such as a Software Development Kit (SDK for short) and an Application Programming Interface (API for short).

[0089] For the solution provided in the embodiment of the present invention, the cloud can provide a service interface for the sample generation service. Users call this service interface through the user device to trigger a call request to the cloud. The cloud determines the computing node that responds to the request, and uses the processing resources in this computing node to execute each step provided in the foregoing embodiment. For example Figure 4 what is shown in is that the user terminal triggers a request for multiple battery images with annotation information to the cloud server cluster, and the cloud server responds to this request and executes the tasks of generating sample images and model training.

[0090] In summary, in the surface defect detection task, different types of defects usually have different appearance features, i.e., morphological features. For example, some defects are elongated, some are circular, some have drastic changes in the horizontal direction, some have drastic changes in the vertical direction, and some have drastic changes in both the horizontal and vertical directions. For different types of defects, the transformation space of elastic transformation can be controlled by adjusting the relevant parameters of elastic transformation. Specifically, by controlling the values of the offset matrices in the horizontal and vertical directions, the direction of elastic transformation can be controlled. For example, in an extreme case: setting the offset matrix in the vertical direction to all zeros will result in no elastic transformation in the vertical direction in the subsequent process. By controlling the standard deviation sigma of the Gaussian kernel, the severity of transformation in adjacent regions of elastic transformation can be controlled. By controlling the amplitude control factor alpha of Gaussian filtering, the overall offset degree of elastic transformation can be controlled. By specifically analyzing the morphological features of different types of defects and using the above parameters, more accurate and realistic morphological feature-based transformations can be performed on each type of defect.

[0091] Figure 5 The flowchart of a sample generation method provided by an embodiment of the present invention is as Figure 5 shown, and the method includes the following steps:

[0092] 501. Obtain the first sample image input during the current round of training. At least one product defect area and the defect category corresponding to each of the at least one product defect area are marked in the first sample image.

[0093] 502. Randomly select a first product defect area for the current round of training from the at least one product defect area.

[0094] 503. According to the defect category corresponding to the first product defect area, obtain the method for generating elastic transformation parameters corresponding to the defect category.

[0095] 504. Generate the target elastic transformation parameters corresponding to the first product defect area according to the method for generating elastic transformation parameters, and perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area.

[0096] 505. Train the defect detection model according to the second sample image. The second product defect area and the defect category are marked in the second sample image.

[0097] In this embodiment, an elastic transformation image can be generated online as a sample image. Briefly speaking, the so-called online method means that during the model training process, based on the collected sample images, new sample images are generated in real time through the elastic transformation method for model training. Moreover, the same sample image can be repeatedly used as the input image to generate different sample images for model training.

[0098] Briefly speaking, taking any first sample image among a large number of initially collected sample images containing defects as an example, the first sample image is repeatedly input during multiple rounds of iterative training of the defect detection model. Suppose there is a first product defect area in the first sample image. If the first product defect area is selected during different rounds of training, the corresponding target elastic transformation parameters of the first product defect area are different in different rounds of training. Therefore, the transformed product defects included in each new sample image generated based on the first product defect area during different rounds of training are different. Thus, a large number of new sample images can be generated based on the same product defect area in the same first sample image.

[0099] As described above, there may be more than one defect in the first sample image, so more than one rectangular box will be marked, that is, at least one product defect area is marked in the first sample image, and each product defect area is also correspondingly marked with the corresponding defect category.

[0100] In the model training stage, in each round, a sample image can be randomly selected from the already collected sample set. Therefore, the same sample image may be selected repeatedly. Assume that the first sample image is currently selected, and assume that there are M labeled rectangular boxes in the first sample image, where M is greater than or equal to 1. The M labeled rectangular boxes can be randomly sampled with a set probability to randomly select N labeled rectangular boxes to be used in this round, where N is less than or equal to M. Then, based on the defect categories corresponding to each of the N labeled rectangular boxes, a method for generating elastic transformation parameters corresponding to each of the N labeled rectangular boxes is determined. Furthermore, based on the determined method for generating elastic transformation parameters, the corresponding elastic transformation parameters are generated. The N labeled rectangular box regions are respectively elastically transformed using the corresponding elastic transformation parameters to obtain the final transformed second sample image. The second sample image contains N labeled rectangular boxes, which respectively correspond to the image regions after transformation of the N labeled rectangular box regions in the first sample image. Moreover, each of the N labeled rectangular boxes in the second sample image is associated with the same defect category as the corresponding labeled rectangular box in the first sample image. The second sample image generated in this round is input into the defect detection model for model training. In the next round of training, assume that the first sample image is still selected, and the above processing process is repeated to obtain a new sample image for model training. However, it can be understood that the labeled rectangular boxes selected from the first sample image at this time may be different from the above N labeled rectangular boxes. Even if they contain the same labeled rectangular boxes, because the corresponding elastic transformation parameters finally generated are often not exactly the same, the finally generated new sample image is also different from the above second sample image.

[0101] Among them, the same selected labeled rectangular box will correspond to different elastic transformation parameters in different rounds of training. The reasons are as follows: In the pre-configured method for generating elastic transformation parameters, the specific values of each parameter are often not specified, but only the determination method of each parameter is specified. For example, assume that the horizontal offset matrix is configured to be determined randomly, and the value range is [-1, 1]. Then, for the same labeled rectangular box region, the horizontal offset matrix generated each time often has different values. Another example is that assume that the standard deviation of the Gaussian kernel corresponding to the horizontal offset matrix has a value range of [0.1W, 0.4W], and the standard deviation values used each time from this range are also different.

[0102] By adopting the above online sample image generation method, a large number of high-quality training samples conforming to the morphological characteristics of each defect category can be generated during the model training process, and finally, the trained defect detection model has good performance.

[0103] In addition, an embodiment of the present invention also provides a defect detection method, including the following steps:

[0104] Obtain a first sample image from a training sample set corresponding to an industrial product with surface defects. The first sample image is marked with a first product defect area and the defect category corresponding to the first product defect area;

[0105] Obtain a method for generating elastic transformation parameters corresponding to the defect category, where the method for generating elastic transformation parameters is pre-configured based on the defect morphological features corresponding to the defect category;

[0106] Generate target elastic transformation parameters corresponding to the first product defect area according to the method for generating elastic transformation parameters;

[0107] Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area, and add the second sample image to the training sample set;

[0108] Based on the defect detection model trained using the training sample set, perform defect detection on the industrial product to be detected.

[0109] In practical applications, the above industrial products are, for example, mechanical parts, lithium batteries, packaging cases, etc. The training sample set obtained based on the above sample image generation method can include rich sample images of various defect categories, so that the defect detection model trained based on this training sample set has good performance and ensures the accuracy of the product surface defect detection results.

[0110] In the above embodiments, only any first sample image in the collected training sample set is taken as an example to illustrate the method for sample image expansion. Based on the above method, in each round of iteration of model training, elastic transformation processing is performed on each sample image collected in the training sample set to obtain the sample images used in this round of iteration. Among them, the input image of one round of iteration process consists of N transformed sample images, where the initially collected training sample set includes N sample images, and N is greater than 1.

[0111] The sample generation device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art can understand that these devices can be configured by using commercially available hardware components through the steps taught by this solution.

[0112] Figure 6 The structural schematic diagram of a sample generation device provided for an embodiment of the present invention is as Figure 6 shown. The device includes: a determination module 11, an acquisition module 12, and a processing module 13.

[0113] The determination module 11 is used to determine the first product defect area marked in the first sample image and the defect category corresponding to the first product defect area.

[0114] An acquisition module 12, configured to acquire an elastic transformation parameter generation method corresponding to the defect category, where the elastic transformation parameter generation method is pre-configured based on defect morphological features corresponding to the defect category.

[0115] A processing module 13, configured to generate a target elastic transformation parameter corresponding to the first product defect area according to the elastic transformation parameter generation method, and perform an elastic transformation process on the first product defect area according to the target elastic transformation parameter, so as to obtain a second sample image including a second product defect area.

[0116] Optionally, the apparatus further includes: a training module, configured to train a defect detection model according to the second sample image, where the second product defect area and the defect category are marked in the second sample image.

[0117] Optionally, the determining module 11 is specifically configured to: acquire the first sample image input during the current round of training, where at least one product defect area and the defect category corresponding to each of the at least one product defect area are marked in the first sample image; randomly select the first product defect area for the current round of training from the at least one product defect area; where the first sample image is repeatedly input during multiple rounds of iterative training of the defect detection model; if the first product defect area is selected during different rounds of training, the target elastic transformation parameters corresponding to the first product defect area are different during the different rounds of training.

[0118] Optionally, the target elastic transformation parameter includes: a horizontal offset matrix, a vertical offset matrix, a Gaussian kernel corresponding to the horizontal offset matrix and a horizontal amplitude control factor, and a Gaussian kernel corresponding to the vertical offset matrix and a vertical amplitude control factor.

[0119] Optionally, the defect morphological feature includes the orientation of the defect, and / or the amplitude fluctuation degree of the defect in the horizontal and vertical directions; the acquisition module 12 is specifically configured to: determine a generation method of the horizontal offset matrix and the vertical offset matrix according to the orientation of the defect, and / or the amplitude fluctuation degree of the defect in the horizontal and vertical directions.

[0120] Optionally, the defect morphological feature includes the size of the defect, and the acquisition module 12 is specifically configured to: determine a value range of the Gaussian kernel corresponding to the horizontal offset matrix and the Gaussian kernel corresponding to the vertical offset matrix according to the size of the defect.

[0121] Optionally, the defect morphological feature includes the amplitude fluctuation degrees of the defect in the transverse and longitudinal directions, and the obtaining module 12 is specifically configured to: determine the value ranges of the transverse amplitude control factor and the longitudinal amplitude control factor according to the amplitude fluctuation degrees of the defect in the transverse and longitudinal directions.

[0122] Figure 6 The device shown can execute the steps provided in the foregoing embodiments. For the detailed execution process and technical effects, refer to the descriptions in the foregoing embodiments, which will not be elaborated herein.

[0123] In a possible design, the above Figure 6 The structure of the sample generation device shown can be implemented as an electronic device. As Figure 7 shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. Among them, executable code is stored on the memory 22. When the executable code is executed by the processor 21, the processor 21 can at least implement the sample generation method provided in the foregoing embodiments.

[0124] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium. Executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of an electronic device, the processor can at least implement the sample generation method provided in the foregoing embodiments.

[0125] The device embodiments described above are merely illustrative. The network elements described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of adding a necessary general hardware platform, and of course, can also be implemented by a combination of hardware and software. Based on such an understanding, the above technical solutions in essence or the parts that contribute to the prior art can be embodied in the form of a computer product. The present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a sample, characterized in that, comprising: determining a first product defect area marked in a first sample image and a defect category corresponding to the first product defect area; obtaining a method for generating elastic transformation parameters corresponding to the defect category, wherein the method for generating elastic transformation parameters is pre-configured based on defect morphological features corresponding to the defect category; generating target elastic transformation parameters corresponding to the first product defect area according to the method for generating elastic transformation parameters; performing an elastic transformation process on the first product defect area according to the target elastic transformation parameters to obtain a second sample image including a second product defect area.

2. The method according to claim 1, characterized in that, the method further comprises: training a defect detection model according to the second sample image, where the second product defect area and the defect category are marked in the second sample image.

3. The method according to claim 2, characterized in that, the determining a first product defect area marked in a first sample image and a defect category corresponding to the first product defect area includes: obtaining the first sample image input in the current round of training process, where at least one product defect area and a defect category corresponding to each of the at least one product defect area are marked in the first sample image; randomly selecting the first product defect area for the current round of training process from the at least one product defect area; wherein, the first sample image is repeatedly input during multiple rounds of iterative training of the defect detection model; if the first product defect area is selected during different rounds of training, the target elastic transformation parameters corresponding to the first product defect area are different during the different rounds of training.

4. The method according to claim 1, characterized in that, the target elastic transformation parameters include: a horizontal offset matrix, a vertical offset matrix, a Gaussian kernel corresponding to the horizontal offset matrix and a horizontal amplitude control factor, and a Gaussian kernel corresponding to the vertical offset matrix and a vertical amplitude control factor.

5. The method according to claim 4, characterized in that, the defect morphological features include the orientation of the defect, and / or, the amplitude fluctuation degree of the defect in the horizontal and vertical directions; the method further comprises: determining a method for generating the horizontal offset matrix and the vertical offset matrix according to the orientation of the defect, and / or, the amplitude fluctuation degree of the defect in the horizontal and vertical directions.

6. The method according to claim 4, characterized in that, the defect morphological features include the size of the defect, the method further comprises: determining a value range of the Gaussian kernel corresponding to the horizontal offset matrix and the Gaussian kernel corresponding to the vertical offset matrix according to the size of the defect.

7. The method according to claim 4, characterized in that, the defect morphological features include the amplitude fluctuation degree of the defect in the horizontal and vertical directions, the method further comprises: determining a value range of the horizontal amplitude control factor and the vertical amplitude control factor according to the amplitude fluctuation degree of the defect in the horizontal and vertical directions.

8. A method for generating a sample, characterized in that, comprising: Receive the defect category input by the user and the defect morphological feature analysis items corresponding to the defect category; According to the defect morphological feature analysis items, perform defect morphological feature analysis on multiple sample images corresponding to the defect category to display the defect morphological feature analysis results; Receive the elastic transformation parameter generation method corresponding to the defect category determined by the user according to the defect morphological feature analysis results; Receive the first sample image marked with the first product defect area, and the first product defect area corresponds to the defect category; Generate the target elastic transformation parameter corresponding to the first product defect area according to the elastic transformation parameter generation method; Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameter to obtain a second sample image including a second product defect area.

9. A sample generation device, characterized in that, comprising: A determination module for determining the first product defect area marked in the first sample image and the defect category corresponding to the first product defect area; An acquisition module for acquiring the elastic transformation parameter generation method corresponding to the defect category, wherein the elastic transformation parameter generation method is pre-configured based on the defect morphological features corresponding to the defect category; A processing module for generating the target elastic transformation parameter corresponding to the first product defect area according to the elastic transformation parameter generation method, and performing elastic transformation processing on the first product defect area according to the target elastic transformation parameter to obtain a second sample image including a second product defect area.

10. An electronic device, characterized in that, comprising: A memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor executes the sample generation method according to any one of claims 1 to 7.

11. A non-transitory machine-readable storage medium, characterized in that, An executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by the processor of the electronic device, the processor executes the sample generation method according to any one of claims 1 to 7.

12. A defect detection method, characterized in that, comprising: Obtain a first sample image from a training sample set corresponding to an industrial product with surface defects, where the first sample image is marked with a first product defect area and the defect category corresponding to the first product defect area; Obtain the elastic transformation parameter generation method corresponding to the defect category, wherein the elastic transformation parameter generation method is pre-configured based on the defect morphological features corresponding to the defect category; Generate the target elastic transformation parameter corresponding to the first product defect area according to the elastic transformation parameter generation method; Perform elastic transformation processing on the first product defect area according to the target elastic transformation parameter to obtain a second sample image including a second product defect area, and add the second sample image to the training sample set; Based on the defect detection model trained using the training sample set, perform defect detection on the industrial product to be detected.

Images