Model training and defect detection method, device, equipment and readable storage medium

Abstract

The embodiments of the present application provide a method, device, equipment and readable storage medium for model training and defect detection, which relates to the field of image processing technology, ensures that there are sufficient number of defect image samples and high-quality annotation of samples, and improves the performance of deep learning models. The method includes: obtaining a sample data set, the sample data set includes multiple synthetic images, and the synthetic images are obtained by performing a first data enhancement on the foreground image and the background image; determining a first loss function and a second loss function, the first loss function is used to represent the similarity loss between different synthetic images corresponding to the same foreground image, and the second loss function is used to represent the difference between the predicted value and the true value; inputting the sample data set into the deep learning model, training the deep learning model according to the first loss function and the second loss function, and obtaining a trained deep learning model, and the deep learning model is used to detect defects in the appearance of the device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, apparatus, device and readable storage medium for model training and defect detection. Background Art

[0002] In recent years, with the rapid development of image processing technology, deep learning models have higher flexibility and accuracy, and can detect defects in electronic devices such as cameras, thereby ensuring the product quality and production efficiency of electronic devices.

[0003] Usually, the training process of deep learning models relies on a large number of high-quality labeled samples to learn various types of defects. At present, images are often labeled manually to obtain labeled samples, which is not only time-consuming and labor-intensive, but also prone to subjective errors, resulting in insufficient number of samples and low sample quality, which restricts the performance and application of deep learning models. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and readable storage medium for model training and defect detection, which ensure a sufficient number of samples and high quality of samples, improve the performance of the deep learning model, can accurately detect defects in the appearance of electronic equipment, help detect the quality and quality of electronic equipment, and facilitate the use of electronic equipment.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a model training method is provided, the method comprising:

[0007] Acquire a sample data set, the sample data set includes a plurality of synthetic images, the synthetic images are obtained by performing a first data enhancement fusion on a foreground image and a background image, the foreground image is an image sample of a device having defects in appearance and defect locations, and the background image is an image sample of a device having no defects in appearance;

[0008] Determine a first loss function and a second loss function, the first loss function is used to represent the similarity loss between different synthetic images corresponding to the same foreground image, and the second loss function is used to represent the difference between the predicted value and the true value;

[0009] The sample data set is input into the deep learning model, and the deep learning model is trained according to the first loss function and the second loss function to obtain a trained deep learning model. The deep learning model is used to detect defects in the appearance of the equipment.

[0010] In the model training method provided in the embodiment of the present application, the model training device can obtain a sample data set, and determine a first loss function and a second loss function based on the sample data set. The first loss function can represent the similarity loss between different synthetic images corresponding to the same foreground image, and the second loss function can represent the difference between the predicted value and the true value. Since the defect features and defect types of the foreground image and the initial foreground image have not changed. Therefore, the defect features and defect types of the same foreground image can also be regarded as the defect features and defect types of multiple synthetic images corresponding to the same initial foreground image. Among them, the first loss function and the second loss function can indicate whether the type of defects in the synthetic image is accurately labeled, can calibrate the detection accuracy of the deep learning model, avoid the overfitting phenomenon of the deep learning model during training, and can improve the stability of the deep learning model.

[0011] The model training device inputs the sample data set into the deep learning model according to the first loss function and the second loss function to train the deep learning model, and then obtains the trained deep learning model. With the help of the trained deep learning model, the defects of the appearance of the electronic device can be accurately detected, which is helpful to detect the quality and quality of the electronic device and facilitate the use of the electronic device. In a possible implementation of the first aspect, the model training device trains the deep learning model according to the first loss function and the second loss function, including: training the deep learning model through formula one, formula two and formula three;

[0012] L = L norm + L seg Formula 1;

[0013] Formula 2:

[0014] Formula 3;

[0015] in, L is the third loss function, L norm is the first loss function, L seg is the second loss function, L (·) represents a distance loss, Indicates that the initial foreground image has been transformed by data enhancement, such as rotation, scaling, and other functions. and represents the inverse transformation of the same initial foreground image into different composite images. y p and y qIt represents the result of neural network prediction of the enhanced image obtained after the transformation of the same foreground image. p ( x ) is the true value of the annotation label for each defect type, q ( x ) is the predicted value of the defect in the synthetic image output by the model training device, x is the number of synthesized images.

[0016] The model training device calculates the regularization loss between different synthetic images obtained after the initial foreground image is enhanced by data through the first loss function and the second loss function. The smaller the regularization loss, the closer the defect features between the two synthetic images are, or the closer the images obtained after the two synthetic images are inversely transformed by data enhancement are to the initial foreground image. Conversely, if the regularization loss is larger, it means that the difference between the two synthetic images is larger, that is, the images obtained after the two synthetic images are inversely transformed by data enhancement are less close to the initial foreground image.

[0017] In this way, the first loss function can be used between different synthetic images obtained after the second data enhancement operation of the same initial foreground image. By changing the first loss function, the similarity loss between the two synthetic images can be changed, so that the two synthetic images are closer after the inverse transformation of data enhancement. The influence of factors such as size, angle, grayscale, contrast and noise can be eliminated, and the defect features in the foreground image can be more highlighted, thereby removing artifacts and false features in the synthetic image, facilitating subsequent processing, reducing the problem of inaccurate labeling in the deep learning model, and avoiding the overfitting phenomenon during the training of the deep learning model, thereby improving the quality of the deep learning model.

[0018] In another possible implementation manner of the first aspect, the first data enhancement includes: transparency fusion, transition boundary blurring, and overall intensity equalization.

[0019] The sample generation device performs a first data enhancement on a foreground image and a background image to obtain multiple composite images. The defect type of the composite image is consistent with the defect type of the foreground image, enriching the sample images of this type, increasing the number of defect type samples, and improving artifacts and false features in the composite image. Different types of foreground images can generate different types of composite images through the above data enhancement method, thereby increasing the number of samples and enriching various types of sample images to meet the needs of actual training.

[0020] Among them, transparency fusion, transition boundary blurring and overall intensity equalization are performed sequentially.

[0021] Compared with the related art, the related art only lists multiple processes combined in multiple data enhancements, without considering whether the combination of multiple processes can bring about the effect of improving image quality. In the first data enhancement, the above three processes performed sequentially are the image fusion stage, which can fully consider the characteristics of defects in the appearance image of the electronic device.

[0022] Specifically, the sample generation device first performs a transparency fusion process, which can make the defects in the image less abrupt and the background less obvious, and closer to the actual situation. The sample generation device then performs a transition boundary blurring process, which fully considers the problem that the image is not easy to present clear boundaries during the actual shooting process, and can simulate the actual situation more realistically. The sample generation device finally performs an overall intensity equalization process to make the brightness distribution of the synthesized image uniform. In summary, the quality of a large number of synthesized images is improved.

[0023] In another possible implementation manner of the first aspect, the foreground image is obtained by performing second data enhancement on the initial foreground image, and the second data enhancement includes: at least one of grayscale, equalization, scaling, rotation or contrast adjustment.

[0024] The sample generating device can perform a second data enhancement on the initial foreground image to obtain a foreground image, so that the defect type corresponding to the foreground image can be consistent with the defect type corresponding to the initial foreground image, which is beneficial to simplify the subsequent image processing.

[0025] In another possible implementation manner of the first aspect, the background image is obtained by performing a third data enhancement on the initial background image, and the third data enhancement includes at least one of grayscale, equalization, scaling, rotation, noise addition, or contrast adjustment.

[0026] The sample generation device can perform a third data enhancement on the initial background image to obtain a background image, thereby increasing the diversity of the background image. The diversity of the background image helps the deep learning model learn more background features, so that the deep learning model can adapt to different application scenarios and improve the generalization ability of the deep learning model.

[0027] In a second aspect, a defect detection method is provided. The method is applied to a defect detection device. The defect detection device includes a deep learning model trained and generated by the above-mentioned model training method. The defect detection method includes:

[0028] Capturing an appearance image of the first device;

[0029] Through the deep learning model, defect detection is performed on the appearance image of the first device, and the detection result is output. The detection result is used to indicate the defect situation of the appearance of the first device.

[0030] In summary, the defect detection device first collects the appearance image of the first device, and uses the trained deep learning model to perform defect detection on the collected appearance image. The defect detection device can output whether the appearance of the first device is defective. When it is detected that the first device has defects, the defect type and defect location can also be obtained.

[0031] In summary, the sample generation method proposed in the embodiment of the present application effectively expands the sample data set without increasing the actual data collection cost, so that the deep learning model can be trained using a more diverse sample data set, improve the problem of sample imbalance, and reduce labor costs and time costs. The model training method proposed in the embodiment of the present application reduces the problem of inaccurate labeling of defect types in images by the deep learning model by determining the loss function related to the sample data set, improves the quality of the synthesized image, reduces the risk of overfitting of the deep learning model, and effectively improves the performance of the model. The defect detection method proposed in the embodiment of the present application, by using the above-mentioned trained deep learning model, can detect whether the appearance of the electronic device has defects, and when there are defects in the appearance of the electronic device, the defect type and defect location.

[0032] In another possible implementation manner of the second aspect, the types of the first device include: a screen, a back panel, a frame of the screen, and a camera.

[0033] In a third aspect, a model training device is provided, which includes: a module for executing the aforementioned model training method.

[0034] In a fourth aspect, a defect detection device is provided, the device comprising: a module for executing the aforementioned defect detection method.

[0035] In a fifth aspect, an electronic device is provided, the electronic device comprising:

[0036] one or more processors;

[0037] Memory;

[0038] Among them, one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the electronic device, the electronic device executes the method in the above-mentioned first aspect and any possible implementation of the first aspect; and / or, the electronic device executes the above-mentioned second aspect and any possible implementation of the second aspect.

[0039] In a sixth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the method in the first aspect and any possible implementation of the first aspect; and / or, the electronic device executes the method in the second aspect and any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the architecture of a defect detection system provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a flow chart of a sample generation method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a sample generation method according to a specific embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a composite image of scratch types of a camera provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a composite image of a stain type of a camera provided in an embodiment of the present application;

[0045] Figure 6 A flowchart of a model training method provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a defect detection method provided in an embodiment of the present application;

[0047] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0049] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0050] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0051] In practical applications, deep learning models are often used to detect defects in electronic devices. The deep learning model can extract features from the appearance image of the electronic device and identify the defects of the appearance of the electronic device based on the features, that is, whether there are defects in the appearance of the electronic device, and the type of defects if there are defects in the appearance of the electronic device. For example, taking the electronic device as a camera as an example, the types of defects may include scratches, stains, hair, etc. In this way, the product quality and production efficiency of the electronic device can be guaranteed. The types of defects hereinafter may also be referred to as defect types.

[0052] The training process of deep learning models relies on a large number of high-quality labeled samples. At present, images are often labeled manually to obtain labeled samples. The process is cumbersome and complicated, time-consuming and labor-intensive, and prone to subjective errors, resulting in a decrease in the quantity and quality of samples. In the actual production of electronic equipment, the probability of one or some types of defects appearing on the appearance is low, resulting in a small number of images of the aforementioned defect types, which increases the difficulty of obtaining samples.

[0053] In order to solve the above problems, the embodiments of the present application provide a sample generation method, a model training method, a defect detection method, a sample generation device, a model training device, a defect detection device, an electronic device, and a computer-readable storage medium. The sample generation method is performed by the sample generation device, the model training method is performed by the model training device, and the defect detection method is performed by the defect detection device.

[0054] The sample generation device can obtain a large number of sample data sets by performing data expansion on a small number of sample data sets in a data enhancement manner, thereby ensuring that the number of samples is sufficient.

[0055] The model training device can improve the quality of the sample data set through the loss function related to the sample data set, and then train the deep learning model. Thus, the high quality of the sample is guaranteed and the performance of the deep learning model is improved.

[0056] The defect detection device can detect the appearance image of the electronic device through the trained deep learning model to determine the defect situation of the appearance of the electronic device, such as whether the appearance of the electronic device has defects and what the type of defects of the appearance of the electronic device are.

[0057] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing the architecture of a defect detection system provided by an embodiment of the present application. Figure 1 As shown, the defect detection system may include: a sample generating device, a model training device and a defect detecting device.

[0058] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0059] The sample generation device can perform data expansion on images of various types of defects to obtain a large number of images, thereby expanding the samples for training the deep learning model. That is, the sample generation device can obtain a large number of samples based on a small number of samples, thereby improving the generalization ability of the deep learning model and its robustness in different scenarios.

[0060] The sample generation device may be arranged in a device such as a computer, a server, or a service cluster composed of multiple servers. Of course, the sample generation device may also be arranged in a terminal with strong computing power, such as a mobile phone or a computer.

[0061] The model training device can determine the relevant loss function based on images of various types of defects, alleviate the impact of inaccurate labeling, improve the quality of the image, and thus improve the performance of the deep learning model.

[0062] The model training device can be set in a device such as a computer, a server, or a service cluster composed of multiple servers. Of course, the model training device can also be set in a terminal with strong computing power, such as a mobile phone or a computer.

[0063] The defect detection device can detect the appearance image of the electronic device to determine the defect status of the appearance of the electronic device.

[0064] The defect detection device may be arranged in a device such as a computer, a server, or a service cluster composed of multiple servers. Of course, the defect detection device may also be arranged in a terminal with strong computing power, such as a mobile phone or a computer.

[0065] It should be noted that the sample generation device, the model training device and the defect detection device can be set separately or integrated, and the embodiments of the present application do not limit this.

[0066] Below, based on Figure 1 The defect detection system of the structure shown introduces in detail the specific implementation process of the sample generation method, model training method, and defect detection method of the embodiment of the present application.

[0067] See also Figure 2 , Figure 2 A schematic flow chart of a sample generation method provided in an embodiment of the present application is shown. Figure 2 In the above example, the sample generation method is given by Figure 1 The sample generation device in is executed. Figure 2 As shown, the sample generation method provided in the embodiment of the present application may include:

[0068] S201: A sample generating device obtains an initial foreground image.

[0069] The number of the initial foreground images may be 1. The initial foreground image is a defective image of the appearance of the electronic device. The defective image may include one or more types of defects. For different electronic devices, the types of defects may be different or the same. The initial foreground image may be represented by the pixel position of the defect in the initial foreground image.

[0070] S202: The sample generating device performs a second data enhancement on the initial foreground image to obtain a foreground image.

[0071] The foreground image is a defective image of the appearance of the electronic device. The foreground image can be represented by the pixel position of the defect in the foreground image. The foreground image is an image obtained by enhancing the initial foreground image with the second data. The type of defect in the foreground image is the same as the type of defect in the initial foreground image. The pixel position of the defect in the foreground image is the pixel position obtained by enhancing the pixel position of the defect in the initial foreground image with the second data.

[0072] The second data enhancement may include at least one of grayscale, equalization, scaling, rotation or contrast adjustment.

[0073] The embodiment of the present application adopts a randomly selected enhancement method for the second data enhancement, and is not limited to a certain data enhancement method set in advance. The embodiment of the present application can use any enhancement method as the second data enhancement, or any two enhancement methods as the second data enhancement, first execute one of the enhancement methods, and then execute the other enhancement method to obtain the enhanced image, and can also use any number of enhancement methods combined as the second data enhancement.

[0074] Grayscale refers to the process of converting a color image into a grayscale image. In a grayscale image, each pixel contains only one grayscale value, which is used to represent the brightness information of the grayscale image, but does not contain the color value in a color image, such as the values ​​of the three channels of red, green and blue, which are used to represent the color information of the grayscale image. Since grayscale images only contain brightness information, no color information, and only contain data in one dimension, they help reduce data dimensions, provide better contrast, and make image features easier to analyze and identify.

[0075] In some embodiments, the sample generation device grayscales the foreground image before grayscale by weighted averaging to obtain the foreground image after grayscale. For example, the foreground image after grayscale can be expressed by Formula 4:

[0076] Y = 0.299 R + 0.587 G + 0.114 B Formula 4;

[0077] in, Y Represents the grayscale value of the foreground image after grayscale conversion. R Represents the value of the red channel of the foreground image before grayscale conversion. G Represents the value of the green channel of the foreground image before grayscale conversion. B Indicates the value of the blue channel of the foreground image before grayscale conversion.

[0078] The embodiment of the present application takes the weighted average method to calculate the grayscale image as an example, which does not exclude the use of other forms of calculation methods and is not limited by the embodiment of the present application.

[0079] The main purpose of equalization is to improve the contrast of the image and make the details in the image clearer. Equalization is achieved by redistributing the pixel values ​​of the image. For images with relatively concentrated brightness information, equalization can equalize the brightness by redistributing the pixel values ​​of the image. In other words, when the image is too dark or too bright as a whole, equalization can make the image brightness evenly distributed, making the image bright and dark. In this way, equalization can enhance the details of the image and improve the overall visual effect of the image.

[0080] In some embodiments, the sample generation device uses a histogram equalization method to perform equalization processing on the foreground image before equalization to obtain the equalized foreground image. For example, the equalized foreground image can be expressed by Formula 5:

[0081] Formula 5;

[0082] in, s k Represents the grayscale value of the equalized foreground image, L Represents the number of gray levels of the foreground image before equalization, p j Represents the gray level in the foreground image before equalization j The probability of Indicates rounding down.

[0083] The embodiment of the present application takes the histogram equalization method to calculate the equalized image as an example, and other forms of calculation methods are not excluded, and the embodiment of the present application does not limit it.

[0084] Scaling and rotation are mainly used to adjust the size and direction of an image. Scaling is mainly to change the width and height of an image to obtain a reduced image or an enlarged image. Scaling can be divided into proportional scaling and non-proportional scaling. Among them, proportional scaling means keeping the ratio of the width and height of the image unchanged, while reducing or enlarging the width and height of the image. For example, if an image is enlarged from 200×200 pixels to 400×400 pixels, the ratio of the width and height of the image is still 1, and the ratio of the width and height of the image is kept unchanged, and the image is enlarged in proportion. Non-proportional scaling can change the width and height of the image according to actual needs, such as only reducing or enlarging the width of the image, or only reducing or enlarging the height of the image. For example, if an image is enlarged from 200×200 pixels to 200×400 pixels, only the height of the image is enlarged, and the width of the image is not changed, and the image is enlarged in proportion.

[0085] In some embodiments, the sample generating device adopts a random scaling method to perform random scaling processing on the foreground image before scaling to obtain the foreground image after scaling.

[0086] Rotation refers to the transformation of one or more points around an image at a specified angle to obtain a rotated image. The rotation angle can be 90 degrees, 180 degrees, or a custom rotation angle. Rotation can usually be divided into two types: clockwise rotation and counterclockwise rotation. For example, a 90-degree clockwise rotation around the center point of the image will produce a rotated image.

[0087] In some embodiments, the sample generating device uses a random rotation method to perform random rotation processing on the foreground image before rotation to obtain the foreground image after rotation.

[0088] Contrast adjustment is mainly to enhance or weaken the contrast between light and dark in an image by changing the difference between the brightness values ​​of the pixels in the image. Appropriate contrast adjustment can make the image more layered, while excessive contrast adjustment may lead to problems such as loss of image details. Contrast adjustment can enhance the defect features in the image, thereby improving the detection ability and generalization performance of the model. If contrast adjustment is missing, the generated image may cause the deep learning model to identify defects inaccurately and increase the risk of overfitting the deep learning model, ultimately affecting the detection effect.

[0089] In some embodiments, the sample generation device uses a linear contrast adjustment method to adjust the contrast of the foreground image before contrast adjustment to obtain the foreground image after contrast adjustment. For example, the foreground image after contrast adjustment can be expressed by Formula 6:

[0090] I' = α·I + β Formula 6;

[0091] in, I' is the brightness value of the foreground image after contrast adjustment, I is the brightness value of the foreground image before contrast adjustment, α is the contrast gain coefficient. In the embodiment of the present application, α Taking the uniform distribution of [0,2] as an example, β is the brightness offset. In the embodiment of the present application, β Take the uniform distribution of [-50,50] as an example.

[0092] The embodiment of the present application takes the linear contrast adjustment method to adjust the brightness and darkness difference of the image as an example, but other forms of methods are not excluded and the present application does not impose any limitation.

[0093] In the case where the second data enhancement includes one enhancement mode, the initial foreground image in S201 and S202, and the foreground image in S202 may be described as follows:

[0094] When the second data enhancement only includes graying, the foreground image before graying is the initial foreground image, and the foreground image after graying is the foreground image.

[0095] When the second data enhancement only includes equalization, the foreground image before equalization is the initial foreground image, and the foreground image after equalization is the foreground image.

[0096] When the second data enhancement only includes scaling, the foreground image before scaling is the initial foreground image, and the foreground image after scaling is the foreground image.

[0097] When the second data enhancement only includes rotation, the foreground image before the rotation is the initial foreground image, and the foreground image after the rotation is the foreground image.

[0098] When the second data enhancement only includes contrast adjustment, the foreground image before the contrast adjustment is the initial foreground image, and the foreground image after the contrast adjustment is the foreground image.

[0099] In the case where the second data enhancement includes multiple enhancement modes, the initial foreground image in S201 and S202, and the foreground image in S202 may be described as follows:

[0100] Taking the first enhancement method and the second enhancement method performed sequentially as an example, the foreground image processed by the first enhancement method is the initial foreground image, the foreground image processed by the first enhancement method is the foreground image processed by the second enhancement method, and the foreground image processed by the second enhancement method is the foreground image.

[0101] For example, the first enhancement method is graying, and the second enhancement method is rotation. Then, the foreground image before graying is the initial foreground, the foreground image after graying is the foreground image before rotation, and the foreground image after rotation is the foreground image.

[0102] In summary, the sample generation device can perform a second data enhancement on the initial foreground image to obtain a foreground image, so that the defect type corresponding to the foreground image can be consistent with the defect type corresponding to the initial foreground image, which is conducive to subsequent simplified image processing.

[0103] When the number of initial foreground images is 1, the number of foreground images is 1. Different enhancement methods are used in the second data enhancement to obtain different foreground images, thereby enriching the types of foreground images.

[0104] Among them, in the above-mentioned second data enhancement, graying, equalization, scaling, rotation or contrast adjustment can be performed sequentially or randomly, and the embodiments of the present application are not limited. In some embodiments, the sample generation device can sequentially perform: graying, equalization, random scaling, random rotation and contrast adjustment.

[0105] Among them, in the above-mentioned second data enhancement, graying, equalization, scaling, rotation or contrast adjustment can be performed only once or multiple times, and the embodiment of the present application does not limit this.

[0106] In addition, in addition to the five enhancement methods of grayscale, equalization, scaling, rotation and contrast adjustment, the second data enhancement may also include other processing methods, which are only examples and are not limited here. It should be noted that the second data enhancement method is not limited to the above five enhancement methods and their combinations, and may also be other enhancement methods that can perform data enhancement.

[0107] S203: The sample generating device obtains an initial background image.

[0108] The number of initial background images may be 1. The initial background image is an image of the appearance of the electronic device without defects. For example, the initial background image may be an appearance image of the electronic device such as a screen, a back panel, a frame of the screen, and a camera, and the appearance image has no defects.

[0109] S204: The sample generating device performs third data enhancement on the initial background image to obtain a background image.

[0110] The background image is an image of the electronic device without any defects in appearance. The background image is an image obtained by performing the third data enhancement on the initial background image.

[0111] The third data enhancement may include at least one of grayscale, equalization, noise addition, contrast adjustment, scaling or rotation.

[0112] The embodiment of the present application adopts a randomly selected enhancement method for the third data enhancement, and is not limited to a certain data enhancement method set in advance. The embodiment of the present application can use any enhancement method as the third data enhancement, or any two enhancement methods as the third data enhancement, first execute one of the enhancement methods, and then execute the other enhancement method to obtain the enhanced image, and can also use any number of enhancement methods combined as the third data enhancement.

[0113] The grayscale conversion is described in the above step S202 and will not be described in detail here.

[0114] In some embodiments, the sample generation device grayscales the background image before grayscale by weighted average to obtain the grayscale image. The present application embodiment uses the weighted average method to calculate the grayscale image as an example, and other forms of calculation methods are not excluded, and the present application embodiment does not limit it.

[0115] The equalization is described in the above step S202 and will not be described in detail here.

[0116] In some embodiments, the sample generation device uses a histogram equalization method to equalize the background image before equalization to obtain an image after equalization. The present application embodiment uses the histogram equalization method to calculate the equalized image as an example, and other forms of calculation methods are not excluded, and the present application embodiment does not limit it.

[0117] The contrast adjustment is described in the above step S202 and will not be described in detail here.

[0118] In some embodiments, the sample generation device uses a linear contrast adjustment method to adjust the contrast of the background image before the contrast adjustment to obtain an adjusted image. The embodiment of the present application takes the linear contrast adjustment method to adjust the brightness difference of the image as an example, but other methods are not excluded, and the present application does not limit it.

[0119] The scaling is described in the above step S202 and will not be described in detail here.

[0120] In some embodiments, the sample generating device uses a random scaling method to perform random scaling processing on the background image before scaling to obtain the scaled image.

[0121] The rotation is described in the above step S202 and will not be described in detail here.

[0122] In some embodiments, the sample generating device uses a random rotation method to perform random rotation processing on the background image before rotation to obtain the rotated image.

[0123] The noise can be classified into Gaussian noise, Poisson noise, multiplicative noise or salt and pepper noise, etc., and the embodiments of the present application do not limit which type of noise is used. The initial background image can increase the diversity of the background image by adding random noise.

[0124] In some embodiments, the sample generation device performs noise processing on the background image before noise addition by randomly adding noise to obtain the image after noise addition.

[0125] It should be noted that in step S202, random noise may also be added to the initial foreground image, which is not limited in the embodiment of the present application.

[0126] In the case where the third data enhancement includes one enhancement mode, the initial background image in S203 and S204, and the background image in S204 may be described as follows:

[0127] When the third data enhancement only includes graying, the image before graying is the initial background image, and the image after graying is the background image after the third data enhancement.

[0128] When the third data enhancement only includes equalization, the image before equalization is the initial background image, and the image after equalization is the background image after the third data enhancement.

[0129] When the third data enhancement only includes adding noise, the image before adding noise is the initial background image, and the image after adding noise is the background image after the third data enhancement.

[0130] When the third data enhancement only includes contrast adjustment, the image before the adjustment is the initial background image, and the image after the adjustment is the background image after the third data enhancement.

[0131] When the third data enhancement only includes scaling, the image before scaling is the initial background image, and the image after scaling is the background image after the third data enhancement.

[0132] When the third data enhancement only includes rotation, the image before rotation is the initial background image, and the image after rotation is the background image after the third data enhancement.

[0133] In the case where the third data enhancement includes multiple enhancement modes, the initial background image in S203 and S204, and the background image in S204 may be described as follows:

[0134] Taking the first enhancement method and the second enhancement method performed sequentially as an example, the image processed by the first enhancement method is the initial background image, the background image processed by the first enhancement method is used as the background image processed by the second enhancement method, and the background image processed by the second enhancement method is the background image.

[0135] For example, the first enhancement method is adding noise, and the second enhancement method is scaling. Then, the background image before adding noise is the initial background image, the background image after adding noise is the foreground image before scaling, and the image after scaling is the background image.

[0136] In summary, the sample generation device can perform a third data enhancement on the initial background image to obtain a background image, thereby increasing the diversity of the background image. The diversity of the background image helps the deep learning model learn more background features, so that the deep learning model can adapt to different application scenarios and improve the generalization ability of the deep learning model.

[0137] When the number of initial background images is 1, the number of background images is 1. In the third data enhancement, different enhancement methods are adopted to obtain different background images, thereby enriching the background images.

[0138] The third data enhancement may also include other processing, which is only an example and not a limitation. It should be noted that the third data enhancement method is not limited to the above six enhancement methods and their combinations, and may also be other enhancement methods that can perform data enhancement.

[0139] Among them, in the above-mentioned third data enhancement, graying, equalization, adding noise, contrast adjustment, scaling or rotation can be performed sequentially or randomly, and the embodiment of the present application is not limited. In some embodiments, the sample generation device can sequentially perform: graying, equalization, adding noise, contrast adjustment, scaling and rotation.

[0140] Among them, in the above-mentioned third data enhancement, graying, equalization, noise addition, contrast adjustment, scaling or rotation can be performed only once or multiple times, and the embodiment of the present application does not limit this.

[0141] In addition, in addition to the six enhancement methods of grayscale, equalization, noise addition, contrast adjustment, scaling or rotation, the third data enhancement may also include other processing methods, which are only examples and are not limited here. It should be noted that the third data enhancement method is not limited to the above six enhancement methods and their combinations, and may also be other enhancement methods that can perform data enhancement.

[0142] S205: The sample generating device performs a first data enhancement on the foreground image and the background image to obtain a plurality of composite images.

[0143] When the number of foreground images is 1 and the number of background images is 1, the number of composite images can be multiple. The defect types of the multiple composite images are the same defect type. When the foreground image contains multiple defect types, the defect types of the multiple composite images are consistent with the defect type of the foreground image. The number of composite images can be set according to actual needs.

[0144] The first data enhancement may include three processes: transparency fusion, transition boundary blurring, and overall intensity equalization. The execution order of the first data enhancement is fixed, and the first data enhancement may be executed in the order of transparency fusion, transition boundary blurring, and overall intensity equalization.

[0145] Transparency fusion refers to the process of combining two images according to a certain transparency ratio to generate a new image. Specifically, it is the process of superimposing one image on another image through transparency fusion. α The value (transparency information) of the corresponding position of another image α When two images are fused, if there is a lack of transparency fusion, the resulting image may lack realism.

[0146] In some embodiments, the sample generation device uses a weighted average method to perform transparency fusion processing on the foreground image and the background image to obtain a fused image. For example, the fused image can be expressed by Formula 7:

[0147] I = α · A +(1- α ) B Formula 7;

[0148] in, I Represents the image after transparency fusion, A represents the foreground image, B Represents the background image, α represents the transparency parameter, α It follows a uniform distribution in [0,1]. α The closer it is to 1, the more opaque the image color is. α A value of 0 means the image color is completely transparent.

[0149] The embodiment of the present application takes the weighted average method to calculate the transparency fusion image as an example, but other forms of methods are not excluded and the present application does not impose any limitation.

[0150] Blurred transition boundaries refer to the process of smoothing the edges of images after transparency fusion. Blurred transition boundaries can make the edge transition of the image after transparency fusion more natural and more realistic to simulate the actual situation. If blurred transition boundaries are missing, the edges of the generated image may appear too sharp and unnatural, causing the deep learning model to perform poorly when processing actual blurred scenes.

[0151] In some embodiments, the sample generation device uses Gaussian blur to blur the transition boundary of the transparency-fused image to obtain a blurred image. For example, the blurred image can be expressed by Formula 8:

[0152] Formula 8;

[0153] in, I' ( x , y ) indicates that the blurred image is at position ( x , y ), k Represents the radius of the Gaussian kernel. In the present embodiment, k =6 as an example, G ( i , j) represents the weight of the Gaussian kernel. In the embodiment of the present application, the standard deviation of the Gaussian kernel is 2 as an example. I ( x + i , y + j ) indicates that the image after transparency fusion is at position ( x + i , y + j )’s pixel value.

[0154] The embodiment of the present application uses Gaussian blur to calculate the blurred transition boundary image as an example, but other forms of methods are not excluded and the present application does not impose any limitation.

[0155] Overall intensity equalization refers to equalizing the image after the blurred transition boundary. Equalization can equalize the brightness by redistributing the pixel values ​​of the image to obtain an equalized image. In this way, equalization can enhance the details of the image, make the brightness distribution of the image more uniform, and improve the overall visual effect of the image. For the specific process, please refer to the equalization description in the above step S202, which will not be repeated here.

[0156] In some embodiments, the sample generation device uses a histogram equalization method to equalize the image after the blurred transition boundary to obtain an equalized composite image. For example, the equalized composite image can be expressed by Formula 5:

[0157] Formula 5;

[0158] in, s k represents the gray value of the equalized composite image, L represents the number of gray levels of the synthesized image, p j Represents the gray level in the image after the blurred transition boundary j The probability of Indicates rounding down.

[0159] The embodiment of the present application takes the histogram equalization method to calculate the equalized image as an example, and other forms of calculation methods are not excluded, and the embodiment of the present application does not limit it.

[0160] In summary, an initial foreground image includes one defect type. After the above operations S201-S205, the sample generation device can obtain multiple composite images of one defect type, and the defect types of the multiple composite images are consistent with the defect type in the initial foreground image. An initial foreground image includes multiple defect types. After the above operations S201-S205, the sample generation device can obtain multiple composite images of multiple defect types, and the defect types of the multiple composite images are consistent with the defect type in the initial foreground image.

[0161] When the initial foreground image is not changed and the initial background image is changed, after the above-mentioned operations S201-S205, the sample generating device can obtain multiple composite images. The multiple composite images here are different from the multiple composite images generated before the initial background image is replaced, and the defect types of the multiple composite images here are consistent with those of the multiple composite images generated before the initial background image is replaced.

[0162] Thus, the sample data sets of the above defect types are enriched. For the foreground images of each defect type, the above S201-S205 operations can be repeated to obtain multiple synthetic images of each defect type, enriching the sample data sets of other defect types. In this way, the sample generation device can use a small number of samples to obtain a large number of samples, and the defect types of the samples are various, which increases the number and diversity of samples, removes artifacts and false features in the synthetic images, improves the quality of the model, and improves the robustness of the model in different scenarios.

[0163] In a specific embodiment, in combination Figure 2 The sample generation device adopts the sample generation method of the above-mentioned embodiment of the present application to obtain a large number of sample data sets.

[0164] For details, please refer to Figure 3 and Figure 4-Figure 5 , Figure 3 A schematic diagram of a sample generation method in a specific embodiment is shown. Figure 4 Schematic diagram showing the camera's process of synthesizing images of scratch types, Figure 5 A schematic diagram showing the process of synthesizing images of camera stain types. Figure 3 , Figure 4 and Figure 5 In the figure, the electronic device is taken as a camera as an example.

[0165] like Figure 3 As shown, the sample generation method of the embodiment of the present application may include:

[0166] Step 31, the sample generation device randomly selects a foreground image set { x}Select an image x ,x This is the initial foreground image, and determines the defect type in the image y Thus, the sample generation device can obtain an initial foreground image. The foreground image set can be a manually annotated image set or an image set selected from a public data set.

[0167] Among them, the defects in the initial foreground image can be expressed using Formula 9:

[0168] Formula 9;

[0169] in, represents the defects in the initial foreground image, x represents the initial foreground image, y Representing images x Defect type annotation label in the annotation label y The defect type and the coordinate position of the defect can be indicated. Here, the embodiment of the present application is described by taking a defect type as an example.

[0170] Thus, Formula 9 can simply represent the image x Defects in the image x The remaining positions in can be set to 0.

[0171] Take the case where the defect type of the camera is a scratch, for example. Figure 4 As shown, the initial foreground image can be expressed as Figure 4 401. Initial foreground image defect labeling y It can indicate the defect type is a scratch and the coordinate position of the scratch.

[0172] Step 32: the sample generation device generates an initial foreground image x Perform the second data enhancement to obtain the foreground image.

[0173] Taking the second data enhancement including sequentially performing grayscale, equalization, scaling, rotation and contrast adjustment as an example, the sample generating device performs the aforementioned second data enhancement on the initial foreground image to obtain a foreground image.

[0174] Specifically, the initial foreground image is grayscaled to obtain a grayscaled foreground image, the grayscaled foreground image is equalized to obtain a equalized foreground image, the equalized foreground image is randomly scaled to obtain a scaled foreground image, the scaled foreground image is randomly rotated to obtain a rotated foreground image, the rotated foreground image is contrast adjusted to obtain a contrast-adjusted foreground image, and the contrast-adjusted foreground image is the foreground image.

[0175] Step 33, the sample generation device randomly selects from the background image set {x Select an image from x 0 as the initial background image. Thus, the sample generating device can obtain the initial background image.

[0176] Among them, the initial background image can be Figure 4 In 402, the initial background image is an image of the camera's appearance without scratches.

[0177] Step 34: the sample generation device generates the initial background image x 0 performs the third data enhancement to obtain the background image.

[0178] Taking the third data enhancement including sequentially executing grayscale, equalization, noise addition and contrast adjustment as an example, the sample generation device performs the aforementioned third data enhancement on the initial background image to obtain the background image.

[0179] Specifically, the initial background image x 0 is grayed to obtain a grayed background image, the grayed background image is equalized to obtain a equalized background image, the equalized background image is noised to obtain a noisy background image, the noisy background image is contrast adjusted to obtain a contrast-adjusted background image, and the contrast-adjusted background image is the background image.

[0180] In step 35, the sample generating device performs a first data enhancement on the foreground image obtained in step 32 and the background image obtained in step 34 to obtain a plurality of composite images.

[0181] The sample generation device stores multiple synthetic images and the defect label corresponding to each synthetic image in the foreground image set { x}middle.

[0182] Taking the first data enhancement including sequentially performing transparency fusion, blurring transition boundaries and overall intensity equalization as an example, the sample generation device performs the first data enhancement on the foreground image obtained in step 32 and the background image obtained in step 34 to obtain multiple composite images. x aug .

[0183] Among them, the synthetic image x aug The defect label is the label of the defect of the initial foreground image y After the second data enhancement and the first data enhancement, the new defect label is obtained . Defect labels for synthetic images It can indicate the defect type and the coordinate location of the defect. Defect label of the synthetic image Can be a labely The true label obtained after the second data enhancement and the first data enhancement may not be the true label. y If an error occurs during annotation, the defect label of the synthesized image There will be deviations, defect labels It’s not a real label.

[0184] Combination Figure 4 In the present application, the composite image is 2 images as an example, and multiple composite images can be Figure 4 403 and 404 in the figure. The defect types of the multiple synthesized images are consistent with those of the initial foreground image. The defect types of the multiple synthesized images are consistent, but the locations of the defects are different.

[0185] After the above steps 31 to 35, the sample generation device performs data enhancement on an initial foreground image and an initial background image to generate multiple synthetic images. The defect type of the synthetic image is consistent with the defect type of the initial foreground image, enriching the sample image of the defect type.

[0186] For foreground images of different defect types, the sample generating device may repeat the above steps 31 to 35 to obtain multiple composite images of different defect types, thereby enriching sample images of other defect types.

[0187] Combination Figure 5 Taking the defect type of the camera appearance as a stain as an example, the sample generating device can repeat the above steps 31 to 35, and obtain multiple composite images 503 and 504 through an initial foreground image 501 and an initial background image 502, thereby enriching the sample images of other defect types.

[0188] The defect types of the multiple synthesized images 503 and 504 are consistent with those of the initial foreground image 501. The defect types of the multiple synthesized images 503 and 504 are consistent, but the locations of the defects are different.

[0189] pass Figure 4 and Figure 5 It can be seen that the composite images 403 and 404, as well as the composite images 503 and 504, have different defect types and different defect positions. It can be seen that the sample generation device adopts the sample generation method of the embodiment of the present application, which increases the number of samples and enriches various types of sample images to meet the needs of actual training.

[0190] The sample generation method provided in the embodiment of the present application can be divided into two stages through the sample generation device to perform data enhancement on the initial foreground image. In the first stage, the sample generation device performs second data enhancement on an initial foreground image to obtain a foreground image. Different enhancement methods in the second data enhancement can be used to obtain different foreground images, thereby increasing the diversity of foreground images and enriching the number of the above-mentioned types of images.

[0191] The sample generation device can be divided into two stages to perform data enhancement on the initial background image. In the first stage, the sample generation device performs the third data enhancement on an initial background image to obtain a background image. By using different enhancement methods in the third data enhancement, different background images can be obtained, which enriches the background image and helps the deep learning model learn more background features, so that the deep learning model can adapt to different application scenarios and improve the generalization ability of the deep learning model.

[0192] In the second stage, the sample generation device needs to fuse the foreground image and the background image. Furthermore, the sample generation device performs a first data enhancement on one foreground image and one background image to obtain one composite image. By repeating the above steps, multiple composite images can be obtained.

[0193] In summary, the sample generation device obtains multiple synthetic images by subjecting an initial foreground image and an initial background image to the second data enhancement, the third data enhancement, and the first data enhancement. The defect type of the synthetic image is consistent with the defect type of the foreground image, enriching the sample images of this type, increasing the number of defect type samples, and improving artifacts and false features in the synthetic image. Different types of foreground images can generate different types of synthetic images through the above data enhancement method, thereby increasing the number of samples and enriching various types of sample images to meet the needs of actual training.

[0194] Based on the description of the above embodiment, the embodiment of the present application provides a model training method. Specifically, the model training device can use multiple synthetic images of various defect types in the above embodiment as sample data sets. Thus, the model training model can train the deep learning model according to the sample data sets.

[0195] Next, combine Figure 6 , which introduces in detail the specific implementation process of the model training device to train the deep learning model.

[0196] See also Figure 6 , Figure 6 A flow chart of a model training method provided in an embodiment of the present application is shown. Figure 6 As shown, the model training method provided in the embodiment of the present application may include:

[0197] S601. A model training device obtains a sample data set, where the sample data set includes a plurality of composite images, where the composite images are obtained by performing a first data enhancement fusion on a foreground image and a background image, where the foreground image is an image sample of an electronic device having defects on its appearance and defect locations, and the background image is an image sample of an electronic device having no defects on its appearance.

[0198] According to the above embodiment, the sample data set can be obtained from the sample generation device. Figure 2 , the sample generation device selects an initial foreground image through step S201, and the initial foreground image is enhanced by the second data through step S202 to obtain a foreground image. The sample generation device selects an initial background image through step S203, and the initial background image is enhanced by the third data through step S204 to obtain a background image. The sample generation device obtains a composite image through step S205 by performing the first data enhancement on the obtained foreground image and background image. Repeating the above steps, multiple composite images of the same type can be obtained. Different types of foreground images are subjected to the above steps to obtain multiple images of different types. The obtained multiple composite images are placed in the same sample data set, which is the sample data set that the model training device needs to obtain.

[0199] S602: The model training device determines a first loss function and a second loss function.

[0200] Among them, the first loss function is used to represent the similarity loss between different synthetic images corresponding to the same foreground image.

[0201] The model training device can determine the first loss function based on the sample data set. One initial foreground image and one initial background image can be regarded as a group. The same group of initial foreground images and initial background images can generate multiple synthetic images. The multiple synthetic images are different, and the similarities between different synthetic images are different. The first loss function refers to the similarity loss between different initial foreground images obtained after different synthetic images are inversely transformed by data enhancement. The initial foreground image is subjected to the second data enhancement to obtain a foreground image, and one foreground image and one background image are subjected to the first data enhancement to obtain multiple synthetic images. The above steps S201-S205 are repeated with the same initial foreground image and an initial background image, and the sample generation device can obtain multiple synthetic images of the same defect type. The similarities between different synthetic images of the same defect type are different.

[0202] In addition, when the initial foreground image is replaced, the sample generation device executes the above steps S201-S205 to obtain multiple synthetic images. The defect types of the synthetic images here are different from those of the synthetic images generated before replacing the initial foreground image. In this way, the similarities between different synthetic images are different.

[0203] When the initial background image is replaced, the sample generating device executes the above steps S201 to S205 to obtain multiple synthetic images. The synthetic images here are different from the synthetic images generated before the initial background image is replaced, and the defect types are different. Thus, the similarities between different synthetic images are different.

[0204] In the embodiment of the present application, two composite images are taken as an example, and the first loss function can be used to calculate the similarity loss between two initial foreground images obtained after the two composite images are respectively inverse transformed.

[0205] In some embodiments, the model training device uses L 2 Regularized loss function, calculate the loss between different synthetic images corresponding to the same initial foreground image. For example, the first loss function can be expressed by formula 2:

[0206] Formula 2:

[0207] in, L norm is the first loss function, L ( · ) represents a distance loss, Indicates that the initial foreground image has been transformed by data enhancement, such as rotation, scaling, and other functions. and represents the inverse transformation of the same initial foreground image into different composite images. y p and y q They represent different synthetic images respectively. The different synthetic images are obtained by enhancing the same set of foreground images and background images through the first data. The foreground image is obtained by enhancing the initial foreground image through the second data. The background image is obtained by enhancing the initial background image through the third data.

[0208] The model training device calculates the regularization loss between different synthetic images obtained after the initial foreground image is enhanced by data through the first loss function. The smaller the regularization loss, the closer the defect features between the two synthetic images are, or the closer the images obtained after the two synthetic images are inversely transformed by data enhancement are to the initial foreground image. Conversely, if the regularization loss is larger, it means that the difference between the two synthetic images is larger, that is, the images obtained after the two synthetic images are inversely transformed by data enhancement are less close to the initial foreground image.

[0209] In this way, the same set of foreground images undergoes the above Figure 2 The data enhancement operation in the data enhancement operation can use the first loss function between the different synthetic images obtained. By reducing the first loss function, the similarity loss between the two synthetic images can be reduced, so that the two synthetic images can be closer after the inverse transformation of the data enhancement. The influence of factors such as size, angle, grayscale and contrast can be eliminated, and the defect features in the foreground image can be more highlighted, so as to remove artifacts and false features in the synthetic image, facilitate subsequent processing, reduce the problem of inaccurate labeling in the deep learning model, and avoid the overfitting phenomenon during the training of the deep learning model, thereby improving the quality of the deep learning model.

[0210] The second loss function is used to represent the difference between the predicted value and the true value of the defect in the synthetic image.

[0211] The model training device can typically output a predicted value of a defect in each synthetic image, and a second loss function can be used to measure the difference between the predicted value and the true value of each synthetic image.

[0212] The model training device can determine the second loss function according to the sample data set. The sample data set is composed of multiple synthetic images, and the synthetic image is generated by one initial foreground image and one initial background image. One initial foreground image and one initial background image repeat the above steps S201 to S205 to obtain multiple synthetic images. Usually, there is a certain difference between the predicted value and the true value of the defect in the synthetic image.

[0213] In some embodiments, the model training device uses a cross entropy loss function to calculate the difference between the predicted value and the true value of the defect in the synthetic image. For example, the second loss function can be expressed using Formula 3:

[0214] Formula 3;

[0215] in, L seg represents the cross entropy loss function, p (x ) is the true value of the annotation label for each defect type, q ( x ) is the predicted value of the defect in the synthetic image output by the model training device, x is the number of synthesized images.

[0216] The cross entropy loss function calculates the negative log-likelihood of the predicted value corresponding to the true value of the defect type. The closer the predicted value of the defect type in the synthetic image is to the true value, the smaller the cross entropy loss will be; conversely, the greater the difference between the predicted value of the defect type in the synthetic image and the true value, the greater the cross entropy loss will be.

[0217] The embodiment of the present application takes the cross entropy loss function as an example, and does not exclude the use of other forms of calculation methods, which are not limited in the embodiment of the present application.

[0218] In some embodiments, the model training device uses a third loss function to calculate the difference between the predicted value and the true value of the defect in the synthetic image. The third loss function includes the first loss function and the second loss function. The model training device reduces the problem of inaccurate defect labeling in the synthetic image by determining the third loss function.

[0219] In some embodiments, the third loss function may be the sum of the first loss function and the second loss function, with the first loss function being L 2 Regularization loss function L norm , the second loss function is the cross entropy loss function L seg For example, the regularized loss function L norm and the cross entropy loss function L seg The sum can be expressed using formula 1:

[0220] L = L norm + L seg Formula 1;

[0221] in, L represents the third loss function, L norm represents the first loss function, L norm The calculation process of can refer to the formula 2 in the aforementioned S602, L seg represents the second loss function, L seg The calculation process of can refer to formula 3 in S602, which will not be repeated here.

[0222] The model training device calculates the regularization loss and cross entropy loss between different synthetic images obtained after data enhancement of the initial foreground image through the third loss function. Among them, the third loss function is the sum of the first loss function and the second loss function. The smaller the third loss function is, the more accurate the defect feature annotation of the synthetic image is. Conversely, if the third loss function is larger, it means that the defect feature annotation of the synthetic image is less accurate.

[0223] Thus, the initial foreground image is Figure 2 The data enhancement operation in the data can use the third loss function to obtain multiple synthetic images. Through the third loss function, the influence of factors such as size, angle, grayscale, contrast and noise can be eliminated, and the defect features in the foreground image can be more highlighted, thereby removing artifacts and false features in the synthetic image, facilitating subsequent processing, reducing the problem of inaccurate labeling in the deep learning model, and avoiding the overfitting phenomenon during the training of the deep learning model, thereby improving the quality of the deep learning model.

[0224] S603: The model training device inputs the sample data set into the deep learning model, and trains the deep learning model according to the first loss function and the second loss function to obtain a trained deep learning model, and the trained deep learning model is used to detect defects in the appearance of the electronic device.

[0225] The model training device can input the sample data set into the deep learning model according to the first loss function and the second loss function, train the deep learning model, and obtain the trained deep learning model.

[0226] Among them, the trained deep learning model can be used to detect defects in the appearance of electronic devices. For example, the defect condition of the appearance of the electronic device can be a type without defects, or when the appearance of the electronic device has defects, the type of defect and the location of the defect. The embodiment of the present application does not limit the type of deep learning model. For example, the type of deep learning model can include but is not limited to: TransUNet, FCN, UNet, SegNet, DeepLab and YOLO series deep learning models.

[0227] Below, taking the TransUNet deep learning model as an example, the model training device can set parameter initialization. Assuming that the TransUNet deep learning model uses the Adam optimizer, the model training device can set the initial learning rate (learningrate) to 0.0001. The model training device can use an adjustment strategy of adjusting the learning rate (StepLR) at equal intervals, set the number of training data samples in a batch (batchsize) to 8, and train a total of 200 data set training cycles (epochs).

[0228] In addition, the model training device may also set different parameters according to different electronic devices, without excluding other training parameters, and the embodiments of the present application are not limited thereto.

[0229] In the training process of the TransUNet deep learning model, the model training device obtains a sample data set through step S601, and the sample data set includes multiple synthetic images. Specifically, the synthetic image is a sample data set obtained by the sample generation device. The sample generation device combines Figure 2 ,Through data enhancement, synthetic images of different defect types can be obtained, enriching the ,sample data sets of various defect types.

[0230] The model training device determines a first loss function and a second loss function related to the sample data set according to the obtained sample data set in step S602. The first loss function can be L 2 regularization loss function, the second loss function can be a cross entropy loss function, and the quality of the sample data set can be improved with the help of the first loss function and the second loss function.

[0231] The model training device inputs the sample data set into the TransUNet deep learning model through step S603, and L 2. Regularize the loss function, train the TransUNet deep learning model, and obtain the trained TransUNet deep learning model. The trained TransUNet deep learning model can be used to detect defects in the appearance of electronic equipment.

[0232] In the model training method provided in the embodiment of the present application, the model training device can obtain a sample data set, and determine a first loss function based on the sample data set. The first loss function can be used to represent the similarity loss between different synthetic images corresponding to the same foreground image, and the second loss function is used to represent the difference between the predicted value and the true value. Since the defect features and defect types of the foreground image and the initial foreground image have not changed. Therefore, the defect features and defect types of the same foreground image can also be regarded as the defect features and defect types of multiple synthetic images corresponding to the same initial foreground image. Among them, the first loss function and the second loss function can indicate whether the type of defects in the synthetic image is accurately labeled, can calibrate the detection accuracy of the deep learning model, avoid the overfitting phenomenon of the deep learning model during training, and can improve the stability of the deep learning model.

[0233] The model training device inputs the sample data set into the deep learning model according to the first loss function and the second loss function to train the deep learning model, thereby obtaining a trained deep learning model. With the help of the trained deep learning model, defects in the appearance of electronic equipment can be accurately detected, which helps to detect the quality of electronic equipment and facilitate the use of electronic equipment.

[0234] Based on the above description, the sample generation device can obtain a large number of sample data sets through a small number of sample data sets, thereby reducing the problem of sample imbalance. The model training device obtains the above-mentioned large number of sample data sets, determines the loss function related to the sample data set, and inputs the sample data set into the model training device. The model training device trains the deep learning model according to the loss function to obtain the trained deep learning model, which can reduce the problem of inaccurate model labeling, avoid overfitting during deep learning model training, and improve the stability of the deep learning model.

[0235] Based on this, the model training device can use the sample data set to train the deep learning training model to obtain a trained deep learning model. The model training device can use a general data set to train the deep learning training model to obtain a trained deep learning model. The sample data set refers to a data set obtained using the sample generation method of the embodiment of the present application. The general data set refers to a data set obtained without using the sample generation method of the embodiment of the present application.

[0236] In order to distinguish the above two deep learning models, the deep learning model obtained after training using the sample data set is called deep learning model 1, and the deep learning model obtained after training using the general data set is called deep learning model 2.

[0237] The deep learning model 1 and the deep learning model 2 can be compared using evaluation indicators. For example, the evaluation indicators may include intersection over union (IoU) and / or Dice Similarity coefficient (dice similarity coefficient), etc.

[0238] in, IoU It is a parameter indicator used to evaluate the degree of match between the pixel-level detection results in an image and the true results. IoU It can be expressed by formula 10:

[0239] Formula 10;

[0240] in, IoUIt refers to the ratio of the area of ​​the intersection of the test result and the true result to the area of ​​their union, that is, the ratio of the area of ​​the repeated part of the test result and the true result to the area of ​​the sum of the test result and the true result. A is the detection result of defects in the image, B is the true result of the defect in the image.

[0241] IoU The higher the value, the better the test result. IoU The closer the value is to 1, the closer the test result is to the true result, and vice versa.

[0242] in, Dice It is a parameter indicator used to measure the similarity between the predicted results and the actual results. Dice It can be expressed by formula 11:

[0243] Formula XI;

[0244] in, A is the detection result of defects in the image, B is the true result of the defect in the image. Dice The value range is between 0 and 1. Dice The larger the value is, the closer the test result is to the true result. Conversely, the smaller the value is, the closer the test result is to the true result.

[0245] Table 1 shows the performance of deep learning model 1 and deep learning model 2. IoU The comparison results.

[0246] Table 1

[0247]

[0248] It can be seen from Table 1 that when the number of sample images is the same, the deep learning model 1 IoU The percentages of IoU As the number of sample images increases, the performance of deep learning model 1 IoU The percentage of deep learning model 2 is gradually increasing. IoU The percentage gradually increases.

[0249] In Table 1, the sample images of deep learning model 1 refer to the synthetic images in the sample dataset, and the sample images of deep learning model 1 refer to the images in the general dataset.

[0250] It can be seen that regardless of the number of sample images, the detection performance of deep learning model 1 is higher than that of deep learning model 2.

[0251] Table 2 shows the performance of deep learning model 1 and deep learning model 2. Dic The comparison result of e.

[0252] Table 2

[0253]

[0254] It can be seen from Table 2 that when the number of sample images is the same, the deep learning model 1 Dic The percentage of e is greater than that of deep learning model 2 Dic When the number of sample images increases, the deep learning model 1 Dic The percentage of e gradually increases. Dic The percentage of e gradually increases.

[0255] In Table 2, the sample images of deep learning model 1 refer to the synthetic images in the sample dataset, and the sample images of deep learning model 1 refer to the images in the general dataset.

[0256] It can be seen that regardless of the number of sample images, the detection performance of deep learning model 1 is higher than that of deep learning model 2.

[0257] Based on the above description, the embodiment of the present application provides a defect detection method. Specifically, the defect detection device can use the deep learning model trained in the above embodiment to perform defect detection on the appearance image of the electronic device.

[0258] Next, combine Figure 7 , which introduces in detail the specific implementation process of the defect detection device to detect the appearance image of the electronic device.

[0259] See also Figure 7 , Figure 7 FIG. 1 shows a flow chart of a defect detection method. Figure 7 As shown, the defect detection method provided in the embodiment of the present application may include:

[0260] S701. A defect detection apparatus collects an appearance image of a first device.

[0261] According to the above embodiment, the defect detection device can use the trained deep learning model to collect the appearance of the first device. The defect detection device collects the appearance image of the first device through a camera.

[0262] Among them, the type of the first device may include: screen, back panel, screen frame and camera, etc.

[0263] The types of defects that may occur in the first equipment type can be divided into the following types:

[0264] Screen defect types can include scratches, cracks, bright spots, and dark spots.

[0265] Back panel defect types may include scratches, wear, color difference, dents and protrusions.

[0266] Screen border defect types may include border cracks, border unevenness and other defect types.

[0267] Camera defect types may include scratches, stains, and hair.

[0268] S702: The defect detection device performs defect detection on the appearance image of the first device through a deep learning model and outputs a detection result, wherein the detection result can be used to indicate the defect condition of the appearance of the first device.

[0269] The defect detection device inputs the appearance image of the first device into the deep learning model, and the deep learning model can perform defect detection on the appearance image of the electronic device to obtain a detection result.

[0270] Among them, the deep learning model is a deep learning model trained based on the above-mentioned model training device.

[0271] The inspection result may include the defect type of the appearance, or the defect type and the defect location, or the inspection result may also be used to indicate that the appearance of the electronic device is free of defects.

[0272] In summary, the defect detection device first collects the appearance image of the first device, and uses the trained deep learning model to perform defect detection on the appearance image. The defect detection device can output whether the appearance of the first device is defective. When it is detected that the first device has defects, the defect type and defect location can also be obtained.

[0273] In summary, the sample generation method proposed in the embodiment of the present application effectively expands the sample data set without increasing the actual data collection cost, so that the deep learning model can be trained using a more diverse sample data set, improve the problem of sample imbalance, and reduce labor costs and time costs. The model training method proposed in the embodiment of the present application reduces the problem of inaccurate labeling of defect types in images by the deep learning model by determining the loss function related to the sample data set, improves the quality of the synthesized image, reduces the risk of overfitting of the deep learning model, and effectively improves the performance of the model. The defect detection method proposed in the embodiment of the present application can detect whether the appearance of an electronic device is defective, and when there are defects in the appearance of the electronic device, the defect type and defect location.

[0274] Embodiments of the present application also provide an electronic device, a computer-readable storage medium, and a computer program product.

[0275] According to an embodiment of the present application, an electronic device includes: one or more processors; and a memory communicatively connected to at least one processor; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the sample generation method provided in the embodiment of the present application; and / or when the instructions are executed by the electronic device, enable the electronic device to execute the model training method provided in the embodiment of the present application; and / or when the instructions are executed by the electronic device, enable the electronic device to execute the defect detection method provided in the embodiment of the present application.

[0276] According to an embodiment of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a sample generation method; and / or the computer instructions are used to cause a computer to execute a model training method; and / or the computer instructions are used to cause a computer to execute a defect detection method.

[0277] According to an embodiment of the present application, a computer program product includes a computer program, which implements a sample generation method when executed by a processor; and / or the computer program implements a model training method when executed by a processor; and / or the computer program implements a defect detection method when executed by a processor.

[0278] See also Figure 8 , Figure 8 A block diagram of an electronic device suitable for implementing a sample generation method, a model training method, or a defect detection method that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as desktop computers, laptop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0279] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0280] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0281] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a sample generation method, a model training method, or a defect detection method. For example, in some embodiments, the sample generation method, the model training method, or the defect detection method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the sample generation method, model training method, or defect detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the sample generation method, the model training method, or the defect detection method in any other appropriate manner (eg, by means of firmware).

[0282] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0283] The program code for implementing the method of the embodiment of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0284] In the context of the present application embodiment, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0285] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0286] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0287] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0288] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the embodiments of the present application can be achieved, and this document is not limited here.

[0289] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample data set, wherein the sample data set includes a plurality of synthetic images, wherein the synthetic images are obtained by performing a first data enhancement fusion on a foreground image and a background image, wherein the foreground image is an image sample of a device having defects in appearance and defect locations, and the background image is an image sample of a device having no defects in appearance; the sample data set is acquired from a sample generating device; the foreground image is obtained by performing a second data enhancement on the initial foreground image after the sample generating device acquires the initial foreground image; the background image is obtained by performing a third data enhancement on the initial background image after the sample generating device acquires the initial background image; Determine a first loss function and a second loss function, wherein the first loss function is used to represent the similarity loss between different synthetic images corresponding to the same foreground image, and the second loss function is used to represent the difference between the predicted value and the true value; The sample data set is input into a deep learning model, and the deep learning model is trained according to the first loss function and the second loss function to obtain a trained deep learning model, and the deep learning model is used to detect defects in the appearance of a device.

2. The method according to claim 1, characterized in that Training the deep learning model according to the first loss function and the second loss function includes: training the deep learning model according to Formula 1, Formula 2, and Formula 3; L = L norm + L seg Formula 1; Formula 2: Formula 3; in, L is the third loss function, L norm is the first loss function, L seg is the second loss function, L (·) represents a distance loss, represents the transformation of the initial foreground image after data enhancement, and represents the inverse transformation of the same initial foreground image into different composite images. y p and y q represents different synthetic images, where the different synthetic images are obtained after the same set of foreground images and background images are enhanced by the first data, p ( x ) is the true value of the annotation label for each defect type, q ( x ) is the predicted value of the defect in the synthetic image output by the model training device, x is the number of synthesized images.

3. The method according to claim 1 or 2, characterized in that: The first data enhancement includes: transparency fusion, transition boundary blurring, and overall intensity equalization.

4. The method according to claim 1 or 2, characterized in that: The second data enhancement includes at least one of grayscale, equalization, scaling, rotation or contrast adjustment.

5. The method according to claim 1 or 2, characterized in that: The third data enhancement includes at least one of grayscale, equalization, scaling, rotation, noise addition or contrast adjustment.

6. A defect detection method, characterized in that: Applied in a defect detection device, the defect detection device comprises a deep learning model trained and generated by the method according to any one of claims 1 to 5, the method comprising: Capturing an appearance image of the first device; Through the deep learning model, defect detection is performed on the appearance image of the first device, and a detection result is output, where the detection result is used to indicate the defect status of the appearance of the first device.

7. The method according to claim 6, characterized in that The types of the first device include: a screen, a back panel, a screen frame, and a camera.

8. A model training device, characterized in that: The apparatus comprises: a module for executing the method as claimed in any one of claims 1-5.

9. A defect detection device, characterized in that: The device comprises: a module for executing the method as claimed in claim 6 or 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the method according to any one of claims 1 to 5; And / or, causing the electronic device to execute the method as claimed in claim 6 or 7.

11. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instruction is executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 5; and / or the electronic device executes the method as claimed in claim 6 or 7.

Images