Image enhancement method based on defect feature migration fusion

Abstract

The invention relates to an image enhancement method based on defect feature migration fusion. The method comprises the following steps: acquiring a training image with migration defect features of a detection object; the training image is utilized to pre-train a feature extraction model, and the pre-trained feature extraction model comprises a first feature extraction branch and a second feature extraction branch; respectively performing feature extraction on the defect image and the defect-free image of the detected object through two feature extraction branches to obtain a defect feature image and a background feature image; and performing weighted fusion on the defect feature image and the background feature image, and intersecting a fusion result with the second feature extraction branch for iterative training to obtain an enhanced image of the defect area in the detection object. The pre-training of the feature extraction model is based on the training image of the migration defect feature, and the feature image extracted by the model is subjected to multi-scale iterative fusion to generate the enhanced image, so that the generation efficiency and the visual effect of the enhanced image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image enhancement method based on defect feature migration and fusion. Background Art

[0002] In the field of industrial manufacturing, due to the characteristics of industrial products and the interference of the external environment, defects on the product surface are inevitable during the mass production process. In addition, the defects of products are characterized by multiple categories, wide distribution, and complex defect patterns, which brings great challenges to the defect detection of industrial products.

[0003] For example, industrial scratches on chip products have different thicknesses, lengths, and depths. Traditional machine learning is not compatible with such complex industrial scenarios, and the training of deep learning models often relies on tens of thousands of data to have excellent generalization. In addition, in the early stages of chip production and manufacturing, very few defect samples can be collected, and some categories of defect samples are even zero. With only a small amount of data, deep learning models are prone to overfitting and miss detection. In response to the problem of few samples of chip appearance defects, most existing technologies tend to use the existing small number of defect samples for image enhancement to generate more data for training deep learning models.

[0004] In the prior art, there are various image enhancement technologies. Based on the model training perspective, image enhancement can be divided into offline enhancement and online enhancement. Offline enhancement is the image data preprocessing stage that occurs before model training. It is a one-time conversion performed on all data sets. This operation can multiply the number of data samples. Since offline enhancement processes all data sets at once, it is suitable for smaller data sets. During the model training stage of online enhancement, random image enhancement is performed on batch image data after obtaining it, and then the enhanced image data is handed over to the model for training, and the total amount of training data remains unchanged. Due to the batch processing characteristics of image data, offline enhancement is suitable for larger data sets.

[0005] Based on the image transformation angle, image enhancement can be divided into traditional data enhancement methods and data enhancement methods based on deep learning models. Traditional image data enhancement methods usually use image processing technology to complete the expansion of data sets and image quality optimization. However, the capabilities and application scenarios of traditional data enhancement are limited. For example, geometric transformation is a mapping of linear space. Excessive use may cause distortion of target defects or loss of local information. The disadvantages of color transformation conversion are that it consumes a lot of memory space and time. Sometimes excessive enhancement will also produce bad effects. For example, the color information on the chip is a fixed color, such as light yellow, but if a large amount of red, green and other color information is enhanced, it is unreasonable. The data enhancement method based on deep learning models refers to the use of deep learning models for image processing, such as generative adversarial networks (GANs), autoencoders, etc. However, it takes a lot of time to build a model to train the generator and discriminator, and it is complex and costly. The data enhancement method based on deep learning models can generate more complex and high-quality images, and improve the accuracy and generalization ability of the model.

[0006] In addition, the data enhancement method based on the deep learning model is used, but the number of defective samples is small. The existing technology also obtains a large number of similar defective samples from other similar scenes and fuses them with defect-free images, using the fused image as a training sample. Specifically, the target area (defective area) in the source image is seamlessly embedded into the target image to create a new image that is visually natural and has no obvious seams. However, this fusion method has certain limitations. First, the fusion of two images will cause the information of pixels that are far apart in the original image to be mixed together, which may introduce irrelevant pixel information and affect the visual effect of the fused image; secondly, the image fusion calculation in the existing technology involves the solution of functional equations, which is time-consuming and inefficient.

[0007] Therefore, those skilled in the art are in urgent need of a technical solution for generating more sample data to train an industrial defect detection model through an efficient image enhancement method when there are few industrial defect detection samples. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an image enhancement method based on defect feature migration and fusion, which solves the technical problems of low efficiency and poor visual effect of the generated enhanced image in the process of enhancing defect images of industrial products in the prior art.

[0010] (II) Technical solution

[0011] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0012] In a first aspect, an embodiment of the present invention provides an image enhancement method based on defect feature migration and fusion, comprising:

[0013] Acquire a training image with migration defect features of the inspection object;

[0014] Pre-training a selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch;

[0015] Through the first feature extraction branch and the second feature extraction branch, feature extraction is performed on the defect image and the non-defect image of the inspection object respectively to obtain a defect feature image and a background feature image;

[0016] The defect feature image and the background feature image are weightedly fused, and the fusion result is handed over to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the inspection object.

[0017] Optionally, obtaining a training image with a migration defect feature of the detection object includes:

[0018] Acquire a defect image sample set and a non-defect image sample set of the inspection object;

[0019] Based on the inspection object, determining a reference object having a migration defect characteristic of the inspection object;

[0020] A defect image sample set of the reference object is obtained, wherein the difference in the number of images of each defect type in the defect image sample set is kept within a set threshold range.

[0021] Optionally, the selected feature extraction model is pre-trained using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch including:

[0022] Obtain a feature extraction model;

[0023] Pre-training the feature extraction model using a sample set of defect-free images of the inspection object and a sample set of defective images of the reference object as training sets;

[0024] Using the defect image sample set of the inspection object as a test set, verifying the pre-trained feature extraction model;

[0025] Among them, the feature extraction model includes a first feature extraction branch and a second feature extraction branch. The pre-trained network parameters of the first feature extraction branch are consistent with those of the second feature extraction branch, and the parameters of the first feature extraction branch do not participate in back-propagation iterative training.

[0026] Optionally, performing feature extraction on the defect image and the non-defect image of the inspection object respectively through the first feature extraction branch and the second feature extraction branch to obtain the defect feature image and the background feature image includes:

[0027] Acquire a defect image and a non-defect image of the inspection object;

[0028] Marking the defective area in the defective image, and intercepting an area with the same size as the defective area from the non-defective image for marking;

[0029] Through the first feature extraction branch, defect feature extraction including convolution, batch normalization and activation function is performed on the marked defect image to obtain a defect feature image;

[0030] Through the second feature extraction branch, background feature extraction including convolution, batch normalization and activation function is performed on the marked defect-free image to obtain a background feature image.

[0031] Optionally, weighted fusion of the defect feature image and the background feature image is performed, and the fusion result is handed over to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object, which includes:

[0032] Using a 1×1 convolution kernel, weighted summing is performed on the defect feature image and the background feature image along the feature channel direction respectively;

[0033] After multi-dimensional addition fusion of the defect features and background features obtained by weighted summation, batch normalization and activation function are performed on the fused features to obtain an initial enhanced image;

[0034] The background feature image is handed over to the second feature extraction branch for iterative training of back propagation, and the training result is multi-scale fused with the defect feature image to obtain an enhanced image of the defect area in the detection object.

[0035] Optionally, the calculation formula of the convolution kernel is:

[0036]

[0037] (1) In the formula, Represents the weighted sum of defect feature images, represents the weighted sum of background feature images, represents a 1×1 convolution kernel, Represents the defect feature image, represents the background feature image, i represents the image sequence number;

[0038] The calculation formula for batch normalization is:

[0039]

[0040] (2) In the formula, Represents normalized image, BN stands for batch normalization;

[0041] The calculation formula of the activation function is:

[0042]

[0043] (3) In the formula, represents the initial enhanced image, and Relu represents the activation function.

[0044] Optionally, the image enhancement method is applied to perform image enhancement processing on defective images of optical communication chips, and the defective features of laser chips are used as migration features, so that the defective sample images of laser chips are used as training images to pre-train the feature extraction model.

[0045] In a second aspect, an embodiment of the present invention provides an industrial defect detection device, including:

[0046] An image acquisition module, used to acquire defect images and non-defect images of the inspection object;

[0047] An image enhancement module, used to perform image enhancement processing on the defect image by using the above-mentioned image enhancement method based on defect feature migration and fusion to obtain an enhanced image;

[0048] A model training module, used for training a selected industrial defect detection model according to the defect-free image and the enhanced image;

[0049] The defect detection module is used to perform industrial defect detection on the detection object according to the trained industrial defect detection model.

[0050] Optionally, the image enhancement module includes:

[0051] A training image acquisition submodule is used to acquire a training image with migration defect features of the inspection object;

[0052] A model pre-training submodule, used to pre-train a selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch;

[0053] A feature extraction submodule, configured to extract features from the defect image and the non-defect image of the inspection object respectively through the first feature extraction branch and the second feature extraction branch, so as to obtain a defect feature image and a background feature image;

[0054] The image enhancement submodule is used to perform weighted fusion on the defect feature image and the background feature image, and to hand over the fusion result to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object.

[0055] In a third aspect, an embodiment of the present invention provides an industrial defect detection system, including:

[0056] A camera for capturing images of the detected object;

[0057] The above-mentioned industrial defect detection equipment connected to the camera is used to perform defect detection on the detection object in the image after receiving the image.

[0058] (III) Beneficial effects

[0059] The beneficial effects of the present invention are as follows: an image enhancement method based on defect feature migration and fusion of the present invention adopts a training image with migrated defect features to pre-train a selected feature extraction model, and then extracts feature images of a defect image and a defect-free image of a detection object respectively through two feature extraction branches of the pre-trained model, and finally performs multi-scale iterative weighted fusion on the feature images to generate a technical solution for training an enhanced image of an industrial defect detection model. Compared with the prior art, the pre-training of the feature extraction model in the present invention generates a new defect image by migrating the defect features in the training image to the defect-free image, which reduces the introduction of irrelevant pixel information and does not have a complicated solution process, thereby improving the efficiency of defect feature migration.

[0060] At the same time, the feature image weighted fusion technology proposed in the present invention is carried out on multiple feature maps simultaneously, which is an iterative training of multi-scale fusion enhancement. The fused defect features and background features are obtained through autonomous learning of the feature extraction model. Taking into account the large difference in background between the training image and the detection image, the fused enhanced image has better visual authenticity, which is conducive to enhancing the generalization ability of the industrial defect detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a flow chart of an image enhancement method based on defect feature migration and fusion proposed in one embodiment of the present invention;

[0062] Figure 2 A calculation diagram for defect feature migration and fusion proposed in one embodiment of the present invention;

[0063] Figure 3 A position map for multi-scale fusion of defect features proposed in an embodiment of the present invention;

[0064] Figure 4This is a statistical diagram of different types of defect data in a defect image sample set proposed by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0066] refer to Figure 1-3 As shown, an image enhancement method based on defect feature migration and fusion proposed in an embodiment of the present invention includes: first, obtaining a training image with migrated defect features of a detection object; second, using the training image, pre-training a selected feature extraction model, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch; then, through the first feature extraction branch and the second feature extraction branch, feature extraction is performed on the defect image and the defect-free image of the detection object respectively to obtain a defect feature image and a background feature image; finally, the defect feature image and the background feature image are weightedly fused, and the fusion result is handed over to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object.

[0067] In this embodiment, a selected feature extraction model is pre-trained using a training image with transferred defect features, and then feature images of a defect image and a defect-free image of the inspection object are extracted respectively through two feature extraction branches of the pre-trained model. Finally, the feature images are iteratively weighted fused at multiple scales to generate an enhanced image for training an industrial defect detection model. Compared with the prior art, the pre-training of the feature extraction model in this embodiment generates a new defect image by transferring the defect features in the training image to the defect-free image, which reduces the introduction of irrelevant pixel information and does not have a complicated solution process, thereby improving the efficiency of defect feature migration.

[0068] At the same time, the feature image weighted fusion technology proposed in this embodiment is performed on multiple feature maps simultaneously, which is an iterative training of multi-scale fusion enhancement. The fused defect features and background features are obtained through autonomous learning of the feature extraction model. The large difference in the background between the training image and the detection image is taken into account, so that the fused enhanced image has better visual authenticity, which is conducive to enhancing the generalization ability of the industrial defect detection model.

[0069] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0070] Specifically, an image enhancement method based on defect feature migration and fusion is proposed in one embodiment of the present invention, which includes:

[0071] S1. Obtain a training image with migration defect features of the detection object.

[0072] In this embodiment, step S1 may include sub-steps S11-S13:

[0073] S11, obtaining a defect image sample set and a non-defect image sample set of the inspection object.

[0074] S12. Based on the inspection object, determine a reference object having a migration defect feature of the inspection object.

[0075] S13, obtaining a defect image sample set of a reference object, wherein the difference in the number of images of each defect type in the defect image sample set is kept within a set threshold range.

[0076] In a specific embodiment, when the inspection object is an optical communication chip, and the image data of the optical communication chip containing industrial defects such as scratches, discoloration, dirt, damage and cracks is insufficient, a laser chip with similar industrial defect characteristics as the optical communication chip is selected as a reference object, and the defect characteristics of the laser chip are transferred to the defect-free image of the optical communication chip to form a new defect image as the defect image sample set of the optical communication chip. Among them, the difference in the number of images of each defect type in the defect image sample set is finally obtained to be within the set threshold range, for example, Figure 4 The defect type data distribution diagram shown maintains a certain balance in the number of defect images of different types, and thus when the defect image sample set is submitted to the feature extraction model for training, the accuracy of the model training can be further improved.

[0077] S2. Pre-train the selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch.

[0078] In this embodiment, step S2 may include sub-steps S21-S23:

[0079] S21. Obtain a feature extraction model.

[0080] For example, when the detection object is an optical communication chip, a relatively simple SSD (Single Shot MultiBoxDetector) model can be used as a feature extraction model. Compared with other single-stage detectors such as YOLO, the SSD model has improved speed and accuracy. At the same time, the SSD model structure is relatively lightweight, making it easy to deploy in practical applications.

[0081] S22, pre-training a feature extraction model using a sample set of defect-free images of the inspection object and a sample set of defective images of the reference object as training sets.

[0082] S23. Using the defect image sample set of the inspection object as a test set, the pre-trained feature extraction model is verified.

[0083] It is worth mentioning that the feature extraction model includes a first feature extraction branch and a second feature extraction branch. The pre-trained network parameters of the first feature extraction branch are consistent with those of the second feature extraction branch, and the parameters of the first feature extraction branch do not participate in back-propagation iterative training.

[0084] In a specific embodiment, when the detection object is an optical communication chip and the reference object is a laser chip, first, by using a sufficient number of images of laser chips with defect labels for training a sufficient number of times, the feature extraction model is ensured to converge and fit the training set well. Then, the feature extraction model is pre-trained using the non-defective image sample set of the optical communication chip and the defective image sample set of the laser chip as the training set, so that the defective features of the laser chip are transferred to the non-defective image of the optical communication chip to generate a new defective image. Finally, the pre-trained feature extraction model is verified using the defective image sample set of the optical communication chip as the test set, that is, the defective images generated by the model pre-training are compared and verified.

[0085] S3. Perform feature extraction on the defect image and the non-defect image of the inspection object respectively through the first feature extraction branch and the second feature extraction branch to obtain a defect feature image and a background feature image.

[0086] In this embodiment, step S3 may include sub-steps S31-S34:

[0087] S31, obtaining a defect image and a non-defect image of the inspection object.

[0088] For example, when the inspection object is an optical communication chip, it is difficult to use a conventional camera to capture industrial defects on the optical communication chip due to its small size, and it is easy to miss the inspection. Therefore, one of the high-definition camera, electron microscope, optical microscope and intelligent industrial camera can be used to capture defective and non-defective images of the optical communication chip. The intelligent industrial camera integrates high-resolution image sensors, image processing algorithms and communication technologies.

[0089] S32, marking the defective area in the defective image, and cutting out an area with the same size as the defective area from the non-defective image for marking.

[0090] S33. Through the first feature extraction branch, defect feature extraction including convolution, batch normalization and activation function is performed on the marked defect image to obtain a defect feature image.

[0091] S34. Through the second feature extraction branch, background feature extraction including convolution, batch normalization and activation function is performed on the marked defect-free image to obtain a background feature image.

[0092] In a specific embodiment, the feature extraction branch is a convolutional neural network (CNN), and the features extracted by the convolutional neural network are translation invariant. Specifically, after the image is translated, the expression on the corresponding feature map is also translated. For example, in a defect image, there is a defect at the position (width W, height H), and the center of the defect is located at the position (W / 2, H / 2). After a series of convolution (Conv), batch normalization (BN), and activation function (Relu) operations, the position (W / 2, H / 2) on the obtained feature map is also the position where the defect feature response is the largest. Since the features extracted by the convolutional neural network are translation invariant, image fusion can be performed directly on the new feature space.

[0093] S4. Perform weighted fusion on the defect feature image and the background feature image, and hand over the fusion result to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object.

[0094] In this embodiment, step S4 may include sub-steps S41-S43:

[0095] S41. Use a 1×1 convolution kernel to perform weighted summation on the defect feature image and the background feature image along the feature channel direction.

[0096] Among them, the calculation formula of the convolution kernel is:

[0097]

[0098] (1) In the formula, Represents the weighted sum of defect feature images, represents the weighted sum of background feature images, represents a 1×1 convolution kernel, Represents the defect feature image, represents the background feature image, and i represents the image sequence number.

[0099] S42. After multi-dimensional addition fusion of the defect features and background features obtained by weighted summation, batch normalization and activation function are performed on the fused features to obtain an initial enhanced image.

[0100] Among them, the calculation formula for batch normalization is:

[0101]

[0102] (2) In the formula, Represents normalized image, BN stands for batch normalization;

[0103] The calculation formula of the activation function is:

[0104]

[0105] (3) In the formula, represents the initial enhanced image, and Relu represents the activation function.

[0106] S43, handing the background feature image to the second feature extraction branch for iterative training of back propagation, and performing multi-scale fusion of the training result and the defect feature image to obtain an enhanced image of the defect area in the detection object.

[0107] In a specific embodiment, the SSD model is selected to perform image enhancement on the defective image of the optical communication chip to generate more training data for training the industrial defect detection model. First, the feature extraction of the defective image and the defect-free image of the optical communication chip is completed respectively through two feature extraction branches to map the features on the low-dimensional image to the high-dimensional space according to the pixel position. Then, the defect feature image and the background feature image are weighted fused to obtain the initial enhanced image. Finally, the background feature image is replaced with the initial enhanced image, and the defect feature image and the initial enhanced image are iteratively trained by back propagation until convergence to generate a multi-scale fused enhanced image. Through the mutual complementation of shallow features and deep semantic features, the generated enhanced image can avoid the information loss problem caused by the size change during the defect extraction process.

[0108] In addition, the present invention also proposes an industrial defect detection device, which includes:

[0109] The image acquisition module is used to acquire defect images and non-defect images of the inspection object.

[0110] The image enhancement module is used to adopt the above-mentioned image enhancement method based on defect feature migration and fusion to perform image enhancement processing on the defect image to obtain an enhanced image.

[0111] The model training module is used to train the selected industrial defect detection model based on the defect-free images and the enhanced images.

[0112] The defect detection module is used to perform industrial defect detection on the detection object based on the trained industrial defect detection model.

[0113] Furthermore, the image enhancement module includes:

[0114] A training image acquisition submodule is used to acquire a training image with migration defect features of the inspection object;

[0115] A model pre-training submodule, used to pre-train the selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch;

[0116] A feature extraction submodule, used to extract features from a defect image and a non-defect image of the inspection object through a first feature extraction branch and a second feature extraction branch, respectively, to obtain a defect feature image and a background feature image;

[0117] The image enhancement submodule is used to perform weighted fusion on the defect feature image and the background feature image, and to hand over the fusion result to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object.

[0118] Furthermore, the present invention further provides an industrial defect detection system, comprising:

[0119] A camera for capturing images of the detected object.

[0120] The above-mentioned industrial defect detection device connected to the camera is used to perform defect detection on the detection object in the image after receiving the image.

[0121] For example, the camera can be an industrial camera that integrates high-resolution image sensors, image processing algorithms and communication technologies. The industrial camera can be set up on a chip production line to capture chip images at various production stages, and send the captured chip images to industrial defect detection equipment for chip defect detection, so that staff can remove defective chips based on the detection results.

[0122] In summary, the present invention proposes an image enhancement method based on defect feature migration and fusion. First, the present invention provides a framework for industrial defect image enhancement, which utilizes similar defect features in other scenarios (such as laser chips) and feature extraction models with excellent generalization performance, and migrates similar defect features to defect-free images in the current scenario (such as optical communication chips). The method of training a network with a feature extraction model to guide a new network achieves the effect of data enhancement. Secondly, the present invention proposes a method for high-dimensional space defect feature fusion, which maps low-dimensional features to high-dimensional space by using a convolutional neural network of a deep learning model, so that the defect feature fusion operation is transferred to the high-dimensional space. Finally, the feature image weighted fusion technology proposed by the present invention is carried out on multiple feature maps simultaneously, which is an iterative training of multi-scale fusion enhancement. The fused defect features and background features are obtained through autonomous learning of the feature extraction model. Taking into account the large background difference between the training image and the detection image, the fused enhanced image has better visual authenticity, which is conducive to enhancing the generalization ability of the industrial defect detection model.

[0123] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0124] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0126] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0127] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. An image enhancement method based on defect feature migration and fusion, characterized in that: include: Acquire a training image with migration defect features of the inspection object; Pre-training a selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch; Through the first feature extraction branch and the second feature extraction branch, feature extraction is performed on the defect image and the non-defect image of the inspection object respectively to obtain a defect feature image and a background feature image; The defect feature image and the background feature image are weightedly fused, and the fusion result is handed over to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the inspection object.

2. The image enhancement method based on defect feature migration and fusion as claimed in claim 1, characterized in that: Obtaining training images with migration defect features of detection objects includes: Acquire a defect image sample set and a non-defect image sample set of the inspection object; Based on the inspection object, determining a reference object having a migration defect characteristic of the inspection object; A defect image sample set of the reference object is obtained, wherein the difference in the number of images of each defect type in the defect image sample set is kept within a set threshold range.

3. The image enhancement method based on defect feature migration and fusion as claimed in claim 2, characterized in that: The selected feature extraction model is pre-trained using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch including: Obtain feature extraction model; Pre-training the feature extraction model using a sample set of defect-free images of the inspection object and a sample set of defective images of the reference object as training sets; Using the defect image sample set of the inspection object as a test set, verifying the pre-trained feature extraction model; Among them, the feature extraction model includes a first feature extraction branch and a second feature extraction branch. The pre-trained network parameters of the first feature extraction branch are consistent with those of the second feature extraction branch, and the parameters of the first feature extraction branch do not participate in back-propagation iterative training.

4. The image enhancement method based on defect feature migration and fusion as claimed in claim 1, characterized in that: By using the first feature extraction branch and the second feature extraction branch, feature extraction is performed on the defect image and the non-defect image of the inspection object respectively, and obtaining the defect feature image and the background feature image comprises: Acquire a defect image and a non-defect image of the inspection object; Marking the defective area in the defective image, and intercepting an area with the same size as the defective area from the non-defective image for marking; Through the first feature extraction branch, defect feature extraction including convolution, batch normalization and activation function is performed on the marked defect image to obtain a defect feature image; Through the second feature extraction branch, background feature extraction including convolution, batch normalization and activation function is performed on the marked defect-free image to obtain a background feature image.

5. The image enhancement method based on defect feature migration and fusion as claimed in claim 1, characterized in that: The defect feature image and the background feature image are weightedly fused, and the fusion result is handed over to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the inspection object, which includes: Using a 1×1 convolution kernel, weighted summing is performed on the defect feature image and the background feature image along the feature channel direction respectively; After multi-dimensional addition fusion of the defect features and background features obtained by weighted summation, batch normalization and activation function are performed on the fused features to obtain an initial enhanced image; The background feature image is handed over to the second feature extraction branch for iterative training of back propagation, and the training result is multi-scale fused with the defect feature image to obtain an enhanced image of the defect area in the detection object.

6. The image enhancement method based on defect feature migration and fusion as claimed in claim 5, characterized in that: The calculation formula of the convolution kernel is: (1) In the formula, Represents the weighted sum of defect feature images, represents the weighted sum of background feature images, represents a 1×1 convolution kernel, Represents the defect feature image, represents the background feature image, i represents the image sequence number; The calculation formula for batch normalization is: (2) In the formula, Represents normalized image, BN stands for batch normalization; The calculation formula of the activation function is: (3) In the formula, represents the initial enhanced image, and Relu represents the activation function.

7. An image enhancement method based on defect feature migration and fusion as claimed in any one of claims 1 to 6, characterized in that: The image enhancement method is applied to perform image enhancement processing on defective images of optical communication chips, and uses defective features of laser chips as migration features, so that defective sample images of laser chips are used as training images to pre-train the feature extraction model.

8. An industrial defect detection device, characterized in that: include: An image acquisition module, used to acquire defect images and non-defect images of the inspection object; An image enhancement module, configured to perform image enhancement processing on the defect image by using an image enhancement method based on defect feature migration and fusion as described in any one of claims 1 to 7 to obtain an enhanced image; A model training module, used for training a selected industrial defect detection model according to the defect-free image and the enhanced image; The defect detection module is used to perform industrial defect detection on the detection object according to the trained industrial defect detection model.

9. An industrial defect detection system as claimed in claim 8, characterized in that: The image enhancement module comprises: A training image acquisition submodule is used to acquire a training image with migration defect features of the inspection object; A model pre-training submodule, used to pre-train a selected feature extraction model using the training image, wherein the pre-trained feature extraction model includes a first feature extraction branch and a second feature extraction branch; A feature extraction submodule, configured to extract features from the defect image and the non-defect image of the inspection object respectively through the first feature extraction branch and the second feature extraction branch, so as to obtain a defect feature image and a background feature image; The image enhancement submodule is used to perform weighted fusion on the defect feature image and the background feature image, and to hand over the fusion result to the second feature extraction branch for iterative training to generate an enhanced image of the defect area in the detection object.

10. An industrial defect detection system, characterized in that: include: A camera for capturing images of the detected object; The industrial defect detection device as described in any one of claims 8 or 9 connected to the camera is used to perform defect detection on the detection object in the image after receiving the image.

Images