Model training method and device, industrial image defect detection method and device and storage medium
By using pre-trained models in industrial image defect detection combined with text encoder and image encoder, the problem of high efficiency and low cost of industrial product quality detection is solved, and the efficient identification and generalization of industrial product defects is achieved. In a variety of industrial fields.
Patent Information
- Application Number
- CN202411062979.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The quality inspection of existing industrial products relies on manual screening, which is costly and inefficient, and cannot meet the needs of large-scale quality inspections.
A model training method is adopted, using a pre-trained model combined with text encoder and image encoder, through feature extraction and abnormal segmentation mask generation, industrial image defect detection is carried out to avoid domain mismatch and improve generalization capabilities.
It realizes efficient identification of defects in industrial products, reduces inspection costs, improves inspection efficiency, and is suitable for defect detection in various industrial fields.
Smart Images

Figure CN119068305A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a model training method, an industrial image defect detection method, a device and a storage medium. Background Art
[0002] Whether it is an airplane wing or a chip battery, industrial products of all sizes are widely used in modern society. While the ubiquitous industrial products bring convenience to life, they also expose some problems, such as poor industrial quality and many defects. Therefore, quality inspection of industrial products is extremely important. However, the previous defect detection relies on manual screening, which is costly and inefficient, and cannot meet large-scale quality inspection needs. Summary of the invention
[0003] In view of the current technical problems of high cost and low efficiency in quality inspection of industrial products, the purpose of the present invention is to provide a model training method, an industrial image defect detection method, device and storage medium.
[0004] In one aspect, an embodiment of the present invention includes a model training method, wherein the model training method and / or the industrial image defect detection method includes the following steps:
[0005] Obtain a pre-trained model; the pre-trained model includes a text encoder, an image encoder and an adapter;
[0006] Acquire image training samples and corresponding text prompt training samples; the image training samples include images of industrial products, and the text prompt training samples are used to indicate the types and abnormal states of the industrial products in the image training samples;
[0007] The image training samples and the text prompt training samples are input into the pre-training model for processing; wherein the text encoder performs feature extraction on the text prompt training samples to obtain a first text feature, the image encoder and the adapter perform feature extraction on the image training samples to obtain a first image feature, and an abnormal segmentation mask map is generated according to the first text feature and the first image feature;
[0008] The pre-training model is trained according to the abnormal segmentation mask image and the text prompt training samples.
[0009] Further, the image encoder includes a plurality of image encoding units;
[0010] The adapter includes a plurality of adaptation units; each of the adaptation units corresponds to one of the image encoding units.
[0011] Furthermore, the adaptation unit is a linear mapping layer.
[0012] Furthermore, the image encoder and the adapter extract features from the image training sample to obtain a first image feature, including:
[0013] Dividing into multiple stages; there is a sequential relationship between the stages, and each stage corresponds to one of the image encoding units and one of the adaptation units;
[0014] For any of the stages, the image encoding unit of the current stage performs feature extraction on the feature information to be processed to obtain the first feature information of the current stage, the adaptation unit of the current stage adjusts the first feature information to obtain the second feature information of the current stage, and the first feature information and the second feature information of the current stage are combined to obtain the third feature information of the current stage; wherein, when the current stage is the first stage, the feature information to be processed is the image training sample, and when the current stage is the other stages, the feature information to be processed is the third feature information obtained in the previous stage;
[0015] The second feature information obtained in each stage is used as the first image feature.
[0016] Furthermore, generating an abnormal segmentation mask map according to the first text feature and the first image feature includes:
[0017] Perform matrix cross product on the first text feature and each of the second feature information to obtain multiple similarity matrices;
[0018] Performing element-by-element averaging on each of the similarity matrices to obtain an average value matrix;
[0019] Performing bilinear interpolation on the average value matrix to obtain an interpolation matrix;
[0020] Inputting the interpolation matrix into a segmentation head for processing;
[0021] The abnormal segmentation mask image output by the segmentation head is obtained.
[0022] Furthermore, the pre-training model is trained according to the abnormal segmentation mask image and the text prompt training sample, including:
[0023] Freezing network parameters of the text encoder and the image encoder;
[0024] Calculating a loss function based on the abnormal segmentation mask image and the text prompt training sample to obtain a gradient parameter;
[0025] According to the gradient parameters, the network parameters of the adapter are updated.
[0026] Furthermore, the loss function is calculated based on the abnormal segmentation mask image and the text prompt training sample to obtain the gradient parameter, including:
[0027] Performing Dice function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a first parameter value;
[0028] Performing Focal function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a second parameter value;
[0029] Performing BCE function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a third parameter value;
[0030] The first parameter value, the second parameter value and the third parameter value are summed to obtain the gradient parameter.
[0031] On the other hand, an embodiment of the present invention includes an industrial image defect detection method, the industrial image defect detection method comprising the following steps:
[0032] Obtain a pre-trained model; the pre-trained model is a pre-trained model obtained by executing the model training method according to any one of claims 1 to 7;
[0033] Acquire the industrial image to be identified;
[0034] Inputting the industrial image to be identified into the pre-trained model for processing;
[0035] Obtain the industrial image defect detection result output by the pre-trained model.
[0036] On the other hand, an embodiment of the present invention also includes a computer device, including a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the model training method and / or industrial image defect detection method in the embodiment.
[0037] On the other hand, an embodiment of the present invention also includes a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to execute the model training method and / or industrial image defect detection method in the embodiment when executed by the processor.
[0038] The beneficial effects of the present invention are as follows: the model training method in the embodiment, by training the pre-trained model, the trained pre-trained model has the performance of identifying the industrial image to be identified and judging whether the industrial products in the industrial image to be identified have defects (abnormalities); the pre-trained model is obtained by adding an adapter on the basis of the text encoder and the image encoder, and can transfer the prior knowledge of natural images to the industrial image, so that the pre-trained model as a whole can process the image training samples belonging to the industrial image, and can avoid the problem of domain mismatch. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of the steps of the model training method in the embodiment;
[0040] Figure 2 It is a schematic diagram of the principle of the model training method in the embodiment;
[0041] Figure 3 Schematic diagram of the steps of the industrial image defect detection method in the embodiment. DETAILED DESCRIPTION
[0042] With the rapid development of artificial intelligence and computer vision technology, industrial defect detection tasks based on target detection and semantic segmentation can be completed. It aims to discover the appearance defects of industrial products through the processing of single-frame industrial images, classify defects and normal samples, and simply describe and locate defect information. This is one of the important means to ensure the quality of industrial products and maintain market security. Industrial image defect detection based on zero-sample learning is more adaptable to industrial scenarios and can well adapt to various defect detections on various industrial products.
[0043] In the fields of unmanned quality inspection, production control, and abnormal inspection, industrial defect detection has become one of the important basic research and technologies. However, the current industrial defect detection has high requirements for data sets. In actual industrial production, various types of defects on various parts emerge in an endless stream, and it is difficult to collect all types of defects for training. Therefore, some methods only use normal samples for training. Such models can capture some abnormal information, but because the distribution of abnormal samples is not learned during the training process, the generalization ability of the model is relatively weak. Therefore, it is very important to train a model that can identify unknown defect types. In the past two years, large visual language models have gradually appeared in people's field of vision. Their strong generalization ability has attracted the attention of many scholars, and the idea and practice of migrating large models to defect detection have continued to emerge.
[0044] Anomaly detection can be divided into two main tasks, visual anomaly classification (AC) and segmentation (AS), which classify and locate defects in industrial manufacturing, predicting images or pixels as normal or abnormal, respectively. In a wide range of industrial fields, objects and their defects (including aerospace, automotive, pharmaceuticals, and electronics) vary greatly in color, texture, and size. These lead to two major challenges in this field. First, some categories of defects are rare, resulting in a lack of representative anomaly samples. Existing works mainly focus on single-class or unsupervised anomaly detection, which only requires normal images. These methods usually fit the model to normal images and regard any deviation as anomaly. When hundreds or thousands of normal images are available, many methods achieve high-precision anomaly detection on public datasets. However, in the few-normal-shot regime, there is still much room for improving performance compared to the upper limit of full supervision. Second, previous works focus on training a customized model for each task, which has poor generalization ability, which has stimulated the interest in zero-shot anomaly classification and segmentation, hoping to train a model that can perform well in various industrial fields.
[0045] In the industrial field, it is impossible to obtain a large number of defect images in advance, so the supervised learning mechanism is not very realistic. Unsupervised learning only needs normal samples to construct features and builds specific models for specific data sets, which can achieve better results, but the generalization ability is weak. Self-supervised learning enriches the data set but also brings a lot of false information, which is not the best choice.
[0046] To address the above issues, we can design a large anomaly detection model based on zero-shot learning with strong robustness and generalization performance. We can use as little known data as possible to adjust the features on the large visual language model so that it can easily face various types of objects and defect categories that have not appeared before during testing, and obtain a model with strong generalization performance.
[0047] Based on the above principles, in this embodiment, the powerful visual and text understanding capabilities of the large visual language model are borrowed, and adapters adjusted to the industrial field are added on its basis. A small number of known industrial field images are used to train each adapter, that is, the CLIP visual encoder is adjusted to understand industrial images, thereby realizing a model training method. The model trained using this model training method can realize a zero-sample industrial image defect detection method with strong robustness and generalization performance.
[0048] In this embodiment, refer to Figure 1 , the model training method includes the following steps:
[0049] P1. Get the pre-trained model;
[0050] P2. Obtain image training samples and corresponding text prompt training samples;
[0051] P3. Input the image training samples and text prompt training samples into the pre-training model for processing;
[0052] P4. Train the pre-trained model based on the abnormal segmentation mask map and text prompt training samples.
[0053] In step P1, the pre-trained model can be improved on the basis of CLIP. CLIP is a multimodal pre-trained neural network, which is a method of learning from natural language supervision. The core idea is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts.
[0054] The structure of CLIP includes a text encoder and an image encoder. In this embodiment, an adapter is added on the basis of CLIP to obtain a pre-trained model.
[0055] In step P2, image training samples and corresponding text prompt training samples are obtained. Specifically, image training samples can be crawled in batches from the Internet, for example, 400 million image training samples can be crawled, and each image training sample has a corresponding text prompt training sample. In this embodiment, one of the image training samples and the corresponding text prompt training sample, and the process of using them to train the pre-trained model are described.
[0056] In this embodiment, the image training sample includes an image of an industrial product, for example, the image training sample may be an image of a finished industrial component, or an image of an industrial semi-finished product, etc. The text prompt training sample is used to indicate the type and abnormal state of the industrial product in the corresponding image training sample in text form.
[0057] In this embodiment, the format of the text prompt training sample can be generated using a template of a unified format such as "'a photo of the'+[damaged / flawless]+[object]", where "a photo of the" indicates that the image training sample corresponding to this text prompt training sample is an image, "[object]" indicates what type of industrial product the image training sample is (for example, an image of an automobile part, a medicine, a circuit board, etc.), and "[damaged / flawless]" indicates whether the industrial product in the image training sample has defects (depending on the type of industrial product, it may be surface damage, color difference, deformation, etc.). Specifically, when the value is "[flawless]", it indicates that the industrial product in the image training sample has no defects, and when the value is "[damaged]", it indicates that the industrial product in the image training sample has defects (the specific type of the defect may not be described). Therefore, the text prompt training sample in this embodiment can only indicate whether the industrial product in the image training sample has defects (abnormalities), that is, the description of the image training sample is coarse-grained.
[0058] In step P3, the image training sample and text prompt training sample obtained in step P2 are input into the pre-training model for processing. In this embodiment, the process of the pre-training model processing one of the image training samples and its corresponding text prompt training sample is used as an example for description.
[0059] In step P3, after the image training samples and the text prompt training samples are input into the pre-training model, the text encoder in the pre-training model performs feature extraction on the text prompt training samples to obtain the first text feature; the image encoder and the adapter in the pre-training model perform feature extraction on the image training samples to obtain the first image feature. Specifically, the image encoder may perform feature extraction on the image training samples, and the adapter may perform linear mapping on the features extracted by the image encoder to obtain the first image feature. Finally, the first text feature output by the text encoder is fused with the first image feature output by the adapter to generate an abnormal segmentation mask map, which is the result obtained by the pre-training model processing the image training samples and the text prompt training samples.
[0060] The abnormal segmentation mask image can represent the recognition result of the pre-trained model on whether the industrial products in the image training sample have defects (abnormalities) in the form of images, while the text prompt training sample represents whether the industrial products in the image training sample have defects (abnormalities) in the form of text. Therefore, in step P4, the network parameters of the pre-trained model can be updated according to the text prompt training sample, so as to train the pre-trained model.
[0061] In this embodiment, by training the pre-trained model, the pre-trained model has the performance of identifying the industrial image to be identified and judging whether the industrial product in the industrial image to be identified has defects (abnormalities); the pre-trained model is obtained by adding an adapter on the basis of the text encoder and the image encoder, so that even if the text encoder and the image encoder are pre-trained using natural image data sets, the adapter can play the role of a learnable linear container (Linear Vector) and can transfer the prior knowledge of natural images to industrial images, so that the pre-trained model as a whole can process image training samples belonging to industrial images and avoid domain mismatch problems.
[0062] In this embodiment, refer to Figure 2 , the image encoder in the pre-trained model can be ViT-L / 14, which includes 24 encoding layers (encoder), and the 24 encoding layers can be divided into multiple parts, each of which is used as an image encoding unit. For example, the 24 encoding layers are divided into 4 parts, each of which includes 6 encoding layers, and each of the 6 encoding layers in each part is used as an image encoding unit.
[0063] In this embodiment, refer to Figure 2 The adapter includes a plurality of adapting units, and the number of the adapting units is the same as the number of the image encoding units. For example, in this embodiment, the image encoder is provided with 4 image encoding units, and then 4 adapting units can be provided, so that a one-to-one correspondence can be formed between the image encoding units and the adapting units, that is, each image encoding unit has a corresponding adapting unit.
[0064] In this embodiment, a linear mapping layer may be used as an adaptation unit, that is, each adaptation unit is a linear mapping layer.
[0065] In this embodiment, when multiple image encoding units are set in the image encoder and multiple adaptation units (linear mapping layers) are set in the adapter, the pre-trained model performs the following steps when executing step P3, that is, the image encoder and the adapter extract features from the image training sample to obtain the first image feature:
[0066] P301. Divide into multiple stages;
[0067] P302. For any stage, the image encoding unit of this stage performs feature extraction on the feature information to be processed to obtain the first feature information of this stage, the adaptation unit of this stage adjusts the first feature information to obtain the second feature information of this stage, and the first feature information of this stage is combined with the second feature information to obtain the third feature information of this stage; wherein, when this stage is the first stage, the feature information to be processed is the image training sample, and when this stage is other stages, the feature information to be processed is the third feature information obtained in the previous stage;
[0068] P303. Use the second feature information obtained in each stage as the first image feature.
[0069] The principle of steps P301-P303 is as follows Figure 2 shown.
[0070] In step P301, the image encoder (ViT-L / 14) in the pre-trained model is divided into four image coding units, corresponding to the adaptation units (linear mapping layers) such as LP1, LP2, LP3 and LP4 respectively. Among them, starting from the left, the first image coding unit and its corresponding adaptation unit LP1 are classified as stage 1, the second image coding unit and its corresponding adaptation unit LP2 are classified as stage 2, the third image coding unit and its corresponding adaptation unit LP3 are classified as stage 3, and the fourth image coding unit and its corresponding adaptation unit LP4 are classified as stage 4. These stages have a sequential relationship of stage 1, stage 2, stage 3, and stage 4.
[0071] In step P302, when the image encoder (ViT-L / 14) in the pre-trained model works, it is processed in turn by the image encoding unit and adaptation unit (linear mapping layer) of each stage.
[0072] For example, refer to Figure 2 , first execute stage 1, at this time the current stage is stage 1; at this time, the feature information to be processed in this stage is the image training sample, and the image coding unit in this stage (i.e. Figure 2 The part marked with "stage 1" in the figure) performs feature extraction on the feature information to be processed (i.e., the image training sample) to obtain the first feature information F1 of this stage, adjusts the first feature information F1 with the adaptation unit LP1 of this stage to obtain the second feature information of this stage, combines the first feature information F1 of this stage with the second feature information to obtain the third feature information F*1 of this stage.
[0073] Reference Figure 2 After executing stage 1, stage 2 is executed. At this time, the current stage is stage 2. At this time, the feature information to be processed in this stage is the third feature information obtained in the previous stage, that is, F*1. The image coding unit in this stage (i.e. Figure 2 The part marked with "Stage 2" in the figure) performs feature extraction on the feature information to be processed (i.e., F*1) to obtain the first feature information F2 of this stage, adjusts the first feature information F2 with the adaptation unit LP2 of this stage to obtain the second feature information of this stage, combines the first feature information F2 of this stage with the second feature information to obtain the third feature information F*2 of this stage.
[0074] Reference Figure 2 , and so on, until the adaptation units (linear mapping layers) in all stages output their respective second feature information.
[0075] For the last stage (e.g. Figure 2 In stage 4), since there is no next stage, there is no need to use the third feature information output by the last stage. Therefore, the second feature information output by the adaptation unit (linear mapping layer) may not be consistent with the first feature information of this stage ( Figure 2 F4) in the figure is combined, so that the last stage does not need to obtain the third feature information of this stage.
[0076] In this embodiment, when combining the first feature information and the second feature information of each stage in this stage, the first feature information and the second feature information may be linearly combined.
[0077] After executing step P302 and the adaptation units (LP1, LP2, LP3, LP4) in each stage output their own second feature information, execute step P303, and use the whole of the second feature information obtained in each stage as the first image feature to be output by the image encoder (ViT-L / 14) in the pre-trained model.
[0078] In this embodiment, refer to Figure 2 , the text encoder in the pre-trained model extracts features from the text prompt training samples to obtain the first text feature F text .
[0079] In this embodiment, by dividing the image encoder in the pre-trained model into multiple image coding units and configuring a corresponding adaptation unit (linear mapping layer) for each image coding unit, the intermediate visual features extracted by each image coding unit can be adjusted by a corresponding adaptation unit (linear mapping layer), so as to cope with the huge data set used in training the pre-trained model. When using a huge training data set, it can produce a good domain knowledge transfer effect, so that the pre-trained model can be more suitable for the industrial field.
[0080] In this embodiment, steps P301-P303 make good use of the visual-text alignment function of the pre-trained model, have strong robustness and generalization performance, and are well migrated from natural images to industrial image tasks. In addition, due to the combination of large models and adapters, the pre-trained model can be trained on a fixed data set and also performs very well on other unknown data sets in the industrial field. Therefore, the training of the pre-trained model has strong robustness and generalization performance.
[0081] In this embodiment, on the basis of executing steps P301-P303, when executing step P3, the following steps are also executed:
[0082] P304. Perform matrix cross product on the first text feature and each second feature information to obtain multiple similarity matrices;
[0083] P305. Perform element-by-element averaging on each similarity matrix to obtain an average value matrix;
[0084] P306. Perform bilinear interpolation on the average value matrix to obtain an interpolation matrix;
[0085] P307. Input the interpolation matrix to the segmentation head for processing;
[0086] P308. Get the abnormal segmentation mask map output by the segmentation head.
[0087] Reference Figure 2 In step P304, the first text feature F text The matrix cross product is performed with the second feature information outputted from each stage to obtain multiple similarity matrices. Specifically, the second feature information outputted from any stage is compared with the first text feature F text Perform matrix cross product to obtain a similarity matrix.
[0088] Reference Figure 2 In steps P305-P306, each similarity matrix is averaged element by element to obtain an average matrix, and then bilinear interpolation is performed on the average matrix to obtain an interpolation matrix. By performing bilinear interpolation on the average matrix, the size of the average matrix can be adjusted, and the size of the interpolation matrix obtained is consistent with the size of the image training sample.
[0089] In this embodiment, in step P307, a segmentation header is set. Figure 2 , a symmetric fusion network (SFNet) can be used as the segmentation head. When executing step P307, the interpolation matrix is input into the segmentation head for processing. SFNet can generate an abnormal segmentation mask map based on the interpolation matrix, so that when executing step P308, Figure 2The anomaly segmentation mask shown.
[0090] Reference Figure 2 ,According to the anomaly segmentation mask map, it can be intuitively observed whether there are defects (anomalies) in the image training samples, that is, the anomaly segmentation mask map can indicate whether there are defects (anomalies) in the image training samples in the form of images.
[0091] In this embodiment, when executing step P4, that is, the step of generating an abnormal segmentation mask map according to the first text feature and the first image feature, the following steps may be specifically performed:
[0092] P401. Freeze the network parameters of the text encoder and image encoder;
[0093] P402. Calculate the loss function based on the abnormal segmentation mask image and the text prompt training sample to obtain the gradient parameter;
[0094] P403. Based on the gradient parameters, update the network parameters of the adapter.
[0095] Reference Figure 2 In step P401, the network parameters of the text encoder and the image encoder are frozen so that the parameters of the network parameters of the text encoder and the image encoder will not be updated or changed.
[0096] In step P402, based on the differences and similarities between the conclusions expressed by the abnormal segmentation mask image [indicates whether there are defects (abnormalities) in the image training sample in the form of an image] and the text prompt training sample [indicates whether there are defects (abnormalities) in the image training sample in the form of a text], the loss function is used to calculate and obtain the gradient parameters. In step P403, the gradient parameters are used to back-transmit and update the network parameters of the pre-trained model. Since the network parameters of the text encoder and the image encoder are frozen in step P401, when executing step P403, refer to Figure 2 , only the network parameters of the adaptation unit (linear mapping layer) are back-transmitted for update.
[0097] By freezing the network parameters of the text encoder and image encoder and only back-transmitting the network parameters of the adaptation unit (linear mapping layer), the trained network parameters of the text encoder and image encoder can be used to achieve the domain knowledge transfer effect from natural image processing to industrial image processing.
[0098] Since the anomaly detection task to be performed by the trained pre-trained model is essentially a binary classification at the image level and pixel level, the loss function used when executing step P402 must take these two tasks into account. In this embodiment, when executing step P402, three loss functions of Dice+Focal+BCE can be selected for gradient calculation, where the Dice and BCE loss functions are used to measure the similarity between the segmentation and classification results and the target, and the Focal function is mainly used to solve the problem of unbalanced distribution of positive and negative samples in the industrial field.
[0099] In step P402, for the i-th (i=1, 2, 3, 4) stage, the corresponding loss function calculation formula is shown below:
[0100]
[0101] Where Dice() represents the Dice function or the first parameter value calculated by it, Focal() represents the Focal function or the second parameter value calculated by it, and BCE() represents the BCE function or the third parameter value calculated by it. seg,i is the i-th feature extracted for segmentation, F cls,i is the i-th feature extracted for classification, S is the segmented GT, and C is the classified GT. is the first text feature F text The transpose of .
[0102] In step P402, Loss i (i=1,2,3,4) are all accumulated, and the resulting sum is used as the gradient parameter.
[0103] The above steps P301-P308 and P401-P402 are explained for an image training sample and its corresponding text prompt training sample. After executing P301-P308 and P401-P402, execute step P403. According to the gradient parameters, the network parameters of the adapter can be updated by using the gradient descent method or the like. If the network parameters of the adapter converge, the training of the adapter is completed; if the network parameters of the adapter do not converge, return to step S2 and start. Obtain a new image training sample and its corresponding text prompt training sample, and re-execute steps P301-P308 and P401-P402 until the network parameters of the adapter converge.
[0104] In this embodiment, after the pre-training model is trained, the pre-training model can be used to perform the industrial image defect detection method. Figure 3 , the industrial image defect detection method includes the following steps:
[0105] S1. Obtain a pre-trained model;
[0106] S2. Acquire the industrial image to be identified;
[0107] S3. Input the industrial image to be identified into the pre-trained model for processing;
[0108] S4. Obtain the industrial image defect detection results output by the pre-trained model.
[0109] In step S1, a trained pre-trained model is obtained by executing steps P1-P4.
[0110] In step S2, the industrial image to be identified is an image containing industrial products, and it is necessary to identify whether the industrial products have defects (abnormalities).
[0111] In step S3, after the pre-trained model receives the industrial image to be identified, it performs a processing process similar to step P3. For example, the text encoder in the pre-trained model outputs a second text feature (which can be a fixed value, such as a matrix with all elements being 1); the image encoder and the adapter in the pre-trained model perform feature extraction on the industrial image to be identified to obtain the second image feature. Specifically, the image encoder can perform feature extraction on the industrial image to be identified, and the adapter can perform linear mapping on the features extracted by the image encoder to obtain the second image feature. Finally, the second text feature output by the text encoder is fused with the second image feature output by the adapter. Specifically, after performing operations such as matrix cross product and element-by-element averaging on the second text feature and the second image feature, the obtained feature is input into the segmentation head SFNet to obtain the abnormal segmentation mask map output by the segmentation head SFNet.
[0112] In step S4, according to the abnormal segmentation mask map obtained in step S3, it is possible to intuitively understand whether there are defects (abnormalities) in the industrial products in the industrial image to be identified, making good use of the visual-text alignment function of the pre-trained model, and having strong robustness and generalization performance. By executing steps S1-S4, the reliance on manual screening can be reduced when performing defect detection on industrial products, reducing costs, improving efficiency, and meeting large-scale quality inspection needs.
[0113] A computer program that executes the model training method and / or the industrial image defect detection method in this embodiment can be written and written into a computer device or storage medium. When the computer program is read out and executed, the model training method and / or the industrial image defect detection method in this embodiment is executed, thereby achieving the same technical effect as the model training method and / or the industrial image defect detection method in the embodiment.
[0114] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.
[0115] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.
[0116] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.
[0117] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.
[0118] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.
[0119] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.
[0120] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.
Claims
1. A model training method, characterized in that: The model training method comprises: Obtain a pre-trained model; the pre-trained model includes a text encoder, an image encoder and an adapter; Acquire image training samples and corresponding text prompt training samples; the image training samples include images of industrial products, and the text prompt training samples are used to indicate the types and abnormal states of the industrial products in the image training samples; The image training samples and the text prompt training samples are input into the pre-training model for processing; wherein the text encoder performs feature extraction on the text prompt training samples to obtain a first text feature, the image encoder and the adapter perform feature extraction on the image training samples to obtain a first image feature, and an abnormal segmentation mask map is generated according to the first text feature and the first image feature; The pre-training model is trained according to the abnormal segmentation mask image and the text prompt training samples.
2. The model training method according to claim 1, characterized in that: The image encoder comprises a plurality of image encoding units; The adapter includes a plurality of adaptation units; each of the adaptation units corresponds to one of the image encoding units.
3. The model training method according to claim 2, characterized in that: The adaptation unit is a linear mapping layer.
4. The model training method according to claim 2 or 3, characterized in that: The image encoder and the adapter extract features from the image training sample to obtain a first image feature, including: Dividing into multiple stages; there is a sequential relationship between the stages, and each stage corresponds to one of the image encoding units and one of the adaptation units; For any of the stages, the image encoding unit of the current stage performs feature extraction on the feature information to be processed to obtain the first feature information of the current stage, the adaptation unit of the current stage adjusts the first feature information to obtain the second feature information of the current stage, and the first feature information and the second feature information of the current stage are combined to obtain the third feature information of the current stage; wherein, when the current stage is the first stage, the feature information to be processed is the image training sample, and when the current stage is the other stages, the feature information to be processed is the third feature information obtained in the previous stage; The second feature information obtained in each stage is used as the first image feature.
5. The model training method according to claim 4, characterized in that: The generating an abnormal segmentation mask map according to the first text feature and the first image feature includes: Perform matrix cross product on the first text feature and each of the second feature information to obtain multiple similarity matrices; Performing element-by-element averaging on each of the similarity matrices to obtain an average value matrix; Performing bilinear interpolation on the average value matrix to obtain an interpolation matrix; Inputting the interpolation matrix into a segmentation head for processing; The abnormal segmentation mask image output by the segmentation head is obtained.
6. The model training method according to claim 1, characterized in that: The step of training the pre-trained model according to the abnormal segmentation mask image and the text prompt training sample includes: Freezing network parameters of the text encoder and the image encoder; Calculating a loss function based on the abnormal segmentation mask image and the text prompt training sample to obtain a gradient parameter; According to the gradient parameters, the network parameters of the adapter are updated.
7. The model training method according to claim 6, characterized in that: The step of calculating the loss function based on the abnormal segmentation mask image and the text prompt training sample to obtain the gradient parameter includes: Performing Dice function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a first parameter value; Performing Focal function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a second parameter value; Performing BCE function calculation according to the abnormal segmentation mask image and the text prompt training sample to obtain a third parameter value; The first parameter value, the second parameter value and the third parameter value are summed to obtain the gradient parameter.
8. An industrial image defect detection method, characterized in that: The industrial image defect detection method comprises: Obtain a pre-trained model; the pre-trained model is a pre-trained model obtained by executing the model training method according to any one of claims 1 to 7; Acquire the industrial image to be identified; Inputting the industrial image to be identified into the pre-trained model for processing; Obtain the industrial image defect detection result output by the pre-trained model.
9. A computer device, characterized in that: It includes a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the model training method and / or industrial image defect detection method described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute the model training method and / or industrial image defect detection method described in any one of claims 1 to 8 when executed by the processor.
Citation Information
Patent Citations
Defect detection method and device based on pre-training
CN116416247A
Text classification method and device, terminal equipment and storage medium
CN116450825A
Image classification method and system based on class incremental learning
CN116977718A
Conceptual distillation and CLIP-based promptable segmentation model pre-training method and system
CN117671426A
Defect detection method, multi-mode fusion network and equipment
CN118196027A
Cited By
Multi-modal medical image anomaly detection method, medium and equipment
CN120047749A
Large model method and device for AVI detection defect data judgment and medium
CN120510107A
