Model training method, defect detection method and related apparatuses

Abstract

Provided in the present application are a model training method, a defect detection method and related apparatuses. The embodiments of the present application can be applied to various scenarios such as computer vision. The model training method trains an initial detection model by means of using a first sampled image set containing some of images that have been used for training and a full newly-added second training image set, and thus, compared with using full historical training data and full newly-added training data to train an initial model, more saves time and reduces the GPU hour consumption; adaptive evaluation of corresponding first weights of model parameters restricts updating of the model parameters with respect to historical training data, so as to solve the problem of knowledge forgetting caused by only using full newly-added data for model fine-tuning, thus improving the learning capability of the detection model; and an optimized detection model obtained by using the model training method is used to detect defects in an image for detection, thus improving the effect and accuracy of defect recognition.

Description

A model training method, defect detection method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on April 2, 2024, with application number 2024103936252 and application name “A model training method, defect detection method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to model training and defect detection. Background Art

[0003] Industrial quality inspection refers to the process of quality inspection and control of products or components during the industrial production process. Its purpose is to ensure that the products meet the prescribed quality standards and requirements, thereby improving the product qualification rate and quality stability.

[0004] Currently, deep learning-based defect detection technology has achieved high accuracy. Specifically, large amounts of training data are acquired in advance for model training, and then the resulting neural network model is used for defect detection. However, as industrial quality inspection projects progress, the amount of data collected to train deep learning models gradually increases. In the middle and late stages of the project, the huge amount of training data, long training times, and high consumption of training cards make the cost of each iteration very high.

[0005] In related technologies, two main approaches are used: using full training data for model iteration and using incremental data for model fine-tuning. However, both approaches present challenges. Using full data for model iteration, if the data volume is large, requires high computational resources and time. Using incremental data for model fine-tuning, as training time increases or the number of fine-tuning cycles increases, the model gradually deviates from its original knowledge, leading to catastrophic forgetting of historical data. This in turn leads to a rapid decline in the model's overall capabilities and instability. Summary of the Invention

[0006] The present application provides a model training method, a defect detection method, and related devices. In the model training method, an initial detection model is trained using a portion of previously trained historical training data and all newly added training data. This solves the problems of excessively high computing resources and time costs associated with using all training data for model updates in related technologies, as well as the catastrophic forgetting caused by fine-tuning the model using all newly added training data. In the defect detection method, the optimized detection model obtained by the model training method is used to detect defects in the image to be inspected, thereby improving the recognition effect and accuracy of defect identification.

[0007] One aspect of the present application provides a model training method, comprising:

[0008] Obtaining a first training image set, a second training image set, and an initial detection model, wherein the initial detection model is trained based on the first training image set, the first training image set includes a first sample defect image, the second training image set includes a second sample defect image, and the second sample defect image is different from the first sample defect image, and the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameters are used to represent the importance of the multiple model parameters of the initial detection model;

[0009] The initial detection model is trained based on the first sampling image set, the second training image set and the first weight parameter to adjust multiple model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes some sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model.

[0010] Another aspect of the present application provides a model training device, comprising: a training image and initial model acquisition module and a model training module; specifically:

[0011] a training image and initial model acquisition module, configured to acquire a first training image set, a second training image set, and an initial detection model, wherein the initial detection model is obtained by training based on the first training image set, the first training image set includes a first sample defect image, the second training image set includes a second sample defect image, and the second sample defect image is different from the first sample defect image; the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameters are used to characterize the importance of the model parameters of the initial detection model;

[0012] A model training module is used to train the initial detection model based on the first sampling image set, the second training image set and the first weight parameter to adjust multiple model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes some sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model.

[0013] Another aspect of the present application provides a defect detection method, comprising:

[0014] Obtain the image to be detected;

[0015] Inputting the image to be detected into the optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using any of the above-mentioned model training methods;

[0016] Based on the feature extraction network in the optimized detection model, feature extraction is performed on the image to be detected to obtain the features of the image to be detected;

[0017] Based on the classification network, defect classification is performed on the features of the image to be detected to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to represent the classification result of whether the image to be detected contains defects.

[0018] Another aspect of the present application provides a defect detection device, comprising:

[0019] An image acquisition module to be detected, used to acquire the image to be detected;

[0020] An image input module to be detected is used to input the image to be detected into an optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using any of the above-mentioned model training methods;

[0021] The feature extraction module of the image to be detected is used to extract features of the image to be detected based on the feature extraction network in the optimized detection model to obtain features of the image to be detected;

[0022] The feature classification module of the image to be detected is used to classify defects of the features of the image to be detected based on the classification network to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to represent the classification result of whether the image to be detected contains defects.

[0023] In another aspect, an embodiment of the present application provides a computer device, including:

[0024] Processor, communication interface, memory and communication bus;

[0025] Wherein, the processor, the communication interface and the memory complete communication with each other through the communication bus; the communication interface is an interface of the communication module;

[0026] The memory is used to store a computer program and transmit the computer program to the processor; the processor is used to call the computer program in the memory to execute the above method.

[0027] On the other hand, an embodiment of the present application provides a storage medium, which is used to store a computer program, and the computer program is used to execute the method of the above aspect.

[0028] On the other hand, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the above method.

[0029] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0030] The present application provides a model training method and related devices. The model training method trains an initial detection model by using part of the historical training data that has been trained (a first sampling image set) and the fully newly added training data (a second training image set). Compared with using the full amount of historical training data and the fully newly added training data to train the initial detection model, it saves time and reduces the consumption of GPU cards. By adaptively evaluating the first weight parameter set for the importance of model parameters, the update amplitude of the training data for some important model parameters is limited during the training phase. Since these important model parameters have an important influence on the accurate identification of defects involved in the first training image set, based on the accurate indication of the first weight parameter set, the parameter adjustment range of these model parameters will be relatively limited, thereby solving the problem of knowledge forgetting caused by fine-tuning the model using only the fully added training data, improving the learning ability of the detection model, and enhancing the recognition effect and accuracy of the detection model for defect identification.

[0031] The present application also provides a defect detection method and related devices, which detect defects in the image to be detected by the optimized detection model obtained by the above-mentioned model training method, and obtain a classification result of whether the image to be detected contains defects, thereby improving the recognition effect and accuracy of defect identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of an architecture of a model training system provided in one embodiment of the present application;

[0033] FIG2 is a flow chart of a model training method provided in an embodiment of the present application;

[0034] FIG3 is a flow chart of a model training method provided by another embodiment of the present application;

[0035] FIG4 is a flow chart of a model training method provided by another embodiment of the present application;

[0036] FIG5 is a flow chart of a model training method provided by another embodiment of the present application;

[0037] FIG6 is a flow chart of a model training method provided by another embodiment of the present application;

[0038] FIG7 is a flow chart of a model training method provided by another embodiment of the present application;

[0039] FIG8 is a flow chart of a model training method provided by another embodiment of the present application;

[0040] FIG9 is a flow chart of a model training method provided by another embodiment of the present application;

[0041] FIG10 is a flow chart of a defect detection method provided by an embodiment of the present application;

[0042] FIG11 is a schematic diagram of the structure of a model training device provided in one embodiment of the present application;

[0043] FIG12 is a schematic structural diagram of a model training device provided in another embodiment of the present application;

[0044] FIG13 is a schematic structural diagram of a model training device provided in another embodiment of the present application;

[0045] FIG14 is a schematic structural diagram of a defect detection device provided in one embodiment of the present application;

[0046] FIG15 is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] An embodiment of the present application provides a model training method, which trains an initial detection model by using part of the trained historical training data (a first sampling image set) and the fully newly added training data (a second training image set). Compared with using the full amount of historical training data and the fully newly added training data to train the initial model, it saves time and reduces the consumption of GPU cards. By adaptively evaluating the corresponding first weights of the model parameters, the update of the model parameters based on the historical training data is limited, which solves the problem of knowledge forgetting caused by fine-tuning the model using only the fully added data, improves the learning ability of the detection model, and detects defects in the image to be detected by using the optimized detection model obtained by the model training method, thereby improving the recognition effect and accuracy of defect recognition.

[0048] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0050] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0051] Defect detection technology: This technology determines the presence of defects in captured images. In modern industrial manufacturing, defective parts are inevitably present. Detecting these defects using defect detection technology plays a crucial role in improving production processes and boosting factory efficiency.

[0052] Incremental learning: refers to a learning system that can continuously learn new knowledge from new samples and can retain most of the knowledge that has been learned previously. Incremental learning is very similar to the human learning model. Because humans learn and accept new things every day during the process of growth, learning is gradual, and humans generally do not forget the knowledge that has been learned. In the embodiment of the present application, incremental learning is used for the training of the defect detection model. Since the number of defects is constantly updated in different production cycles, the defect detection model also needs to continuously learn knowledge from new defect images.

[0053] Incremental learning includes data increment scenarios and category increment scenarios.

[0054] Incremental data scenarios: This refers to scenarios where the data category distribution remains roughly the same between incremental stages, with only a simple increase in the amount of data. In the embodiments of this application, incremental data learning is used to train the defect detection model. Since the number of defects is constantly updated during different production cycles, the defect detection model also needs to continuously learn from new defect images.

[0055] Category increment scenario: refers to the scenario where data categories do not overlap between different incremental stages.

[0056] Full data model: refers to a deep learning model obtained by iterative model training using all accumulated data.

[0057] Incremental data model: refers to a model that is trained iteratively using a small amount of existing data and new data, or using only new data without using existing data.

[0058] Catastrophic forgetting: Catastrophic forgetting refers to the ability of a model in incremental learning to forget previously learned tasks when learning a new task. Specifically, when a model learns a new task, it adjusts its parameters to adapt to the requirements of the new task. However, these parameter adjustments may cause the model's performance on the previously learned task to degrade, or even completely forget the previously learned task. Catastrophic forgetting is a significant issue in incremental learning, limiting its scope of application. There are two main causes of catastrophic forgetting: first, parameter conflict, where the parameter adjustments for a new task conflict with those for an old task, causing information from the old task to be forgotten; second, representation space limitations, where the model's limited representation space prevents it from simultaneously representing information from multiple tasks.

[0059] The following is a brief introduction to the design concept of the embodiment of this application:

[0060] In industrial manufacturing, defective parts are inevitably present in various parts produced by companies. To ensure product quality, companies must identify and process these defective parts. Furthermore, identifying defective parts and analyzing their morphological characteristics and proportions is crucial for improving production processes and increasing production line yields.

[0061] In traditional industrial manufacturing, companies often rely on manual inspection to detect and classify product defects. However, for quality inspectors, the workload is high and the work content is monotonous, leading to high employee turnover. For companies, this translates to high inspection costs (personnel costs) and low quality inspection efficiency. However, given sufficient data, machine learning technology can surpass the accuracy of manual quality inspection and largely replace it, reducing costs and increasing efficiency for companies.

[0062] Currently, deep learning-based defect detection technology has achieved high accuracy. Specifically, a large amount of training data is acquired in advance for model training, and then the trained neural network model is used for defect detection. Taking industrial quality inspection of steel strip as an example, during this process, defect detection is often performed on strip images to achieve defect detection and identification. Specifically, deep learning-based defect detection methods are trained on a large amount of strip image data to establish a defect detection model. The defect detection model then analyzes and identifies images of the strip surface to detect defects.

[0063] Existing common machine learning techniques use a batch learning model, assuming all training samples are available at once before training. After learning these samples, the learning process ceases, and no new knowledge is learned. As industrial quality inspection projects progress, the amount of data collected for training deep learning models gradually increases. For the same defect category, historical training data and newly added data must be consolidated and retrained using the full amount of training data to ensure the new model can accurately identify defects in that category.

[0064] In related technologies, two main technical solutions are adopted: using full training data for model iteration and using incremental data for model fine-tuning.

[0065] Among them, using the full amount of training data for model iteration means that in the full amount of data increment scenario, the full amount of historical training data for training the current model and the newly added data are used as the entire training data of the model to iteratively train the model. This method can stably improve and iterate the model, but as the amount of data increases, the training time and cost will also increase. If the amount of data is large, the computing resources and time cost required for training are very high.

[0066] Using incremental data for model fine-tuning means fine-tuning an existing model using only newly added data. Therefore, when training a model, to maintain the model's existing capabilities, a smaller learning rate can be used or only the last few layers of the deep learning model can be trained. Because fine-tuning doesn't utilize past historical data, even with a reduced learning rate and training parameter range, the model still risks forgetting previously learned knowledge, leading to a decrease in overall performance and overfitting to the current data. This risk increases significantly with increasing fine-tuning cycles.

[0067] In order to solve the problem of high computing resources and time cost required for training when using full data for model iteration, and the problem of catastrophic forgetting of historical data when using incremental data for model fine-tuning, which leads to a rapid decline in the overall model capability and instability, transfer learning algorithms and incremental learning algorithms came into being.

[0068] Transfer learning is a method of migrating the capabilities of a model to a target data domain. The target data domain is usually quite different from the original data domain of the model, and the performance on the original data domain is not considered. Therefore, the effect of the model on the original data after transfer learning is usually greatly reduced. In the actual business handled by the embodiments of this application, the newly added data is usually not much different from the original data in terms of domain, and the actual requirements in the business must maintain the performance on the original data, so this type of method is not applicable.

[0069] Incremental learning is primarily used for incremental task learning. Although data increment is a special form of incremental learning, there is relatively little research on this topic. Existing transfer learning and incremental learning techniques have not achieved good results in real-world business scenarios and are relatively complex to apply.

[0070] In view of this, the present application provides a model training method and related devices. The model training method includes: first, obtaining a first training image set, a second training image set and an initial detection model, wherein the initial detection model is obtained by training based on the first training image set, the first training image set includes sample defect images, the second training image set includes sample defect images and is different from the sample defect images included in the first training image set, and the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameter is used to characterize the importance of the model parameters of the initial detection model; then, based on the first sampling image set, the second training image set and the first weight parameter, the initial detection model is trained to adjust the model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes some sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model. The model training method provided in the embodiment of the present application uses a partially trained first sampling image set and a fully newly added second training image set to train the initial detection model. Compared with using the full amount of historical training data and the fully newly added training data to train the initial model, it saves time and reduces the consumption of GPU cards. By adaptively evaluating the corresponding first weights of the model parameters, the update of the model parameters based on historical training data is limited. Compared with the problem of knowledge forgetting caused by fine-tuning the model using only the full amount of newly added data, the learning ability of the model is improved, and the recognition effect and accuracy of the model for defect recognition are enhanced.

[0071] In addition, the embodiment of the present application also provides a defect detection method and related devices, the defect detection method includes: first, obtaining an image to be detected; then, inputting the image to be detected into an optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using the model training method as described above; then, based on the feature extraction network in the optimized detection model, feature extraction is performed on the image to be detected to obtain features of the image to be detected; finally, based on the classification network, defect classification is performed on the features of the image to be detected to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to characterize the classification result of whether the image to be detected contains defects. The defect detection method provided in the embodiment of the present application detects defects in the image to be detected using the optimized detection model obtained by the model training method, and obtains a classification result of whether the image to be detected contains defects, thereby improving the recognition effect and accuracy of defect recognition.

[0072] For ease of understanding, please refer to Figure 1, which is an application environment diagram of the model training method or defect detection method in the embodiment of the present application. As shown in Figure 1, the model training method in the embodiment of the present application is applied to the model training system, and the defect detection method is applied to the defect detection system. The model training system includes: a server and a terminal device; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and the embodiment of the present application is not limited here.

[0073] The server first obtains a first training image set, a second training image set and an initial detection model, wherein the initial detection model is trained based on the first training image set, the first training image set includes sample defect images, the second training image set contains sample defect images and is different from the sample defect images contained in the first training image set, and the initial detection model contains a first weight parameter set, the first weight parameter set contains first weight parameters of multiple model parameters in the initial detection model, and the first weight parameter is used to characterize the importance of the model parameters of the initial detection model; then, the server trains the initial detection model based on the first sampling image set, the second training image set and the first weight parameter to adjust the model parameters of the initial detection model to obtain the trained optimized detection model, wherein the first sampling image set contains some sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model.

[0074] The following will introduce the model training method in this application from the perspective of the server. Please refer to Figure 2. The model training method provided in this embodiment includes: Step S110 to Step S120. Specifically:

[0075] S110: Obtain a first training image set, a second training image set, and an initial detection model.

[0076] Among them, the initial detection model is obtained by training based on the first training image set, the first training image set includes a first sample defect image, the second training image set contains a second sample defect image and the second sample defect image is different from the first sample defect image, the initial detection model includes a first weight parameter set, the first weight parameter set includes the first weight parameters of multiple model parameters in the initial detection model, and the first weight parameter is used to characterize the importance of multiple model parameters of the initial detection model.

[0077] It is understood that the first training image set includes at least one first sample defect image, which is the basic data used to train the initial detection model. The second training image set includes at least one second sample defect image sampled during the production cycle, which contains data on over-detection and missed detection situations that occurred during the actual use of the initial detection model for defect detection. The initial detection model is a model that has been trained based on the first training image set. It has a feature extraction network and a classification network and can identify product quality defects in images. Specifically:

[0078] The sample defect images in the first training image set are the full amount of training data when training the initial detection model. The initial detection model is obtained by training based on the first training image set. The initial detection model can recognize defects in products in the image, such as strip steel. The initial detection model includes a feature extraction network and a classification network. The feature extraction network is used to extract features from the image input into the initial detection model, while the classification network is used to classify image features to obtain a prediction result of whether the product in the image contains defects. The classification network in the initial detection model has a first weight parameter set, which can be in a matrix form or a set form. The first weight parameter set includes multiple first weight parameters, each weight parameter corresponds to a model parameter in the classification network. Therefore, the first weight parameter is used to characterize the importance of the model parameters of the initial detection model to the identification of quality defects.

[0079] After the initial detection model is deployed online, some over-detection and missed detection situations will occur as the initial detection model is used. Over-detection refers to the situation where the initial detection model mistakenly judges normal products or samples as defective or unqualified. This means that the initial detection model makes excessive judgments on normal products, resulting in misjudgment of them as problematic products. Missed detection refers to the situation where the initial detection model fails to detect products or samples that are actually defective or unqualified. This means that the model misses some problematic products that should have been detected, resulting in missed detections. The amount of data generated by these over-detection and missed detection situations is smaller than the amount of data in the first training image set used for training before the initial detection model was put online, but this data is still crucial for the optimization and performance improvement of the initial detection model. Therefore, after the initial detection model has been put online for a period of time, the over-detection and missed detection situations in the production cycle are counted and sampled to obtain the second training image set.

[0080] S120 , training the initial detection model based on the first sampling image set, the second training image set, and the first weight parameter to adjust multiple model parameters of the initial detection model to obtain a trained optimized detection model.

[0081] The first sampling image set includes some sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model.

[0082] It is understandable that the first sampling image set is obtained by sampling the sample defect images in the first training image set according to the sampling ratio. The purpose of sampling is to reduce the size of the training data while retaining important information. The sample defect images in the first sampling image set obtained after sampling the first training image set and the sample defect images in the second training image set are used together as training samples for the initial detection model to further train the initial detection model. Compared with using the full first training image set and the second training image set to train the initial detection model, this reduces the time required to train the initial model and reduces the consumption of GPU card time.

[0083] The initial detection model is fine-tuned using the first sampling image set and the second training image set to obtain the trained optimized detection model. In the process of fine-tuning the initial detection model, for those model parameters that are very important in the initial detection model, that is, the model parameters that have an important impact on the accurate recognition of the defects involved in the first training image set, in this training, under the accurate guidance of the first weight parameter set, the parameter adjustment range of these model parameters will be relatively limited, so that the optimized detection model after training will not excessively forget the knowledge that has been learned before, and maintain the recognition accuracy of the defects that can be originally recognized. Therefore, the update of the model parameters in the initial detection model is targetedly restricted by the first weight parameter set, and the update amplitude of the important model parameters in the initial detection model is limited to ensure that the knowledge learned before is not excessively forgotten during the fine-tuning process. The optimized detection model obtained in this way can make correct detection results for the over-killing and missed detection situations of the original initial detection model, thereby improving the accuracy of defect detection.

[0084] The model parameters in the initial model are crucial for accurately identifying sample categories (e.g., whether a sample is defective). Some model parameters are crucial (highly important), while others may be less important (lowly important). That is, for input images that were originally accurately identified (e.g., the first sample defect image), significant changes in important model parameters will directly affect the initial model's recognition accuracy for these images.

[0085] Generally speaking, the higher the importance indicated by the first weight parameter of a model parameter, the smaller the numerical range that can be adjusted for this model parameter during the model training phase.

[0086] The method provided in the embodiments of this application optimizes the initial detection model by utilizing sampling data and weight constraints to address over-detection and missed detection issues and improve model performance. This method can effectively improve the model's effectiveness with limited computing resources. In practical applications, it is necessary to appropriately set the sampling ratio and training strategy based on the specific situation to achieve optimal detection results.

[0087] For ease of understanding, assume that the initial detection model launched at time t is Among them, t represents the time node when the initial detection model is launched, i represents the model parameters in the initial detection model, and the initial detection model In the first training image set D 1：t After the initial detection model has been deployed for a period of time, at time t+1, the over-detection data and missed detection data of the initial detection model are counted to obtain the second training image set D t+1 In related technologies, the full amount of training data is generally used, that is, D1：t +D t+1 , and the method provided in the embodiment of the present application determines the sampling ratio p% of the first training image set, and obtains the first training image set D by sampling the first training image set D. 1：t The sample defect images in are sampled to obtain the first sampling image set. The first sampling image set is combined with the second training image set D t+1 Merge together to form new training data D′ t+1 , through the first sampling image set and the second training image set D t+1 The training data D′ is combined together t+1 The initial detection model is Perform training to adjust the model parameters of the initial detection model to obtain the optimized detection model after training

[0088] The model training method provided in the embodiment of the present application uses a partially trained first training image set and a fully newly added second training image set to train the initial detection model. Compared with using the full amount of historical training data and the fully newly added training data to train the initial model, it saves time and reduces the consumption of GPU cards. By adaptively evaluating the corresponding first weights of the model parameters, the update range of the training data for some important model parameters is limited. Compared with the problem of knowledge forgetting caused by fine-tuning the model using only the full amount of newly added data, the learning ability of the model is improved, and the recognition effect and accuracy of the model for defect recognition are enhanced.

[0089] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG. 2 of the present application, please refer to FIG. 3 , step S120 further includes sub-steps S121 to S123. Specifically:

[0090] S121. Obtain model parameters of the basic detection model.

[0091] The initial detection model is obtained by adjusting the model parameters of the basic detection model. For example, the initial detection model can be obtained by training the basic detection model based on the first training image set.

[0092] It is understandable that, assuming that the basic detection model launched at time t-1 is Among them, t-1 represents the time node when the basic detection model is launched, i represents the model parameters in the basic detection model, and the basic detection model It is the basic training image set D of all previous t-1 After the basic detection model has been deployed for a period of time, at time node t, statistics are collected on the over-killed data and missed detection data to obtain the third training image set.t-1 The sampling ratio p% is obtained by sampling p% from the basic training image set D t-1 The sample defect images in the image are sampled to obtain a second sample image set. The second sample image set is combined with the third training image set to form new training data. The new training data formed by combining the second sample image set and the third training image set is used to test the basic detection model. Perform training to adjust the model parameters of the basic detection model to obtain the trained initial detection model

[0093] S122. Calculate the objective function of the initial detection model according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set.

[0094] It is understood that the objective function of the initial detection model is calculated based on the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampled image set, and the second training image set. The objective function is generally a metric used to evaluate model performance, and can be accuracy, misclassification rate, recall rate, etc. The purpose of calculating the objective function is to quantitatively evaluate the performance of the initial detection model on the second training image set.

[0095] S123. Based on the objective function of the initial detection model, the initial detection model is trained to adjust the model parameters of the initial detection model.

[0096] For ease of understanding, the objective function of the initial detection model can be calculated using the following formula (1):

[0097] Among them, L t+1 represents the objective function of the initial detection model at time t+1. λ is an adjustable hyperparameter that controls the detection model's plasticity (its ability to learn new tasks at time t+1) and stability (its ability to remember old tasks at time t). It represents the first weight parameter set corresponding to the initial detection model at time t. The first weight parameter set includes the first weight parameters of multiple model parameters in the initial detection model. The first weight parameter is used to characterize the importance of the model parameters of the initial detection model. The first weight can be used to measure the importance of multiple model parameters of the initial detection model to the old task. The importance weight is obtained iteratively. Represents the model parameters of the initial detection model at time t. represents the model parameters of the basic detection model at time t-1. t+1 |θ) represents the probability function. D′ t+1 Represents the first sampling image set and the second training image set Dt+1 The data are merged together. θ represents the model parameters of the basic detection model.

[0098] When the second training image set D t+1 Fine-tune the initial detection model to obtain an optimized detection model Finally, the entire training data set is computed in all epochs to ensure that all training data is accessed, thereby learning the importance of model parameters for the current task. At this point, each iteration consists of a forward propagation and a backpropagation. However, it is important to note that during the process of learning parameter importance, the model is not updated according to the gradient descent algorithm during backpropagation.

[0099] The method provided in the embodiment of the present application uses an objective function to make more precise adjustments and optimizations to the initial detection model. The calculation of the objective function provides a quantitative assessment of the performance of the model on a specific data set. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation method based on the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine-tuning and optimization of these steps are to enable the model to better adapt to changes and demands of data in practical applications, and to improve its performance and generalization ability.

[0100] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG3 of the present application, please refer to FIG4 , sub-step S122 further includes sub-steps S1221 to S1224. Specifically:

[0101] S1221. Generate an incremental data objective function of the initial detection model as a first optimization item based on the first sampling image set and the second training image set.

[0102] It can be understood that the incremental data objective function of the initial detection model can be calculated by the following formula (2): ZT = -logp(D′ t+1 |θ) (2);

[0103] Among them, p(D′ t+1 |θ) represents the probability function. D′ t+1 Represents the first sampling image set and the second training image set D t+1 The data is merged together. θ represents the basic model parameters of the detection model.

[0104] The incremental data objective function may reflect a loss function when the initial detection model is trained using training samples consisting of the first sampling image set and the second training image set.

[0105] S1222: Calculate the parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model.

[0106] It can be understood that the parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model can reflect the parameter value difference between the model parameters of the two detection models.

[0107] The parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model can be calculated by the following formula (3):

[0108] Among them, θ i represents the parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model, Represents the model parameters of the initial detection model at time t. Represents the model parameters of the basic detection model at time t-1.

[0109] S1223. Generate a second optimization item according to the first weight parameter, the parameter difference, and the hyperparameter.

[0110] The second optimization item is used to limit the adjustment range of important parameters during training, the important parameters are identified by the first weight parameters, and the hyperparameter is used to identify the influence of the second optimization item when training the initial detection model.

[0111] S1224. Construct an objective function of the initial detection model according to the first optimization item and the second optimization item.

[0112] It can be understood that the hyperparameter λ is an adjustable hyperparameter used to control the plasticity (ability to learn new tasks at time t+1) and stability (ability to remember old tasks at time t) of the detection model.

[0113] Optionally, a second optimization item may be generated according to the first weight parameter, the parameter difference, and the hyperparameter, and an objective function may be constructed according to the first optimization item and the second optimization item in the following manner:

[0114] Multiply the hyperparameter, the first weight parameter, and the square of the parameter difference, and add the multiplication result to the incremental data objective function of the initial detection model to obtain the objective function of the initial detection model.

[0115] It can be understood that the objective function of the initial detection model can be calculated by the following formula (4):

[0116] Among them, L t+1represents the objective function of the initial detection model at time t+1. λ is an adjustable hyperparameter. It represents the first weight parameter set corresponding to the initial detection model at time t. The first weight parameter set can be used to measure the importance of the model parameters of the initial detection model to the old task. The weight parameters in the first weight parameter set can be obtained iteratively. Represents the model parameters of the initial detection model at time t. represents the model parameters of the basic detection model at time t-1. t+1 |θ) represents the probability function. D′ t+1 Represents the first sampling image set and the second training image set D t+1 The data is merged together. θ represents the basic model parameters of the detection model.

[0117] The method provided in the embodiments of the present application calculates the objective function of the initial detection model by processing sampled data, comparing model parameters, and applying weights. This objective function can be used as an indicator to evaluate model performance, helping us understand the model's performance on a specific dataset and guiding further optimization and improvement. This calculation process can help us more accurately evaluate and improve the detection model to enhance its performance and accuracy.

[0118] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG. 3 of the present application, please refer to FIG. 5 , step S122 further includes sub-steps S124 to S125 .

[0119] S124: Obtain a clipping threshold.

[0120] The clipping threshold is used to represent the gradient threshold when the parameters of the initial detection model are updated, so as to limit the gradient range when the parameters of the initial detection model are updated.

[0121] S125. Construct an objective function of the initial detection model according to the cropping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set.

[0122] Preferably, step S125 includes the following three sub-steps:

[0123] Step 1), calculating the root of the first weight parameter, and calculating the parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model, multiplying the root of the first weight parameter and the parameter difference to obtain a first calculated value;

[0124] Step 2), if the first calculated value is less than or equal to the cropping threshold, calculating the objective function of the initial detection model according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set and the second training image set;

[0125] Step 3) If the first calculated value is greater than the clipping threshold, the objective function of the initial detection model is calculated based on the clipping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set and the second training image set.

[0126] It can be understood that the objective function of the initial detection model can be calculated by the following formula (5):

[0127] The hyperparameter λ in formula (5) is actually balancing the ability to remember old tasks and the ability to learn new tasks, that is, it is used to control the plasticity (ability to learn new tasks at time t+1) and stability (ability to remember old tasks at time t) of the detection model. When λ = 0, the optimization of the model is actually the most common fine-tuning, which will lead to catastrophic forgetting. When λ is equal to positive infinity, the new model will always maintain the same parameters as the old model, so the model cannot learn the knowledge of the new task. In actual business, this application tried λ to take values ​​such as 0.2, 0.4, 1, 2, and found that the model accuracy will get better and better. However, when the value of λ reaches 10, the training will become extremely unstable, which makes it impossible for this application to continue to find the best balance between new and old tasks. Through further observation, before the training gradient explodes, the gradient of some parameters will become very large.

[0128] Taking a single parameter θ in formula (5) as an example, the quadratic term The gradient of with respect to θ is Since the data distributions between different tasks vary greatly, the model parameters will quickly adapt to the new data distribution in the initial stage of incremental fine-tuning. At this time, the gradient is very likely to grow rapidly, leading to gradient explosion. To solve this problem, this application adopts the following method to limit the gradient to a reasonable range. Specifically, replace the quadratic term in formula (5) with formula (6):

[0129] Among them, β is a hyperparameter, representing a pre-set clipping threshold, that is, when the gradient exceeds this threshold, restrictive measures will be taken.

[0130] Therefore, when calculating the objective function of the initial detection model, it is necessary to perform case-by-case calculations. Specifically, first calculate the root of the first weight parameter, as well as the parameter difference between the model parameters of the initial detection model and the model parameters of the basic detection model, and multiply the root of the first weight parameter by the parameter difference to obtain a first calculated value; if the first calculated value is less than or equal to the clipping threshold, calculate the objective function of the initial detection model based on the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set; if the first calculated value is greater than the clipping threshold, calculate the objective function of the initial detection model based on the clipping threshold, the root of the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set.

[0131] The objective function of the initial detection model can be expressed by the following formula (7):

[0132] By taking the derivative of formula (6) with respect to θ, we can obtain the following formula (8):

[0133] From formula (8), we can see that the gradient is effectively limited to within the range, and when When , the objective function of the initial detection model can be calculated using formula (7). After the above transformation, λ in formula (7) can take a larger value to make the model have better performance and avoid gradient explosion.

[0134] For the problem of gradient explosion when λ is large, gradient clipping can also be used. Specifically, it can be achieved by the following formula (9):

[0135] Among them, g is the gradient vector of the loss function with respect to the parameter θ, g i represents an item in the vector, β g is the clipping threshold of the norm of g. Although formula (9) can also prevent gradient explosion, the final learning effect of the model is not as good as that of formula (6) and formula (7). The reason is that in formula (9), the gradient clipping of all parameters is coupled together, and the clipping ratio is fixed; while the solution using formula (7) is decoupled. Whether each gradient is clipped and the clipped value are independent of other gradients, and only depend on the importance of the current parameter to the old task.

[0136] The method provided in the embodiments of the present application performs more refined adjustments and optimizations on the optimized detection model through the calculation of the objective function. The calculation of the objective function provides a quantitative assessment of the performance of the model on a specific data set. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation method based on the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine-tuning and optimization of these steps are to enable the model to better adapt to data changes and needs in practical applications, and to improve its performance and generalization ability.

[0137] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG. 3 of the present application, referring to FIG. 6 , step S120 further includes step S130, specifically:

[0138] S130: Calculate a second weight parameter set of the optimized detection model according to the objective function of the initial detection model and the first weight parameter set.

[0139] Among them, the second weight parameter set includes second weight parameters of multiple model parameters in the optimized detection model, the second weight parameters are used to characterize the importance of the model parameters of the optimized detection model, and the second weight parameter set is used to limit the adjustment range of the model parameters in the optimized detection model.

[0140] It is understandable that the calculation of the second weight parameter set is based on the objective function and the first weight parameter set. It reflects the performance of the optimized detection model and the impact of different model parameters on the performance. By calculating the second weight parameter set, the importance of the model parameters can be further adjusted and optimized to improve the performance and generalization ability of the model, and provide a basis for the subsequent training of the optimized detection model. When the model is subsequently iterated through the newly added training data, the weight parameters corresponding to each iterated model need to be calculated to limit the adjustment range of the model parameters during the model iteration.

[0141] The second weight parameter set corresponding to the optimized detection model can be calculated by the following formula (10):

[0142] in, Represents the second weight parameter set corresponding to the optimized detection model, represents the first weight parameter set corresponding to the initial detection model, which can also be obtained by iteration through formula (10), j represents the number of losses, Indicates the derivation of the objective function. The weight matrix representing the importance of the second training image set to the model parameters of the optimized detection model is compared with the weight matrix of the importance of the model parameters of the initial detection model. Accumulate and obtain the weight matrix of the importance of the model parameters of the optimized detection model This is the second set of weight parameters. This weight matrix represents the importance of parameters for all learned tasks. When fine-tuning the initial detection model at time t+2, this weight matrix is ​​used to better remember the knowledge from time 1 to time t+1.

[0143] When the second training image set D t+1 Fine-tune the initial detection model to obtain an optimized detection model Finally, the full training data is calculated in all epochs to ensure that all training data is accessed during training, thereby learning the importance of model parameters for the current task. At this time, each iteration consists of a forward propagation and a backpropagation. However, it is important to note that during the process of learning parameter importance, the model is not updated according to the gradient descent algorithm during backpropagation.

[0144] The method provided in the embodiment of the present application, through the determination of the second weight parameter set, performs more refined adjustment and optimization on the optimized detection model. The calculation of the second weight parameter set adjusts the parameter importance of the model according to the evaluation results. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation method based on the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine adjustment and optimization of these steps are to enable the model to better adapt to the changes and needs of the data in practical applications, and to improve its performance and generalization ability.

[0145] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG6 of the present application, please refer to FIG7 , step S130 further includes sub-steps S131 to S132. Specifically:

[0146] S131. Perform multiple loss calculations on the objective function of the initial detection model to obtain multiple third weight parameter sets.

[0147] The loss calculation is performed by performing derivative calculation on the objective function of the initial detection model. The third weight parameter set includes multiple third weight parameters, and the third weight parameters are used to characterize the importance of the model parameters of the initial detection model for correctly identifying the second training image set.

[0148] The number of third weight parameter sets may be related to the number of loss calculations. For example, a corresponding third weight parameter set may be obtained for each loss calculation.

[0149] It can be understood that the third weight parameter set is a weight matrix of the importance of the second training image set to the model parameters of the optimized detection model, which can be expressed by the following formula (11):

[0150] in, Indicates the derivation of the objective function.

[0151] S132: Add the multiple third weight parameter sets to the first weight parameter set to obtain a second weight parameter set corresponding to the optimized detection model.

[0152] It is understandable that the second weight parameter set corresponding to the optimized detection model can be calculated by the following formula:

[0153] in, Represents the second weight parameter set corresponding to the optimized detection model, represents the first weight parameter set corresponding to the initial detection model. The first weight can also be obtained by iteration through formula (12). j represents the number of losses. Indicates the derivation of the objective function. The weight matrix representing the importance of the second training image set to the model parameters of the optimized detection model is compared with the weight matrix of the importance of the model parameters of the initial detection model. Accumulate and obtain the weight matrix of the importance of the model parameters of the optimized detection model This is the second set of weight parameters. This weight matrix represents the importance of parameters for all learned tasks. When fine-tuning the initial detection model at time t+2, this weight matrix is ​​used to better remember the knowledge from time 1 to time t+1.

[0154] The method provided in the embodiments of the present application allows for a more accurate evaluation of the importance of model parameters for optimizing the detection model, which helps to improve the performance and generalization ability of the model.

[0155] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG. 2 of the present application, please refer to FIG. 8 , step S120 further includes sub-steps S126 to S129. Specifically:

[0156] S126. Obtain the number of images corresponding to the training batch when training the initial detection model.

[0157] The number of images is N, and N is an integer greater than 1.

[0158] It can be understood that the number of images corresponding to the training batch refers to the number of training batch pictures.

[0159] S127 . Take K first sample defect images from the first sampling image set, and take L second sample defect images from the second training image set.

[0160] Wherein, K and L are both integers greater than or equal to 1, and K+L=N.

[0161] It can be understood that by sampling proportion p% from the first training image set D 1：t The sample defect images in the training set are sampled to obtain a first sample image set. The first sample image set and the second training image set are combined to form new training data. K sample defect images are taken from the first sample image set, and L sample defect images are taken from the second training image set. Preferably, K and L are controlled to be as equal as possible, that is:

[0162] in, Indicates rounding down.

[0163] Take the first sample image set A picture is taken from the second training image set merged to form new training data D′ t+1 .

[0164] S128. Iteratively train the initial detection model based on the K first sample defect images in the first sampling image set, the L second sample defect images in the second training image set, and the first weight parameter.

[0165] S129: Delete the K first sample defect images from the first sampling image set to obtain an updated first sampling image set.

[0166] The updated first sampling image set is used as the sampling set of K sample defect images during iterative training.

[0167] It can be understood that in order to use different K sample defect images in the first sampling image set in each iterative training, it is necessary to delete the K sample defect images taken away from the first sampling image set each time to update the first sampling image set.

[0168] For ease of understanding, assume that the second training image set consisting of the full amount of new data of the task at time t+1 is D t+1 , and the first training image set composed of the full amount of historical training data at time t is D 1：t In related technologies, the full data model is to use Dt+1 +D 1：t The new model obtained by training data has good performance, but the training cost is high; the ordinary fine-tuning model is to use D t+1 The trained model has low training cost but the worst performance, and catastrophic forgetting occurs for old training data. In order to enable only a small amount of historical training data to be added during the iteration process of new data, thereby reducing training costs and achieving training results that are basically equivalent to using the full amount of data, the embodiment of the present application adopts a strategy of dynamic balanced sampling of old training data. Specifically, in each training epoch, this method randomly samples p% of the sample defect data in the first training image set and the sample defect data in the second training image set to form the training data of the current epoch. Thereafter, in each iteration, a first sampling image set is sampled from the sample defect data in the first training image set, and a 1:1 ratio of historical sample defect images and newly added sample defect images are sampled from the first sampling image set and the second training image set to form a batch for model parameter updating. Random sampling in each epoch can ensure that the historical sample defect images used in each training are in a dynamic change process, so as to maintain the capabilities or knowledge that the model has learned under the old task data as much as possible after the entire task training is completed (usually dozens of epochs are required). 1:1 balanced sampling in each batch helps to further improve model performance and prevent the model from performing well only on new tasks or only on old tasks. The ratio of historical sample defect images to new sample defect images in each batch is also changed to 1:2 or 2:1, but the effect is far inferior to 1:1 balanced sampling. For the full data model, since each batch of data is randomly composed of historical data and incremental data, the ratio of historical data to incremental data in a batch is close to its overall data ratio. Usually the incremental data is much smaller than the historical data, which to a certain extent limits the performance of the model trained in this way on new tasks. On the contrary, the ordinary fine-tuning model, since only the newly added data is used, the model will completely shift to the newly added data, resulting in catastrophic forgetting of the old tasks. This method uses a simple way to effectively avoid the problems existing in full data training and ordinary model fine-tuning, while greatly reducing the training cost.

[0169] The method provided in the embodiment of the present application can greatly speed up the process of iterative updating of the model (reduce costs) while maintaining an effect comparable to that of the full data model. With the accumulation of business data, the time required for training the full data model continues to increase, and it may even take several days to complete a training session. However, the incremental training mode proposed by this method can maintain the training time at a relatively low level, greatly reducing the GPU card time. This is because the proposed method randomly selects part of the historical data for training in each epoch, ensuring the dynamic variability of the historical data involved in the training. At the same time, the ratio of new and old data is maintained at 1:1 in the training of each batch, and the changes in parameters that are very important for the old tasks are limited by the dynamic weight restriction method, so that the model will not suffer from catastrophic forgetting when using only a small amount of historical data, meeting the project requirements while greatly reducing the iteration cost.

[0170] In an optional embodiment of the model training method provided in the embodiment corresponding to FIG. 2 of the present application, please refer to FIG. 9 , step S111 to step S112 are further included before step S120. Specifically:

[0171] S111. Obtain a sampling ratio of a first training image set.

[0172] S112 : Sampling the sample defect images in the first training image set according to a sampling ratio to obtain a first sampled image set.

[0173] It is understood that the sampling ratio of the first training image set is a preset sampling ratio. The purpose is to select a portion of sample defect images from the first training image set for subsequent training of the initial detection model. The sampling ratio of the first training image set is obtained, and the sample defect images in the first training image set are sampled according to this ratio. The purpose of sampling is to reduce the size of the training data while retaining important information.

[0174] The method provided in the embodiment of the present application uses the sample defect images in the first sampling image set obtained after sampling the first training image set and the sample defect images in the second training image set as training samples of the initial detection model to further train the initial detection model. Compared with using the full first training image set and the second training image set for training, the training of the initial model is reduced, which saves time and reduces the consumption of GPU cards.

[0175] The following will introduce the defect detection method in this application from the perspective of the server. Referring to Figure 10, the defect detection method provided by the embodiment of this application includes: Steps S210 to S240. Specifically:

[0176] S210: Acquire an image to be detected.

[0177] It can be understood that an image to be inspected that needs to be inspected for defects is obtained.

[0178] S220: Input the image to be detected into the optimized detection model.

[0179] The optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using the above-mentioned model training method.

[0180] It is understood that the acquired image to be inspected is input into the optimized inspection model. This optimized inspection model is obtained through the aforementioned model training method and includes a feature extraction network and a classification network. The feature extraction network is used to extract features from the input image, while the classification network is used to classify defects based on the extracted image features.

[0181] S230 , extracting features from the image to be detected based on the feature extraction network in the optimized detection model to obtain features of the image to be detected.

[0182] It is understandable that the feature extraction network in the optimized detection model processes the image to be inspected and extracts key features from the image. These features can be information such as color, shape, texture, etc., which can represent the characteristics of defects in the image.

[0183] S240 , performing defect classification on features of the image to be detected based on a classification network to obtain predicted defect information of the image to be detected.

[0184] The predicted defect information is used to represent the classification result of whether the image to be detected contains defects.

[0185] It is understood that the classification network will classify defects based on the extracted features of the image to be inspected and output predicted defect information. The predicted defect information is used to indicate whether the image to be inspected contains defects and the type of defects.

[0186] The method provided in the embodiments of the present application inputs the image to be inspected into an optimized inspection model, processes and analyzes it using the model's feature extraction network and classification network, and ultimately obtains predicted defect information for the image to be inspected. Using the trained optimized inspection model, defect detection can be performed quickly and automatically on the image to be inspected, improving inspection efficiency and accuracy. Furthermore, the optimized inspection model can be continuously trained and updated to adapt to different types of defects and images, thereby improving the model's generalization capabilities.

[0187] The model training device in this application is described in detail below, with reference to Figure 11. Figure 11 is a schematic diagram of an embodiment of a model training device 10 in this application embodiment, the model training device 10 includes: a training image and initial model acquisition module 110 and a model training module 120; specifically:

[0188] A training image and initial model acquisition module 110 is configured to acquire a first training image set, a second training image set, and an initial detection model, wherein the initial detection model is obtained by training based on the first training image set, the first training image set includes a first sample defect image, the second training image set includes a second sample defect image, and the second sample defect image is different from the first sample defect image; the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameters are used to represent the importance of the multiple model parameters of the initial detection model;

[0189] The model training module 120 is used to train the initial detection model based on the first sampling image set, the second training image set and the first weight parameter to adjust multiple model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes a portion of the first sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the model parameters in the initial detection model.

[0190] The model training device provided in the embodiment of the present application trains the initial detection model by using a partially trained first training image set and a fully newly added second training image set. Compared with using the full amount of historical training data and the fully newly added training data to train the initial model, it saves time and reduces the consumption of GPU cards. By adaptively evaluating the corresponding first weights of the model parameters, the update of the model parameters based on historical training data is limited. Compared with the problem of knowledge forgetting caused by fine-tuning the model using only the full amount of newly added data, the learning ability of the model is improved, and the recognition effect and accuracy of the model for defect recognition are enhanced.

[0191] In another implementation of the embodiment of the present application, the model training module 120 is further configured to:

[0192] Obtaining model parameters of a basic detection model, wherein the initial detection model is obtained by adjusting the model parameters of the basic detection model;

[0193] Constructing an objective function of the initial detection model according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set;

[0194] Based on the objective function of the initial detection model, the initial detection model is trained to adjust model parameters of the initial detection model.

[0195] The device provided in the embodiment of the present application performs more precise adjustment and optimization of the initial detection model through the objective function. The calculation of the objective function provides a quantitative evaluation of the performance of the model on a specific data set. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation device based on the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine adjustment and optimization of these steps are to enable the model to better adapt to data changes and needs in practical applications, and to improve its performance and generalization ability.

[0196] In another implementation of the embodiment of the present application, the model training module 120 is further configured to:

[0197] Generate an incremental data objective function of an initial detection model as a first optimization item according to the first sampling image set and the second training image set;

[0198] Calculating parameter differences between model parameters of the initial detection model and model parameters of the basic detection model;

[0199] generating a second optimization item according to the first weight parameter, the parameter difference, and a hyperparameter, wherein the second optimization item is used to limit the adjustment range of important parameters during training, the important parameters are identified by the first weight parameter, and the hyperparameter is used to identify the influence of the second optimization item when training the initial detection model;

[0200] An objective function of the initial detection model is constructed according to the first optimization item and the second optimization item.

[0201] The apparatus provided in the embodiments of the present application calculates the objective function of the initial detection model by processing sampled data, comparing model parameters, and applying weights. This objective function can be used as an indicator to evaluate model performance, helping us understand the model's performance on a specific dataset and guiding further optimization and improvement. This calculation process can help us more accurately evaluate and improve the detection model to enhance its performance and accuracy.

[0202] In another implementation of the embodiment of the present application, the model training module 120 is further configured to:

[0203] Obtaining a clipping threshold, wherein the clipping threshold is used to represent a gradient threshold when updating parameters of the initial detection model, so as to limit a gradient range when updating parameters of the initial detection model;

[0204] An objective function of the initial detection model is constructed according to the cropping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set.

[0205] In another implementation of the embodiment of the present application, the model training module 120 is further configured to:

[0206] Calculating a root form of a first weight parameter, and calculating a parameter difference between model parameters of the initial detection model and model parameters of the basic detection model, and multiplying the root form of the first weight parameter by the parameter difference to obtain a first calculated value;

[0207] If the first calculated value is less than or equal to the cropping threshold, calculating the objective function of the initial detection model according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set;

[0208] If the first calculated value is greater than the clipping threshold, the objective function of the initial detection model is calculated based on the clipping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set and the second training image set.

[0209] The device provided in the embodiment of the present application performs more precise adjustments and optimizations on the optimized detection model through the calculation of the objective function. The calculation of the objective function provides a quantitative evaluation of the performance of the model on a specific data set. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation device based on the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine adjustment and optimization of these steps are to enable the model to better adapt to data changes and needs in practical applications, and to improve its performance and generalization ability.

[0210] In another implementation of the embodiment of the present application, referring to FIG. 12 , the model training device 10 further includes a weight calculation module 130 . Specifically, the weight calculation module 130 is configured to:

[0211] Based on the objective function of the initial detection model and the first weight parameter set, the second weight parameter set of the optimized detection model is calculated, wherein the second weight parameter set includes second weight parameters of multiple model parameters in the optimized detection model, the second weight parameters are used to characterize the importance of the model parameters of the optimized detection model, and the second weight parameter set is used to limit the adjustment range of the model parameters in the optimized detection model.

[0212] The device provided in the embodiment of the present application, by determining the second weight, performs more precise adjustment and optimization on the optimized detection model. The calculation of the second weight adjusts the parameter importance of the model according to the evaluation results. Such a process can help us find a more optimal model configuration and improve the accuracy and reliability of the model. In practical applications, it is necessary to select a suitable objective function and weight calculation device according to the specific problem and data characteristics. At the same time, it is also necessary to conduct multiple experiments and verifications to find the optimal parameter settings and model structure. The fine adjustment and optimization of these steps are to enable the model to better adapt to the changes and needs of the data in practical applications, and to improve its performance and generalization ability.

[0213] In another implementation of the embodiment of the present application, referring to FIG. 12 , the weight calculation module 130 is further configured to:

[0214] performing multiple loss calculations on the objective function of the initial detection model to obtain multiple third weight parameter sets, wherein the loss calculations are performed by performing derivative calculations on the objective function of the initial detection model, and the third weight parameter sets include multiple third weight parameters, each of which is used to represent the importance of the model parameters of the initial detection model for correctly identifying the second training image set;

[0215] The plurality of third weight parameter sets are added to the first weight parameter set to obtain a second weight parameter set corresponding to the optimized detection model.

[0216] The device provided in the embodiment of the present application can more accurately evaluate the importance of model parameters for optimizing the detection model, which helps to improve the performance and generalization ability of the model.

[0217] In another implementation of the embodiment of the present application, referring to FIG11 , the model training module 120 is further configured to:

[0218] Get the number of images corresponding to the training batch when training the initial detection model, where the number of images is N, and N is an integer greater than 1;

[0219] Select K first sample defect images from the first sampling image set, and select L second sample defect images from the second training image set, where K and L are both integers greater than or equal to 1, and K+L=N;

[0220] The initial detection model is iteratively trained based on K first sample defect images in the first sampling image set, L second sample defect images in the second training image set, and a first weight parameter.

[0221] The K first sample defect images are deleted from the first sampling image set to obtain an updated first sampling image set, wherein the updated first sampling image set is used as the sampling set of the K first sample defect images during iterative training.

[0222] The device provided in the embodiment of the present application can greatly speed up the process of iterative updating of the model (reduce costs) while maintaining an effect comparable to that of the full data model. With the accumulation of business data, the time required for training the full data model continues to increase, and it may even take several days to complete a training session. However, the incremental training mode proposed by this device can maintain the training time at a relatively low level, greatly reducing the GPU card time. This is because the proposed device randomly selects part of the historical data for training in each epoch, ensuring the dynamic variability of the historical data involved in the training. At the same time, the ratio of new and old data is maintained at 1:1 in the training of each batch, and the device with dynamic weight limitation is used to limit the changes in parameters that are very important for the old tasks, so that the model will not suffer from catastrophic forgetting when using only a small amount of historical data, meeting the project requirements while greatly reducing the iteration cost.

[0223] In another implementation of the embodiment of the present application, referring to FIG. 13 , the model training device 10 further includes an image sampling module 111 . Specifically, the image sampling module 111 is configured to:

[0224] Obtaining a sampling ratio of a first training image set;

[0225] The sample defect images in the first training image set are sampled according to a sampling ratio to obtain a first sampled image set.

[0226] The device provided in the embodiment of the present application further trains the initial detection model by using the sample defect images in the first sampling image set obtained after sampling the first training image set and the sample defect images in the second training image set as training samples of the initial detection model. Compared with using the full first training image set and the second training image set for training, the training of the initial model is reduced, which saves time and reduces the consumption of GPU cards.

[0227] The defect detection device in the present application is described in detail below, with reference to FIG14 . FIG14 is a schematic diagram of an embodiment of a defect detection device 20 in the present application. The defect detection device 20 includes: an image acquisition module 210 to be detected, an image input module 220 to be detected, an image feature extraction module 230 to be detected, and an image feature classification module 240 to be detected; specifically:

[0228] The image acquisition module 210 is used to acquire the image to be detected;

[0229] An image input module 220 for inputting the image to be detected into an optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using the above-mentioned model training device;

[0230] The image feature extraction module 230 is used to extract features of the image to be detected based on the feature extraction network in the optimized detection model to obtain features of the image to be detected;

[0231] The image feature classification module 240 is used to classify the features of the image to be detected based on the classification network to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to represent the classification result of whether the image to be detected contains defects.

[0232] The device provided in the embodiments of the present application inputs the image to be inspected into an optimized inspection model, processes and analyzes it using the model's feature extraction network and classification network, and ultimately obtains predicted defect information for the image to be inspected. Using the trained optimized inspection model, defects in the image to be inspected can be quickly and automatically detected, improving inspection efficiency and accuracy. Furthermore, the optimized inspection model can be continuously trained and updated to adapt to different types of defects and images, improving the model's generalization capabilities.

[0233] Figure 15 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more massive storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0234] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSDTM etc.

[0235] The steps executed by the server in the above embodiment may be based on the server structure shown in FIG15 .

[0236] In addition, an embodiment of the present application further provides a storage medium, which is used to store a computer program, and the computer program is used to execute the method provided by the above embodiment.

[0237] An embodiment of the present application further provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the method provided in the above embodiment.

[0238] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0239] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0240] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0241] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0243] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, the method being performed by a computer device, the method comprising: Obtaining a first training image set, a second training image set, and an initial detection model, wherein the initial detection model is trained based on the first training image set, the first training image set includes a first sample defect image, the second training image set includes a second sample defect image, and the second sample defect image is different from the first sample defect image, and the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameter is used to represent the importance of the multiple model parameters of the initial detection model; The initial detection model is trained based on a first sampling image set, the second training image set and the first weight parameter to adjust the multiple model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes part of the first sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the multiple model parameters in the initial detection model.

2. The model training method according to claim 1, wherein the training of the initial detection model based on the first sampling image set, the second training image set, and the first weight parameter to adjust the multiple model parameters of the initial detection model comprises: Acquiring model parameters of a basic detection model, wherein the initial detection model is obtained by adjusting the model parameters of the basic detection model; An objective function of the initial detection model is constructed according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set, wherein: Based on the objective function of the initial detection model, the initial detection model is trained to adjust the multiple model parameters of the initial detection model.

3. The model training method according to claim 2, wherein constructing the objective function of the initial detection model based on the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set comprises: generating, according to the first sampling image set and the second training image set, an incremental data objective function of the initial detection model as a first optimization item; Calculating parameter differences between model parameters of the initial detection model and model parameters of the basic detection model; generating a second optimization item according to the first weight parameter, the parameter difference, and a hyperparameter, wherein the second optimization item is used to limit the adjustment range of important parameters during training, the important parameters are identified by the first weight parameter, and the hyperparameter is used to identify the influence of the second optimization item when training the initial detection model; An objective function of the initial detection model is constructed according to the first optimization item and the second optimization item.

4. The model training method according to claim 2, wherein constructing the objective function of the initial detection model based on the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set comprises: Obtaining a clipping threshold, wherein the clipping threshold is used to represent a gradient threshold when updating parameters of the initial detection model, so as to limit a gradient range when updating parameters of the initial detection model; An objective function of the initial detection model is constructed according to the cropping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set.

5. The model training method according to claim 4, wherein constructing the objective function of the initial detection model based on the cropping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set comprises: Calculating a radical of the first weight parameter, and calculating a parameter difference between a model parameter of the initial detection model and a model parameter of the basic detection model, and multiplying the radical of the first weight parameter by the parameter difference to obtain a first calculated value; If the first calculated value is less than or equal to the cropping threshold, calculating the objective function of the initial detection model according to the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set, and the second training image set; If the first calculated value is greater than the cropping threshold, the objective function of the initial detection model is calculated based on the cropping threshold, the first weight parameter, the model parameters of the initial detection model, the model parameters of the basic detection model, the first sampling image set and the second training image set.

6. The model training method of claim 1, wherein after training the initial detection model based on the first sample image set, the second training image set, and the first weight to adjust the model parameters of the initial detection model and obtain the trained optimized detection model, the method further comprises: Based on the objective function of the initial detection model and the first weight parameter set, the second weight parameter set of the optimized detection model is calculated, wherein the second weight parameter set includes second weight parameters of multiple model parameters in the optimized detection model, the second weight parameters are used to characterize the importance of the model parameters of the optimized detection model, and the second weight parameter set is used to limit the adjustment range of the model parameters in the optimized detection model.

7. The model training method according to claim 6, wherein the step of calculating the second set of weight parameters of the optimized detection model based on the objective function of the initial detection model and the first set of weight parameters comprises: performing multiple loss calculations on the objective function of the initial detection model to obtain multiple third weight parameter sets, wherein the loss calculations are performed by performing derivative calculations on the objective function of the initial detection model, the third weight parameter sets including multiple third weight parameters, the third weight parameters being used to represent the importance of the model parameters of the initial detection model for correctly identifying the second training image set; The multiple third weight parameter sets are added to the first weight parameter set to obtain a second weight parameter set corresponding to the optimized detection model.

8. The model training method according to claim 1, wherein the training of the initial detection model based on the first sampling image set, the second training image set, and the first weight comprises: Obtaining the number of images corresponding to a training batch when training the initial detection model, wherein the number of images is N, and N is an integer greater than 1; Select K first sample defect images from the first sampling image set, and select L second sample defect images from the second training image set, where K and L are both integers greater than or equal to 1, and K+L=N; The initial detection model is iteratively trained based on the K first sample defect images in the first sampling image set, the L second sample defect images in the second training image set, and the first weight parameter.

9. The model training method according to claim 8, further comprising: after extracting K first sample defect images from the first sampling image set; The K first sample defect images are deleted from the first sampling image set to obtain an updated first sampling image set, wherein the updated first sampling image set is used as the sampling set of the K first sample defect images during iterative training.

10. The model training method of claim 1, wherein before training the initial detection model based on the first sampling image set, the second training image set, and the first weight to adjust the multiple model parameters of the initial detection model to obtain the trained optimized detection model, the method further comprises: Obtaining a sampling ratio of a first training image set; The sample defect images in the first training image set are sampled according to the sampling ratio to obtain a first sampled image set.

11. A defect detection method, the method being performed by, the method comprising: Obtain the image to be detected; Inputting the image to be detected into an optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using the model training method according to any one of claims 1 to 10; Extracting features of the image to be detected based on the feature extraction network in the optimized detection model to obtain features of the image to be detected; Defect classification is performed on the features of the image to be detected based on the classification network to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to represent a classification result of whether the image to be detected contains defects.

12. A model training device comprising: a training image and initial model acquisition module, configured to acquire a first training image set, a second training image set, and an initial detection model, wherein the initial detection model is obtained by training based on the first training image set, the first training image set includes a first sample defect image, the second training image set includes a second sample defect image, and the second sample defect image is different from the first sample defect image; the initial detection model includes a first weight parameter set, the first weight parameter set includes first weight parameters of multiple model parameters in the initial detection model, and the first weight parameter is used to represent the importance of the multiple model parameters of the initial detection model; A model training module is used to train the initial detection model based on a first sampling image set, the second training image set and the first weight parameter to adjust the multiple model parameters of the initial detection model to obtain a trained optimized detection model, wherein the first sampling image set includes part of the first sample defect images in the first training image set, and the first weight parameter set is used to limit the adjustment range of the multiple model parameters in the initial detection model.

13. A defect detection device comprising: An image acquisition module to be detected, used to acquire the image to be detected; An image input module for detecting, configured to input the image to be detected into an optimized detection model, wherein the optimized detection model includes a feature extraction network and a classification network, and the optimized detection model is obtained using the model training method according to any one of claims 1 to 10; A feature extraction module for the image to be detected, configured to extract features of the image to be detected based on the feature extraction network in the optimized detection model to obtain features of the image to be detected; The feature classification module of the image to be detected is used to perform defect classification on the features of the image to be detected based on the classification network to obtain predicted defect information of the image to be detected, wherein the predicted defect information is used to represent the classification result of whether the image to be detected contains defects.

14. A computer device comprising: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is configured to execute the program in the memory, including executing the model training method according to any one of claims 1 to 10 or the defect detection method according to claim 11; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

15. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the model training method according to any one of claims 1 to 10 or the defect detection method according to claim 11. 16 . A computer program product, comprising a computer program, wherein a processor executes the model training method according to claim 1 or the defect detection method according to claim 11 .

Images