Incremental training method of image classification model and image classification method

Abstract

An incremental training method of an image classification model and an image classification method, wherein the incremental training method is used for incremental training of an initial image classification model, and the incremental training comprises multiple training stages, each training stage comprising: obtaining a training image set, the training image set comprising a historical training image set and a new training image set; training the image classification model using the training image set; determining whether a preset training stop condition is reached, if not, evaluating a forgetting degree of the trained image classification model to old knowledge, determining whether the forgetting degree of the trained image classification model to the old knowledge exceeds a preset limit, if yes, adjusting a weight of a regularization term in a loss function to enhance a regularization constraint, and continuing the training of the next training stage using the adjusted weight of the regularization term; thus, the image classification model can effectively maintain the memory of the old knowledge while continuously learning new knowledge, and the problem of catastrophic forgetting can be alleviated.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to an incremental training method for image classification models and an image classification method. Background Technology

[0002] With the development of machine vision technology, machine learning is being used more and more widely in image classification tasks, such as identifying the types of animals or objects in images, detecting the types of defects in objects in images, and face recognition.

[0003] Image classification technology based on machine learning requires training data to train a machine learning model, enabling it to learn the ability to perform image classification tasks. However, training data is often difficult to collect completely at once in the initial stage and must be accumulated gradually. Considering the limitations of training time and computing resources, incremental training is a common strategy when new training data is collected and used to retrain the image classification model. However, incremental training is generally plagued by the catastrophic forgetting problem, where the model easily forgets previously learned knowledge when learning new knowledge. Summary of the Invention

[0004] This invention provides an incremental training method for an image classification model, an image classification method, and a computer-readable storage medium, mainly addressing the catastrophic forgetting problem in the incremental training process of image classification models.

[0005] According to a first aspect, one embodiment provides an incremental training method for an image classification model, used to incrementally train an initial image classification model, wherein the incremental training includes multiple training stages, each training stage including:

[0006] Obtain a training image set, which includes a historical training image set and a newly added training image set;

[0007] The image classification model is trained using the training image set, wherein the loss function used when training the image classification model includes one or more regularization terms;

[0008] Determine whether the preset training stop condition has been met. If not, assess the degree to which the trained image classification model has forgotten the old knowledge. Determine whether the degree to which the trained image classification model has forgotten the old knowledge exceeds the preset limit. If so, adjust the weight of the regularization term to enhance the regularization constraint on the image classification model, and continue training in the next training stage using the adjusted weight of the regularization term.

[0009] In some embodiments, evaluating the degree to which the trained image classification model has forgotten old knowledge, and determining whether the degree to which the trained image classification model has forgotten old knowledge exceeds a preset limit, includes:

[0010] The forgetting rate of the trained image classification model is evaluated to determine whether the forgetting rate of the trained image classification model is greater than a preset forgetting rate threshold. If so, it is determined that the degree to which the trained image classification model has forgotten old knowledge exceeds the preset limit.

[0011] In some embodiments, evaluating the degree to which the trained image classification model has forgotten old knowledge, and determining whether the degree to which the trained image classification model has forgotten old knowledge exceeds a preset limit, includes:

[0012] The forgetting rate and the rate of change of the forgetting rate of the trained image classification model are evaluated. It is determined whether the forgetting rate of the trained image classification model is greater than a preset forgetting rate threshold or whether the rate of change of the forgetting rate is greater than a preset first rate of change threshold. If so, it is determined that the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit.

[0013] In some embodiments, adjusting the weights of the regularization term to enhance the regularization constraints on the image classification model includes:

[0014] The weights of most or all of the regularization terms may be increased by a fixed amount, or the weights of most or all of the regularization terms may be increased by an amount proportional to the rate of change of the forgetting rate.

[0015] In some embodiments, the forgetting rate threshold and the first rate of change threshold are determined by the following steps:

[0016] Using fixed regularization term weights, the initial image classification model is incrementally trained, and the forgetting rate of the image classification model after each training stage is recorded to obtain the forgetting rate change curve.

[0017] Determine the location in the forgetting rate change curve where catastrophic forgetting occurs;

[0018] For the portion of the forgetting rate change curve before the location of catastrophic forgetting, the average or median forgetting rate is obtained as the forgetting rate threshold, and the average or median forgetting rate change rate is obtained as the first change rate threshold.

[0019] In some embodiments, determining the location of catastrophic forgetting in the forgetting rate change curve includes:

[0020] Set a sliding window of size N, slide the sliding window on the forgetting rate change curve, and at each position, determine whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit. If so, determine that position as the position where catastrophic forgetting occurs in the forgetting rate change curve, where N is an integer not less than 2.

[0021] In some embodiments, determining whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit includes:

[0022] Calculate the difference between the last forgetting rate and the first forgetting rate within the sliding window. If the difference between the last forgetting rate and the first forgetting rate is greater than a preset first difference threshold, then determine that the increase in the forgetting rate within the sliding window exceeds a preset increase limit.

[0023] Alternatively, calculate the difference between the last forgetting rate and the first forgetting rate within the sliding window, as well as the standard deviation or range of the forgetting rate within the sliding window. If the difference between the last forgetting rate and the first forgetting rate within the sliding window is greater than the first difference threshold, and the standard deviation of the forgetting rate within the sliding window is greater than a preset first standard deviation threshold or the range is greater than a preset first range threshold, then determine that the increase in the forgetting rate within the sliding window exceeds a preset increase limit.

[0024] In some embodiments, evaluating the degree to which the trained image classification model has forgotten old knowledge, and determining whether the degree to which the trained image classification model has forgotten old knowledge exceeds a preset limit, includes:

[0025] The performance metrics of the trained image classification model are evaluated to determine whether the performance metrics of the trained image classification model are less than a preset performance metric threshold. If so, it is determined that the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit.

[0026] In some embodiments, evaluating the degree to which the trained image classification model has forgotten old knowledge, and determining whether the degree to which the trained image classification model has forgotten old knowledge exceeds a preset limit, includes:

[0027] The performance metrics and rate of change of the trained image classification model are evaluated. It is determined whether the performance metrics of the trained image classification model are less than a preset performance metric threshold or whether the rate of change of the performance metric is greater than a preset second rate of change threshold. If so, it is determined that the degree of forgetting of old knowledge by the trained image classification model exceeds a preset limit.

[0028] In some embodiments, adjusting the weights of the regularization term to enhance the regularization constraints on the image classification model includes:

[0029] The weights of most or all of the regularization terms may be increased by a fixed amount, or the weights of most or all of the regularization terms may be increased by an amount proportional to the rate of change of the performance metric.

[0030] In some embodiments, the performance metric threshold and the second rate of change threshold are determined through the following steps:

[0031] Using fixed regularization term weights, the initial image classification model is incrementally trained, and the performance index of the image classification model after each training stage is recorded to obtain the performance index change curve.

[0032] Determine the location in the performance index change curve where catastrophic amnesia occurs;

[0033] For the portion of the performance index change curve before the point of catastrophic forgetting, the average or median value of its performance index is obtained as the performance index threshold, and the average or median value of its performance index change rate is obtained as the second change rate threshold.

[0034] In some embodiments, the newly acquired training image set in the step of acquiring the training image set is the same as the newly acquired training image set in the corresponding training stage of the step of incrementally training the initial image classification model using the weights of a fixed regularization term.

[0035] In some embodiments, determining the location of catastrophic forgetting in the performance index change curve includes:

[0036] Set a sliding window of size M, slide the sliding window on the performance index change curve, and at each position, determine whether the decrease of the performance index within the sliding window exceeds the preset decrease limit. If so, determine that position as the position where catastrophic forgetting occurs in the forgetting rate change curve, where M is an integer not less than 2.

[0037] In some embodiments, determining whether the decrease in the performance metric within the sliding window exceeds a preset decrease limit includes:

[0038] Calculate the difference between the last performance indicator and the first performance indicator within the sliding window. If the difference between the last performance indicator and the first performance indicator is greater than a preset second difference threshold, then determine that the decrease in the performance indicator within the sliding window exceeds a preset decrease limit.

[0039] Alternatively, calculate the difference between the last performance indicator and the first performance indicator within the sliding window, as well as the standard deviation or range of the performance indicators within the sliding window. If the difference between the last performance indicator and the first performance indicator within the sliding window is greater than the second difference threshold, and the standard deviation of the performance indicators within the sliding window is greater than the preset second standard deviation threshold or the range is greater than the preset second range threshold, then determine that the decline in the performance indicators within the sliding window exceeds the preset decline limit.

[0040] Incremental training methods in some embodiments further include: if the degree of forgetting of old knowledge by the trained image classification model does not exceed a preset limit, then the weight of the regularization term is kept unchanged or the weight of the regularization term is increased, wherein the increase in the weight of the regularization term is less than the increase in the weight of the regularization term when the degree of forgetting of old knowledge by the trained image classification model exceeds a preset limit.

[0041] According to a second aspect, one embodiment provides an image classification method, comprising:

[0042] Obtain the image to be classified;

[0043] The image to be classified is input into the image classification model trained by the incremental training method described in the first aspect above, so as to obtain the category of the image to be classified.

[0044] According to a third aspect, one embodiment provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the incremental training method described in the first aspect or the image classification method described in the second aspect.

[0045] In embodiments of the present invention, in each training stage of incremental training of the image classification model, the image classification model is trained using a training image set. The loss function used during training includes one or more regularization terms. After training, it is determined whether a preset training stopping condition has been met. If not, the degree of forgetting of old knowledge by the trained image classification model is evaluated. It is determined whether the degree of forgetting of old knowledge by the trained image classification model exceeds a preset limit. If so, the weights of the regularization terms are adjusted to strengthen the regularization constraint on the image classification model, and the adjusted weights of the regularization terms are used to continue training in the next training stage. This allows for tracking the degree of forgetting of the image classification model during incremental training. When the degree of forgetting is too high, the weights of the regularization terms are adjusted in a timely manner. Through the constraint effect of the regularization terms, the image classification model can effectively maintain its memory of old knowledge while continuously learning new knowledge, avoiding catastrophic forgetting problems during incremental training and ensuring the stability and accuracy of the image classification model in image classification tasks. Attached Figure Description

[0046] Figure 1 A flowchart illustrating the incremental training method for image classification models in some embodiments;

[0047] Figure 2 This is a flowchart for determining the forgetting rate threshold in some embodiments;

[0048] Figure 3 This is a flowchart for determining performance metric thresholds in some embodiments;

[0049] Figure 4 This is a flowchart of an image classification method in some embodiments. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0051] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0052] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. "Multiple" refers to two or more. Unless otherwise specified, the terms "connection" and "linkage" used in this application include both direct and indirect connections (linkages).

[0053] Incremental learning refers to a learning system's ability to continuously learn new knowledge from new samples while retaining most of the previously learned knowledge. This learning method is very similar to the human learning pattern, because people learn and receive new things every day during their growth process, and generally do not forget the knowledge they have already learned.

[0054] Incremental learning, introduced into image classification, allows models to continuously receive new image data and learn from it, while retaining their memory of previously learned image data. For example, in wafer defect detection, incremental learning allows models to continuously receive new defect sample images and learn new defect types, while maintaining their memory of previously learned defect sample images.

[0055] Incremental learning typically involves incremental training based on an initial image classification model, which is a pre-trained model with initial parameters. When a new batch of image data is collected, this new data, along with some older image data, is used to perform mixed training on the image classification model, allowing it to learn new knowledge; this constitutes one training phase. A complete incremental training process includes multiple training phases.

[0056] Typically, the parameters of an image classification model may change due to the arrival of new image data, leading to a decline in image classification performance, a phenomenon known as "catastrophic forgetting." For example, in a wafer defect classification task, the parameters of a defect detection model may change after training with new defect sample images, resulting in a decrease in the performance of identifying wafer defects.

[0057] This invention aims to optimize the incremental training process, focusing on improving the ability of image classification models to retain old knowledge. It proposes an incremental training method for image classification models that can track the degree of forgetting in the incremental training process and adjust the weight of the regularization term in a timely and adaptive manner. With the constraint of the regularization term, the model can effectively retain the old knowledge it has learned while continuously learning new knowledge, maximizing the retention of old knowledge and ensuring the overall performance and stability of the model in image classification tasks.

[0058] This invention provides an incremental training method for an image classification model, used for incremental training of an initial image classification model, which can be a machine learning model such as a convolutional neural network. Please refer to [link / reference]. Figure 1 Each training phase of the incremental training includes steps 100 to 500, which are described in detail below.

[0059] Step 100: Obtain the training image set.

[0060] The training image set includes the historical training image set (referred to as "old images") and the newly added training image set (referred to as "new images"). The historical training image set is the set of sample images that have been used to train the image classification model, while the newly added training image set is the set of sample images newly acquired in the current training phase.

[0061] Step 200: Train the image classification model using the training image set, wherein the loss function used when training the image classification model includes one or more regularization terms.

[0062] The loss function used in incremental training includes one or more regularization terms to regularize the image classification model, protecting the model's learned knowledge from being overwritten by new knowledge. Regularization terms can be L2 norm regularization terms, weight regularization terms, or data regularization terms, etc. Initial weights for the regularization terms can be set before training begins.

[0063] In some embodiments, old and new images can be input into the image classification model for training in a certain ratio, such as an old image to new image ratio of 7:3.

[0064] Step 300: Determine whether the preset training stop condition has been met. If not, proceed to step 400.

[0065] Training will stop when a preset training stopping condition is met, such as reaching a preset maximum number of training stages or being unable to acquire a new set of sample images.

[0066] Step 400: Evaluate the degree of forgetting of old knowledge by the trained image classification model, and determine whether the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit. If so, proceed to step 500.

[0067] Step 500: Adjust the weights of the regularization term to enhance the regularization constraints on the image classification model, so that the adjusted weights of the regularization term can be used to continue training in the next training phase. For example, increase the weights of the regularization term.

[0068] This invention evaluates the forgetting level of the image classification model after each training stage, tracks the forgetting level of the image classification model during incremental training, and adjusts the weight of the regularization term in a timely manner when the forgetting level is too high to enhance the regularization constraint on the image classification model. This enhances the memory of old knowledge, allowing the image classification model to effectively absorb new knowledge and firmly remember old knowledge during incremental training, avoiding catastrophic forgetting problems, and ensuring the overall performance, stability, and generalization ability of the image classification model in image classification tasks.

[0069] In some embodiments, evaluating the degree to which the trained image classification model has forgotten prior knowledge, and determining whether the degree to which the trained image classification model has forgotten prior knowledge exceeds a preset limit, includes: evaluating the forgetting rate of the trained image classification model, determining whether the forgetting rate of the trained image classification model is greater than a preset forgetting rate threshold; if so, it is determined that the degree to which the trained image classification model has forgotten prior knowledge exceeds the preset limit. In some embodiments, if the forgetting rate is not greater than the preset forgetting rate threshold, it is considered that the degree to which the trained image classification model has forgotten prior knowledge has not exceeded the preset limit.

[0070] The forgetting rate can be calculated using the following formula:

[0071] Forgetting rate = (Performance index of the image classification model before training on old images – Performance index of the image classification model after training on old images) / Performance index of the image classification model before training on old images × 100%.

[0072] Performance metrics can include F1-score, accuracy, recall, etc.

[0073] In some embodiments, evaluating the degree to which the trained image classification model has forgotten prior knowledge, and determining whether the degree to which the trained image classification model has forgotten prior knowledge exceeds a preset limit, includes: evaluating the forgetting rate and the rate of change of the forgetting rate of the trained image classification model, and determining whether the forgetting rate of the trained image classification model is greater than a preset forgetting rate threshold or whether the rate of change of the forgetting rate is greater than a preset first rate of change threshold. If so, it is determined that the degree to which the trained image classification model has forgotten prior knowledge exceeds the preset limit. In some embodiments, if the forgetting rate is not greater than the preset forgetting rate threshold and the rate of change of the forgetting rate is not greater than the preset first rate of change threshold, it is considered that the degree to which the trained image classification model has forgotten prior knowledge has not exceeded the preset limit.

[0074] The rate of change in the forgetting rate can be calculated using the following formula:

[0075] Forgetting rate = (Forgetting rate in this training phase – Forgetting rate in the previous training phase) / Forgetting rate in the previous training phase × 100%.

[0076] When the rate of change of the forgetting rate is too large, it indicates that the image classification model is forgetting old knowledge too quickly and needs to be stopped in time. Therefore, in this embodiment, the forgetting rate and the rate of change of the forgetting rate are combined to determine whether the weight of the regularization term needs to be adjusted, so that the weight of the regularization term can be adjusted in time.

[0077] The forgetting rate threshold and the first rate of change threshold can be set empirically. Please refer to [reference needed]. Figure 2 The forgetting rate threshold in some embodiments of this application is determined by the following steps.

[0078] Step 11: Using fixed regularization term weights, incrementally train the initial image classification model, record the forgetting rate of the image classification model after each training stage, and obtain the forgetting rate change curve.

[0079] In this step, the initial image classification model underwent a full-process incremental training, during which the weights of the regularization term remained unchanged.

[0080] Step 12: Determine the location of catastrophic forgetting in the forgetting rate curve.

[0081] The location in the forgetting rate change curve where catastrophic forgetting occurs can be a location where the forgetting rate change rate exceeds a set threshold, etc.

[0082] In some embodiments, determining the location of catastrophic forgetting in the forgetting rate change curve includes: setting a sliding window of size N, sliding the sliding window on the forgetting rate change curve, and determining whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit each time the window is slid to a position. If so, the position is determined to be the location of catastrophic forgetting in the forgetting rate change curve, where N is an integer not less than 2.

[0083] In some embodiments, determining whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit includes: calculating the difference between the last forgetting rate and the first forgetting rate within the sliding window; if the difference between the last forgetting rate and the first forgetting rate is greater than a preset first difference threshold, then it is determined that the increase in the forgetting rate within the sliding window exceeds the preset increase limit.

[0084] The difference between the last forgetting rate and the first forgetting rate refers to the absolute value of the difference between the last forgetting rate and the first forgetting rate, or the difference between the larger of the last forgetting rate and the first forgetting rate and the smaller of the two.

[0085] In other embodiments, determining whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit includes: calculating the difference between the last forgetting rate and the first forgetting rate within the sliding window, and the standard deviation or range of the forgetting rate within the sliding window. If the difference between the last forgetting rate and the first forgetting rate within the sliding window is greater than a first difference threshold, and the standard deviation of the forgetting rate within the sliding window is greater than a preset first standard deviation threshold or the range is greater than a preset first range threshold, then it is determined that the increase in the forgetting rate within the sliding window exceeds the preset increase limit.

[0086] Standard deviation and range can measure the degree of abrupt change in the forgetting rate. When the standard deviation or range is too large, it indicates that catastrophic forgetting is very likely to occur. Therefore, introducing standard deviation and range is helpful to accurately determine the location where catastrophic forgetting occurs.

[0087] Step 13: For the portion of the forgetting rate change curve before the location of catastrophic forgetting, obtain the average or median forgetting rate as the forgetting rate threshold.

[0088] In some embodiments, the first rate of change threshold can be determined by taking the average or median of the rate of change of forgetting rate of the portion of the forgetting rate change curve before the location of catastrophic forgetting in step 13.

[0089] In the above embodiments, by analyzing the forgetting rate change curve of the image classification model, the location where catastrophic forgetting occurs is identified. The average or median forgetting rate of the portion before the location where catastrophic forgetting occurs is obtained as the forgetting rate threshold, and the average or median forgetting rate change rate is obtained as the first change rate threshold. In other words, the forgetting rate threshold and the first change rate threshold are determined based on the normal forgetting rate in the stage where no severe forgetting occurs when the image classification model performs an incremental task. This can obtain forgetting rate threshold and forgetting rate change rate threshold that are consistent with the difficulty of performing the incremental task, making the forgetting rate threshold and forgetting rate change rate threshold more accurate and more adaptable.

[0090] In some embodiments, adjusting the weights of regularization terms to enhance the regularization constraints on the image classification model includes increasing the weights of some or all regularization terms by a fixed amount, or increasing the weights of some or all regularization terms by an amount proportional to the rate of change of the forgetting rate.

[0091] In one embodiment, the weights of some or all regularization terms are increased by a fixed margin. For example, the loss function includes an L2 norm regularization term, a weight regularization term, and a data regularization term, with weights λ respectively. L2 , λ w and λ d Then the weights of the three regularization terms can be adjusted to λ respectively. L2 ×(1+0.1), λ w ×(1+0.1) and λ d ×(1+0.1). In another embodiment, a fixed value can be added to the weight. The magnitude of the weight increase can be set according to actual needs, and the magnitude of the weight increase for different regularization terms can be the same or different.

[0092] By increasing the weights of most or all regularization terms by a factor proportional to the rate of change of the forgetting rate, the proportion or value of the weight increase can be proportional to the rate of change of the forgetting rate. By increasing the weights of the regularization terms by a factor proportional to the rate of change of the forgetting rate, the strength of the regularization constraint can be adaptively adjusted according to the rate of change of the forgetting rate.

[0093] During incremental training, the performance metrics of the image classification model also reflect the degree to which the image classification model has forgotten old knowledge. When the performance metrics are too low, it indicates that a large degree of forgetting has occurred. Therefore, in some embodiments of this invention, performance metrics are used to determine the degree of forgetting.

[0094] Specifically, in some embodiments, evaluating the degree of forgetting of old knowledge by the trained image classification model and determining whether the degree of forgetting of old knowledge by the trained image classification model exceeds a preset limit includes: evaluating the performance index of the trained image classification model, determining whether the performance index of the trained image classification model is less than a preset performance index threshold, and if so, determining that the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit. In some embodiments, if the performance index is not less than the performance index threshold, it is considered that the degree of forgetting of old knowledge by the trained image classification model does not exceed the preset limit. In some embodiments, if the performance index is not less than the performance index threshold and is greater than a preset lower limit of the performance index, it is considered that the degree of forgetting of old knowledge by the trained image classification model does not exceed the preset limit; if the performance index is not less than the performance index threshold and is not greater than the lower limit of the performance index, then the forgetting rate is calculated and compared with the forgetting rate threshold, and the result of the comparison determines whether it exceeds the preset limit.

[0095] The performance metrics here can be those on older images.

[0096] In some embodiments, evaluating the degree to which the trained image classification model has forgotten prior knowledge, and determining whether the degree to which the trained image classification model has forgotten prior knowledge exceeds a preset limit, includes: evaluating the performance index and the rate of change of the performance index of the trained image classification model, and determining whether the performance index of the trained image classification model is less than a preset performance index threshold or whether the rate of change of the performance index is greater than a preset second rate of change threshold. If so, it is determined that the degree to which the trained image classification model has forgotten prior knowledge exceeds the preset limit. In some embodiments, if the performance index is not less than the preset performance index threshold and the rate of change of the performance index is not greater than the second rate of change threshold, it is considered that the degree to which the trained image classification model has forgotten prior knowledge has not exceeded the preset limit.

[0097] When the rate of change of performance metrics is too large, it indicates that the image classification model is forgetting old knowledge too quickly and needs to be stopped in time. Therefore, in this embodiment, the weight of the regularization term is determined by combining the performance metrics and the rate of change of performance metrics, so that the weight of the regularization term can be adjusted in a timely manner.

[0098] The performance metric threshold and the second rate of change threshold can be set empirically. Please refer to [reference needed]. Figure 3 In some embodiments of this application, the performance index thresholds are determined through the following steps.

[0099] Step 21: Using fixed regularization term weights, incrementally train the initial image classification model, record the performance index of the image classification model after each training stage, and obtain the performance index change curve.

[0100] In this step, the initial image classification model underwent a full-process incremental training, during which the weights of the regularization term remained unchanged.

[0101] Step 22: Determine the location in the performance index change curve where catastrophic forgetting occurs.

[0102] The location in the performance indicator change curve where catastrophic amnesia occurs could be where the rate of change of the performance indicator exceeds a set threshold, etc.

[0103] In some embodiments, determining the location of catastrophic forgetting in the performance index change curve includes: setting a sliding window of size M, sliding the sliding window on the performance index change curve, and determining whether the decrease in the performance index within the sliding window exceeds a preset decrease limit each time the sliding window is reached. If so, the location is determined to be the location of catastrophic forgetting in the forgetting rate change curve, where M is an integer not less than 2.

[0104] In some embodiments, determining whether the decline of a performance metric within a sliding window exceeds a preset decline limit includes: calculating the difference between the last performance metric and the first performance metric within the sliding window; if the difference between the last performance metric and the first performance metric is greater than a preset second difference threshold, then it is determined that the decline of a performance metric within the sliding window exceeds the preset decline limit.

[0105] The difference between the last performance indicator and the first performance indicator refers to the absolute value of the difference between the last performance indicator and the first performance indicator, or the difference between the larger of the last performance indicator and the smaller of the first performance indicator.

[0106] In other embodiments, determining whether the decline in performance metrics within the sliding window exceeds a preset decline limit includes: calculating the difference between the last performance metric and the first performance metric within the sliding window, and the standard deviation or range of the performance metrics within the sliding window; if the difference between the last performance metric and the first performance metric within the sliding window is greater than a second difference threshold, and the standard deviation of the performance metrics within the sliding window is greater than a preset second standard deviation threshold or the range is greater than a preset second range threshold, then it is determined that the decline in performance metrics within the sliding window exceeds the preset decline limit.

[0107] Standard deviation and range can measure the degree of abrupt change in performance indicators. When the standard deviation or range is too large, it indicates that catastrophic forgetting is very likely to occur. Therefore, introducing standard deviation and range is helpful to accurately determine the location where catastrophic forgetting occurs.

[0108] Step 23: For the portion of the performance index change curve before the location of catastrophic forgetting, obtain the average or median value of the performance index as the performance index threshold.

[0109] In some embodiments, the second rate of change threshold can be determined by taking the average or median of the rate of change of the performance index in the portion of the performance index change curve before the position where catastrophic forgetting occurs in step 23.

[0110] In the above embodiments, by analyzing the performance index change curve of the image classification model, the location where catastrophic forgetting occurs is identified. The average or median value of the performance index before the location where catastrophic forgetting occurs is obtained as the performance index threshold, and the average or median value of the performance index change rate is obtained as the first change rate threshold. In other words, the performance index threshold is determined based on the normal performance index of the stage where no severe forgetting occurs when the image classification model performs the incremental task, and the second change rate threshold is determined based on the normal performance index change rate of the stage where no severe forgetting occurs. This can obtain the performance index threshold and performance index change rate threshold that are consistent with the difficulty of performing the incremental task, making the performance index threshold and performance index change rate threshold more accurate and more adaptable.

[0111] In some embodiments, adjusting the weights of regularization terms to enhance the regularization constraints on the image classification model includes increasing the weights of some or all regularization terms by a fixed amount, or increasing the weights of some or all regularization terms by an amount proportional to the rate of change of the performance metric.

[0112] For details on how to increase the weights of most or all regularization terms by a fixed margin, please refer to the above. Increasing the weights of most or all regularization terms by a margin proportional to the rate of change of the performance index means that the percentage or value of the increase in weight is proportional to the rate of change of the performance index. By increasing the weights of the regularization terms by a margin proportional to the rate of change of the performance index, the strength of the regularization constraint can be adaptively adjusted according to the rate of change of the performance index.

[0113] In some embodiments, the newly acquired training image set in step 100 is the same as the newly acquired training image set obtained in the corresponding training stage of incremental training of the initial image classification model using the weights of a fixed regularization term in step 11 or step 21. That is, if the current training is in the first training stage, the newly acquired training image set is the same as the newly acquired training image set in the first training stage of incremental training in step 11 or step 21; if the current training is in the second training stage, the newly acquired training image set is the same as the newly acquired training image set in the second training stage of incremental training in step 11 or step 21, and so on.

[0114] In wafer defect detection tasks, new training image sets can be obtained in the following ways: using a wafer image preprocessing module, the wafer images obtained by wafer defect detection equipment (such as AOI (Automatic Optical Inspection) equipment, dark field inspection equipment, etc.) are converted into a dataset format suitable for incremental training; the converted wafer images are manually labeled and defect category labels are added, and the wafer images and their corresponding defect category labels are used as labeling data pairs to form new training image sets.

[0115] In some embodiments, if the degree of forgetting of old knowledge by the trained image classification model does not exceed a preset limit, the weight of the regularization term is kept unchanged or the weight of the regularization term is increased. In this case, the increase in the weight of the regularization term is less than the increase in the weight of the regularization term when the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit. By appropriately increasing the weight of the regularization term even when the degree of forgetting does not exceed the preset limit, the catastrophic forgetting problem is further alleviated.

[0116] For example, the loss function includes an L2 norm regularization term, a weight regularization term, and a data regularization term, with weights λ respectively. L2 , λ w and λ d If the degree of forgetting exceeds the preset limit, the weights of the three regularization terms will be adjusted to λ respectively. L2 ×(1+0.1), λ w ×(1+0.1) and λ d If the result is ×(1+0.1), and this does not exceed the limit, then the weights of the three regularization terms are adjusted to λ respectively. L2 ×(1+0.01), λ w ×(1+0.01) and λ d ×(1+0.01).

[0117] This invention also provides an image classification method, please refer to... Figure 4 In some embodiments, the method includes the following steps:

[0118] Step 600: Obtain the image to be classified;

[0119] Step 700: Input the image to be classified into the image classification model trained by the incremental training method of any embodiment of the present invention to obtain the category of the image to be classified.

[0120] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0121] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. An incremental training method for an image classification model, used to incrementally train an initial image classification model, wherein the incremental training includes multiple training stages, characterized in that, Each of the training phases includes: Obtain a training image set, which includes a historical training image set and a newly added training image set; The image classification model is trained using the training image set, wherein the loss function used when training the image classification model includes one or more regularization terms; Determine whether the preset training stop condition has been met. If not, assess the degree of forgetting of old knowledge by the trained image classification model. Determine whether the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit. If so, adjust the weight of the regularization term to enhance the regularization constraint on the image classification model, and continue training in the next training stage using the adjusted weight of the regularization term. The step of evaluating the degree to which the trained image classification model has forgotten old knowledge, and determining whether the degree to which the trained image classification model has forgotten old knowledge exceeds a preset limit, includes: The forgetting rate and the rate of change of the forgetting rate of the trained image classification model are evaluated. It is determined whether the forgetting rate of the trained image classification model is greater than a preset forgetting rate threshold or whether the rate of change of the forgetting rate is greater than a preset first rate of change threshold. If so, it is determined that the degree of forgetting of old knowledge by the trained image classification model exceeds the preset limit. The forgetting rate threshold and the first change rate threshold are determined through the following steps: Using fixed regularization term weights, the initial image classification model is incrementally trained, and the forgetting rate of the image classification model after each training stage is recorded to obtain the forgetting rate change curve. Set a sliding window of size N, slide the sliding window on the forgetting rate change curve, and at each position, determine whether the increase in the forgetting rate within the sliding window exceeds the preset increase limit. If so, determine that position as the position where catastrophic forgetting occurs in the forgetting rate change curve, where N is an integer not less than 2. For the portion of the forgetting rate change curve before the location of catastrophic forgetting, the average or median forgetting rate is obtained as the forgetting rate threshold, and the average or median forgetting rate change rate is obtained as the first change rate threshold. The determination of whether the increase in the forgetting rate within the sliding window exceeds a preset increase limit includes: Calculate the difference between the last forgetting rate and the first forgetting rate within the sliding window. If the difference between the last forgetting rate and the first forgetting rate is greater than a preset first difference threshold, then determine that the increase in the forgetting rate within the sliding window exceeds a preset increase limit. Alternatively, calculate the difference between the last forgetting rate and the first forgetting rate within the sliding window, as well as the standard deviation or range of the forgetting rate within the sliding window. If the difference between the last forgetting rate and the first forgetting rate within the sliding window is greater than the first difference threshold, and the standard deviation of the forgetting rate within the sliding window is greater than a preset first standard deviation threshold or the range is greater than a preset first range threshold, then determine that the increase in the forgetting rate within the sliding window exceeds a preset increase limit.

2. The incremental training method as described in claim 1, characterized in that, The adjustment of the weights of the regularization term to enhance the regularization constraints on the image classification model includes: The weights of most or all of the regularization terms may be increased by a fixed amount, or the weights of most or all of the regularization terms may be increased by an amount proportional to the rate of change of the forgetting rate.

3. The incremental training method as described in claim 1, characterized in that, The newly acquired training image set in the step of acquiring the training image set is the same as the newly acquired training image set in the corresponding training stage of the step of incrementally training the initial image classification model using the weights of a fixed regularization term.

4. The incremental training method as described in claim 2, characterized in that, Also includes: If the degree of forgetting of old knowledge by the trained image classification model does not exceed a preset limit, then the weight of the regularization term remains unchanged or the weight of the regularization term is increased, wherein the increase in the weight of the regularization term is less than the increase in the weight of the regularization term when the degree of forgetting of old knowledge by the trained image classification model exceeds a preset limit.

5. An image classification method, characterized in that, include: Obtain the image to be classified; The image to be classified is input into an image classification model trained by the incremental training method as described in any one of claims 1 to 4 to obtain the category of the image to be classified.

6. A computer-readable storage medium, characterized in that, The medium stores a computer program that can be executed by a processor to implement the incremental training method as described in any one of claims 1 to 4 or the image classification method as described in claim 5.

Images