Online learning method and system for image recognition based on large model fine-tuning

Abstract

This invention discloses an online learning method and system for image recognition based on large model fine-tuning, relating to the field of image recognition large model update technology. The method includes: obtaining newly added samples in the current period; extracting a portion of samples from the historical sample set, mixing the selected center samples with all the newly added samples to form an enhanced training dataset for the current period; adding a classification head to the pre-trained recognition large model and randomly initializing the parameters of the classification head; freezing the parameters of the initialized recognition model except for the last two layers and the classification head, unfreezing the last two layers and the classification head, and then performing optimization training; sampling representative samples from the newly added training images in the current period and adding them to the historical sample set. This invention avoids the repetitive work of retraining with full data for each update in traditional methods, significantly improving the efficiency and practicality of model updates.

Description

Technical Field

[0001] This invention relates to the field of image recognition large model update technology, and in particular to an online learning method and system for image recognition based on large model fine-tuning. Background Technology

[0002] Image recognition technology has been widely applied in various practical scenarios, such as real-time analysis of video streams in camera surveillance systems and automatic review of user-uploaded images on social media platforms. As application scenarios continue to expand and change, models need to be able to identify newly emerging categories, such as new types of hazardous materials appearing in surveillance footage or new content appearing in user-uploaded images. However, within the existing technological framework, when it is necessary to increase the model's ability to identify new categories, there is often a challenge in efficiently updating the model without losing existing knowledge.

[0003] Currently, mainstream image recognition methods all employ deep learning models, such as ResNet and EfficientNet. These models acquire the ability to extract image features and classify images through supervised learning on large-scale training sets. When the model needs to recognize new image categories, the most direct approach is to merge the new training data with all the existing training data and retrain a new model using the complete dataset. While this method ensures that the model simultaneously masters knowledge of both old and new categories, each update requires reprocessing all historical data.

[0004] This method of retraining based on full dataset has significant drawbacks. First, as historical data accumulates, the size of the complete training set continuously expands, leading to a sharp increase in computational cost and time consumption for each retraining session, resulting in a large amount of repetitive computational work. Second, if only new data is used for incremental training, the update of model parameters will cause changes in its feature representation of historical data, leading to a catastrophic forgetting problem and a significant decline in the model's recognition performance on historical categories. Therefore, how to achieve continuous model learning with low computational cost while maintaining the ability to recognize historical categories, without resorting to full retraining, has become an urgent technical problem to be solved.

[0005] This invention addresses the aforementioned problems by proposing an online learning method for image recognition based on large model fine-tuning. It aims to leverage the powerful representational capabilities and flexible fine-tuning strategies of large models to achieve incremental learning of the model on new data, while effectively suppressing the forgetting of historical knowledge. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically by providing an online learning method and system for image recognition based on large model fine-tuning, as detailed below: 1) In a first aspect, the present invention provides an online learning method for image recognition based on large model fine-tuning, the specific technical solution of which is as follows: The newly added training images in the current period are preprocessed to obtain new samples; A portion of samples are extracted from the historical sample set, and a clustering algorithm is used to select the center samples of each category. The selected center samples are then mixed with all the newly added samples to form the augmented training dataset for the current period. A classification head is added to the pre-trained large recognition model, and the parameters of the classification head are randomly initialized to obtain an initialized recognition model. Freeze the parameters of all layers in the initial recognition model except the last two layers and the classification head, unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset at a preset learning rate. During the optimization training process, evaluate the recognition performance of historical categories to ensure that the forgetting rate of historical categories is lower than a preset threshold. Representative samples are sampled from the newly added training images in the current period and added to the historical sample set for sample mixing in the next period.

[0007] The beneficial effects of the online learning method for image recognition based on large model fine-tuning provided by this invention are as follows: By preprocessing newly added training images in the current period, new samples that meet the input requirements of the large-scale recognition model are obtained, ensuring data consistency and model training stability. A portion of samples is extracted from the historical sample set, and a clustering algorithm is used to select the center samples for each category. These center samples are then mixed with all new samples to form an augmented training dataset. This approach introduces new knowledge while preserving the typical features of historical categories, effectively preventing catastrophic forgetting. A classification head is added to the pre-trained large-scale recognition model, and parameters are randomly initialized, enabling the model to adapt to the classification task of the current period. By freezing the parameters of all layers except the last two layers and the classification head, and only unfreezing the last two layers and the classification head, optimization training is performed at a preset learning rate. This significantly reduces the number of parameters that need to be updated, lowers computational costs, and maintains stable memory of historical knowledge. During the optimization training process, the recognition performance of historical categories is evaluated, and the forgetting rate is kept below a preset threshold, achieving quantitative control over the recognition ability of historical categories. Representative samples are sampled from the newly added training images in the current cycle and added to the historical sample set, which can be dynamically updated for sample mixing in the next cycle. This allows the model to continuously maintain its recognition performance of historical categories during long-term online learning, avoiding the repetitive work of retraining with full data for each update in traditional methods, and significantly improving the efficiency and practicality of model updates.

[0008] Based on the above scheme, the online learning method for image recognition based on large model fine-tuning of the present invention can be further improved as follows.

[0009] Furthermore, representative samples are sampled from the newly added training images in the current period and added to the historical sample set for sample mixing in the next period, including: From the newly added training images in the current period, a preset number of image samples are evenly extracted according to the category ratio. The extracted image samples are added to the historical sample set as representative samples to update the historical sample set, which is then used to mix with the newly added samples corresponding to the newly added training images in the next period.

[0010] The beneficial effects of adopting the above-mentioned further scheme are as follows: By uniformly extracting a predetermined number of image samples from the newly added training images in the current cycle according to the category ratio and adding them to the historical sample set as representative samples, it is possible to ensure that the number of samples in each category in the historical sample set remains balanced, avoiding bias in some categories due to an excessive number of samples or forgetting in some categories due to an insufficient number of samples. The updated historical sample set is used to mix with the newly added samples in the next cycle, so that the augmented training dataset in subsequent cycles always contains a balanced category distribution, thereby stably maintaining the model's ability to recognize all historical categories in long-term online learning and effectively controlling catastrophic forgetting.

[0011] Furthermore, a clustering algorithm is used to select the center samples for each category, including: using the k-means clustering algorithm to cluster a portion of the samples extracted from the historical sample set, and selecting the cluster center samples of each category as the center samples.

[0012] The beneficial effects of adopting the above-mentioned further approach are as follows: Using the k-means clustering algorithm to cluster a subset of samples extracted from the historical sample set, and selecting the cluster center of each category as the center sample, allows for the extraction of the most representative and typical samples from a large number of historical samples, removing redundant and marginal samples. This ensures that the hybrid augmented training dataset retains the core distribution characteristics of each category to the greatest extent possible within a limited sample capacity. This reduces the amount of training data and computational overhead while ensuring the effective inheritance of historical knowledge and improving the efficiency of remembering historical categories during model fine-tuning.

[0013] Furthermore, it also includes: after optimizing and training the thawed model part with a preset learning rate, obtaining the image to be recognized, inputting the image to be recognized into the optimized and trained large recognition model, and using the optimized and trained large recognition model to perform recognition processing on the image to be recognized and output the category information corresponding to the image to be recognized.

[0014] The beneficial effects of adopting the above-mentioned further approach are as follows: After optimizing and training the thawed model part with a preset learning rate, the image to be recognized is acquired and input into the optimized and trained recognition model. The model outputs the corresponding category information through forward propagation, thus realizing the transformation of training results into practical applications. Since the model has mastered the characteristics of new and historical categories during training through augmented datasets and fine-tuning with a small learning rate, it can accurately output the category information of the image to be recognized during actual recognition, meeting the real-time recognition needs in scenarios such as monitoring alarms and content classification, and ensuring the practical value of the online learning system.

[0015] 2) Secondly, the present invention also provides an online learning system for image recognition based on large model fine-tuning, the specific technical solution of which is as follows: It includes a preprocessing module, an extraction and clustering module, a model optimization module, a model training module, and a sampling and insertion module; The preprocessing module is used to preprocess newly added training images in the current period to obtain new samples; The extraction and clustering module is used to: extract a portion of samples from the historical sample set, use a clustering algorithm to select the center samples of each category, and mix the selected center samples with all the newly added samples to form the augmented training dataset for the current period. The model optimization module is used to: add a classification head to a pre-trained large recognition model and randomly initialize the parameters of the classification head to obtain an initialized recognition model; The model training module is used to: freeze the parameters of the initial recognition model except for the last two layers and the classification head, unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset at a preset learning rate, and evaluate the recognition performance of historical categories during the optimization training process to ensure that the forgetting rate of historical categories is lower than a preset threshold. The sampling addition module is used to: sample representative samples from the newly added training images in the current period and add them to the historical sample set for sample mixing in the next period.

[0016] Based on the above scheme, the online learning system for image recognition based on large model fine-tuning of the present invention can be further improved as follows.

[0017] Furthermore, the sampling addition module is specifically used to: uniformly extract a preset number of image samples from the newly added training images in the current period according to the category ratio, and add the extracted image samples as representative samples to the historical sample set to update the historical sample set, which is then used to mix with the newly added samples corresponding to the newly added training images in the next period.

[0018] Furthermore, the extraction and clustering module is specifically used to: use the k-means clustering algorithm to cluster a portion of the samples extracted from the historical sample set, and select the samples with the cluster centers of each category as the center samples.

[0019] Furthermore, it also includes a model application module, which is specifically used to: after optimizing and training the unfrozen model part with a preset learning rate, obtain the image to be recognized, input the image to be recognized into the optimized and trained large recognition model, and perform recognition processing on the image to be recognized through the optimized and trained large recognition model to output the category information corresponding to the image to be recognized.

[0020] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so as to enable the electronic device to implement any of the above-mentioned online learning methods for image recognition based on large model fine-tuning.

[0021] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned online learning methods for image recognition based on large model fine-tuning.

[0022] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below: Figure 1 This is a flowchart illustrating an online learning method for image recognition based on large model fine-tuning according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an online learning system for image recognition based on large model fine-tuning, according to an embodiment of the present invention. Detailed Implementation

[0024] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0025] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0026] like Figure 1 As shown in the figure, an online learning method for image recognition based on large model fine-tuning according to an embodiment of the present invention includes the following steps: S1. Preprocess the newly added training images in the current period to obtain new samples. The specific implementation process is as follows: S10. Since the newly added training images from different sources have different original sizes and resolutions, in order to meet the consistency requirements of the input data size for large recognition models, the size of all newly added training images needs to be adjusted to a uniform size. Specifically, the size of each newly added training image is adjusted to 224 pixels × 224 pixels. When adjusting the size, an image scaling algorithm is used to scale the width and height of the original image to 224 pixels respectively. For non-square images, they are first scaled according to the aspect ratio of the original image so that the shorter side of the image reaches 224 pixels, and then the longer side is centered and cropped to a 224 pixel × 224 pixel area, ensuring that the final image size is strictly uniform at 224 pixels × 224 pixels, while retaining the core content of the image.

[0027] S11. After standardizing the image size, the pixel values ​​of the image need to be normalized to accelerate the convergence speed of model training and improve the stability of model training. Specifically, the pixel values ​​of each newly added training image are converted from the original integer range to the floating-point range. For each pixel, its original pixel value is usually an integer between 0 and 255. First, the pixel value of each pixel is divided by 255, mapping it to the floating-point range of 0 to 1, to obtain the normalized pixel value. Then, based on the statistical information pre-calculated from the large-scale image dataset, the normalized pixel values ​​are further standardized. Specifically, for the red, green, and blue color channels of the image, the corresponding mean is subtracted and divided by the corresponding standard deviation. The mean of the red channel is 0.485, the mean of the green channel is 0.456, and the mean of the blue channel is 0.406; the standard deviation of the red channel is 0.229, the standard deviation of the green channel is 0.224, and the standard deviation of the blue channel is 0.225. After the above standardization process, the pixel value distribution of each color channel is adjusted to a standard normal distribution with a mean of 0 and a standard deviation of 1, so that the data distribution of the newly added training images is consistent with the data distribution used in the pre-training stage of the recognition model.

[0028] S12. To increase the diversity of newly added training images and improve the model's generalization ability, data augmentation operations can be performed on the newly added training images before size standardization and pixel value normalization. Data augmentation operations include, but are not limited to, random horizontal flipping, random rotation, random color jittering, and random Gaussian noise addition. Random horizontal flipping mirrors the image with a 50% probability; random rotation rotates the image at random angles within the range of -10 degrees to 10 degrees; random color jittering randomly adjusts the brightness, contrast, saturation, and hue of the image within a preset range; random Gaussian noise addition adds Gaussian noise with a mean of 0 and a preset standard deviation to the image. Through these data augmentation operations, more diverse new training image samples are generated, thereby effectively expanding the data scale of new training images and preventing overfitting in subsequent training processes.

[0029] S13. After the above steps, each newly added training image is converted into the tensor format required for model training. Specifically, the image data is converted from a three-dimensional array of height × width × number of channels into a three-dimensional tensor of number of channels × height × width to meet the input data format requirements of large recognition models. The final new samples are image tensors that have undergone size normalization, pixel value normalization, optional data augmentation, and format conversion. These new samples will be used to subsequently mix with historical samples to form an augmented training dataset.

[0030] S2. Extract a portion of samples from the historical sample set, use a clustering algorithm to select the center samples of each category, and mix the selected center samples with all the newly added samples to form the augmented training dataset for the current period; wherein, using a clustering algorithm to select the center samples of each category includes: using the k-means clustering algorithm to cluster the portion of samples extracted from the historical sample set, and selecting the cluster center samples of each category as the center samples.

[0031] The specific implementation process for extracting a subset of samples from the historical sample set is as follows: Based on the category information of the new samples in the current period, samples are drawn from the historical sample set according to the principle of category balance. Specifically, firstly, a list of all categories in the new samples of the current period is obtained, and the number of new samples in each category is counted. Then, for each category, all historical samples belonging to that category are retrieved from the historical sample set. If the number of samples of that category in the historical sample set is sufficient, a certain number of samples are drawn using a random sampling method. The number of samples drawn is set to be proportional to the number of samples of that category in the new samples, for example, 50% of the number of new samples. At the same time, an upper limit of 100 samples and a lower limit of 10 samples are set for each category to ensure that the drawn samples are representative and do not increase the computational burden too much. If the number of samples of some categories in the historical sample set is less than the lower limit, all historical samples of that category are drawn. After the drawing is completed, a subset of historical samples is obtained, which is used for subsequent clustering screening. During the drawing process, the category label of each drawn sample also needs to be recorded so that it can be processed separately by category during clustering.

[0032] The process of selecting center samples for each category using clustering algorithms includes: using the k-means clustering algorithm to cluster a subset of samples extracted from the historical sample set, and selecting the cluster centers of each category as center samples. The specific implementation process is as follows: All samples from the extracted historical sample subset are input into the current large-scale recognition model, and the feature vector of each sample is extracted. These feature vectors are taken from the output of the penultimate layer of the large-scale recognition model, i.e., the feature representation layer before the classification head, and have a dimension of d, typically a high-dimensional vector. For each sample i, its feature vector is obtained. Then, for each category c, the feature vector set of all samples in that category is... As input, the samples are clustered into k clusters using the k-means clustering algorithm. The value of k is dynamically determined based on the number of samples in each cluster. For example, if the number of center samples extracted from each cluster is set to 10, then k=10. However, if the number of samples in a cluster is less than 10, then k takes the actual number of samples in that cluster. The k-means algorithm iteratively optimizes the distribution of samples into k clusters, minimizing the sum of squared distances from each sample to the center of its cluster. After the algorithm converges, the cluster center vector of each cluster is obtained. Where j=1,...,k. Then, for each cluster, select the sample closest to the cluster center from all samples in that cluster, i.e., calculate the distance from all samples... and The Euclidean distance is used to select the sample with the smallest distance as the center sample of the cluster. In this way, k center samples are selected for each class c, and the feature vectors of these center samples can represent the typical distribution of the class. For classes with fewer than k samples, all samples of that class are used as center samples to ensure that historical information is not lost.

[0033] The selected center samples are mixed with all the newly added samples to form the augmented training dataset for the current period. The specific implementation process is as follows: All the selected center samples are collected to form a center sample set. Meanwhile, all the newly added samples obtained after preprocessing in the current period constitute the new sample set. Then, and The datasets are merged to obtain the augmented training dataset. During the merging process, it is necessary to ensure that all samples have a consistent data format, i.e., a 3D tensor of channel number × 224 × 224, and that pixel values ​​have been normalized. To maintain class balance in the dataset, the number of samples in each class is counted after merging. If the number of samples in some classes is found to be too low or too high, adjustments can be made by oversampling newly added samples or undersampling center samples. However, the selection of center samples usually already takes class balance into account, so direct merging is sufficient. The final augmented training dataset is obtained. This will be used in subsequent model training steps, including adding a classifier head to the large recognition model and randomly initializing the classifier head parameters, unfreezing the classifier head and the last two layers of the large recognition model, optimizing training with a preset learning rate, and evaluating the recognition performance of historical categories during training to ensure that the forgetting rate is below a preset threshold.

[0034] The historical sample set refers to the collection of representative samples extracted from the newly added training images in all training cycles prior to the current cycle. These representative samples are used to mix with the new samples in subsequent cycles to maintain the model's memory of historical categories and prevent catastrophic forgetting. Each sample in the historical sample set has undergone preprocessing, including image size normalization and pixel value normalization, and its data format is consistent with that of the new samples, allowing it to be directly input into a large recognition model for feature extraction and training.

[0035] In another possible implementation, a subset of samples is extracted from the historical sample set, including: 1) Input all samples from the historical sample set into the current recognition model, extract the feature vector output of each sample in the penultimate layer, and record the model's predicted probability distribution for each sample. By calculating the cross-entropy loss value between the predicted probability distribution of each sample and its true label, the learning difficulty index of each sample for the current model is obtained. The larger the cross-entropy loss value, the easier it is for the model to forget the sample. Specifically: All samples from the historical sample set are sequentially input into the current recognition model for forward propagation computation. The feature vector for each sample is obtained from the second-to-last layer of the model, the feature representation layer before the classification head, denoted as... ,in, , The total number of historical samples. This is the feature dimension. Simultaneously, the predicted probability distribution for each sample is obtained from the model's classification head output layer, denoted as... ,in, This represents the total number of categories. Each historical sample corresponds to a true category label, represented using one-hot encoding, denoted as . If the sample belongs to category ,but The One component is set to 1, and the rest to 0. The cross-entropy loss between the predicted probability distribution and the true label for each sample is calculated as an indicator of learning difficulty. ,in, It is a sample The real label in the first Values ​​for each category The model predicts the sample. Category The probability of cross-entropy loss is as follows: The larger the cross-entropy loss value, the greater the difference between the model's prediction of the sample and the true label. The more difficult it is for the current model to classify the sample correctly, the easier it is for the model to forget it, and it needs to be prioritized for retention in subsequent training to strengthen memory.

[0036] 2) For each sample in the historical sample set, calculate the pairwise similarity between its feature vector and the feature vectors of all other samples in the same category. Based on the similarity matrix, construct a representativeness index for that sample among samples in the same category. The representativeness index is measured by the average similarity between that sample and other samples in the same category. The higher the average similarity, the more representative the sample is of the central distribution of that category. Specifically: For each sample in the historical sample set First, determine its category. Then, from the historical sample set, all samples belonging to the same category are selected. Other samples, to obtain a set of samples of the same category. The set contains One sample, including sample For each sample in the set, its feature vector has been extracted. The sample... The cosine similarity between the feature vector of each sample in the same category and the feature vector of the other sample in the same category is calculated using the following formula: ,in and Samples and samples eigenvectors, Let L2 norm represent the vector. All vectors related to the sample The similarity scores are grouped into a set, and the arithmetic mean of this set is taken as the sample value. Representative indicators among similar samples: If there are only samples in the same category For the sample itself, the representativeness index is defined as 1. The value of the representativeness index ranges from -1 to 1; the closer the value is to 1, the stronger the sample. The more consistent the average direction of the feature vector of a sample with that of other samples of the same category, that is, the closer the sample is to the central region of the category distribution, the more representative it is of the typical characteristics of that category.

[0037] 3) For each sample in the historical sample set, its learning difficulty index and representativeness index are multiplied to obtain the sample's comprehensive value score. The learning difficulty index and representativeness index form a complementary relationship, allowing samples that are both difficult for the model to remember and located in the center of the category to obtain higher comprehensive value scores. Therefore, these types of samples are prioritized for retention during extraction. Specifically: Because the learning difficulty index and the representativeness index have different dimensions, they need to be normalized to unify their value ranges into the same interval in order to perform multiplication operations. For the set of learning difficulty indices... Calculate the maximum value and minimum value Then, perform min-max normalization to obtain the normalized learning difficulty: For representative indicator sets Since cosine similarity is in the range [-1, 1], but is usually positive, it can be linearly mapped to the range [0, 1], or the original values ​​can be directly used for subsequent multiplication. However, in order to be consistent with the scale of the difficulty index, min-max normalization is also performed to obtain a normalized representative index: ,in, and These represent the maximum and minimum values ​​of the representative indicators, respectively. Then, the comprehensive value score is calculated: This product ensures that only samples with both high learning difficulty (easy to forget) and high representativeness (located in the center of the category) can obtain high scores, thus prioritizing the retention of samples that are crucial to maintaining the model's historical memory during extraction.

[0038] 4) Sort the samples in each category of the historical sample set according to their comprehensive value scores from high to low. Select a predetermined number of samples with the highest scores from each category. Use these samples as a subset of samples drawn from the historical sample set. Mix these samples with all new samples in the current period to form the augmented training dataset for the current period. Specifically: For each category The category is obtained by selecting all samples belonging to that category from the historical sample set. sample subset Each sample in this subset The overall value score has been calculated. .Will The samples are sorted from highest to lowest according to their comprehensive value scores, and a predetermined number of samples with the highest scores are selected. One sample, The value can be dynamically set based on the importance of the category or the number of samples, for example, taking 20% ​​of the total number of samples in the category or a fixed value of 10. The samples drawn from each category collectively constitute a partial sample set drawn from the historical sample set. These extracted samples are then mixed with all newly added samples in the current period to form the augmented training dataset for the current period, which is used for subsequent model fine-tuning training. In this way, key samples in the historical sample set are preserved, while redundant samples are filtered out, ensuring the effective inheritance of historical knowledge while controlling the size of the training dataset.

[0039] S3. Add a classification head to the pre-trained recognition model and randomly initialize the parameters of the classification head to obtain an initialized recognition model. The specific implementation process is as follows: S30. The large-scale recognition model uses DINOv2 as the base model for pre-training. DINOv2 is a large-scale visual transformation model based on self-supervised learning. Its pre-training process does not rely on manually labeled categories, but instead learns the internal structure and semantic features of images from massive amounts of unlabeled image data to obtain general visual representation capabilities. Specifically, the pre-training process first collects a diverse dataset containing hundreds of millions of images, covering various visual content such as natural scenes, objects, and text. Then, a self-distillation learning paradigm is used to train a teacher model and a student model. The teacher model generates stable feature representations by taking different perspectives of the images as input, while the student model attempts to mimic the output of the teacher model, updating its parameters by minimizing the similarity loss between the two. After multiple rounds of iterative optimization, a DINOv2 model with powerful feature extraction capabilities is finally obtained. In this technical solution, the DINOv2 core model with a total of 1.03B parameters is selected. After pre-training, the network parameters of this model can extract rich and discriminative image features, providing a solid foundation for subsequent online learning tasks. The pre-trained recognition model contains multiple transformer layers. Each layer uses a self-attention mechanism and a feedforward neural network to progressively abstract and transform the features of the input image, ultimately outputting a high-dimensional feature vector.

[0040] S31. The specific network layer structure of the classification head is designed as a single-layer fully connected network, i.e., a linear layer. The input dimension of this linear layer is equal to the feature dimension of the last layer output of the pre-trained recognition model, denoted as . Depending on the selected DINOv2 model, its feature output dimension is typically a fixed value. For example, for a model with 1.03B parameters, its feature dimension is a specific numerical value. The output dimension of the classification head is equal to the total number of categories to be identified in the current online learning task, denoted as . Therefore, the network layer structure of the classification head can be represented as: ,in, It is the feature vector output by the pre-trained recognition model. It is the weight matrix of the classification head. It is the bias vector of the classification head. This is the raw score vector output by the classification head, which is then converted into predicted probabilities for each category using the Softmax function. The network layer structure of the classification head is directly stacked on top of the last layer of the pre-trained recognition model, and the two are connected by feature vectors. The connection is established. The pre-trained recognition model is responsible for extracting high-dimensional features from the input image, while the classification head is responsible for mapping these features to a specific class space to make classification decisions.

[0041] S32. The parameters of the classification head include the weight matrix. and bias vector During the initialization phase, these parameters have not yet undergone any training and need to be assigned initial values ​​through random initialization so that they can be optimized through gradient descent during subsequent training. The specific implementation process of random initialization is as follows: For the weight matrix... Each element in ,in, Indicates category index, This represents the feature dimension index, using a mean of 0 and a standard deviation of 0. Random sampling is performed from a normal distribution. This initialization method is called Kaiming initialization, which is suitable for networks that subsequently use the ReLU activation function. It can effectively avoid the vanishing or exploding gradient problem and accelerate model convergence. For the bias vector Each element in Initialize all parameters of the classification head to 0. After initialization, the parameters of the classification head are assigned specific values. These values ​​will be continuously updated during subsequent training with the backpropagation algorithm, ultimately enabling the large recognition model to accurately identify various types of images. At this point, the pre-trained large recognition model and the randomly initialized classification head together constitute the initialized recognition model, ready to enter the next stage of unfreezing partial layers and optimization training.

[0042] S4. Freeze the parameters of all layers in the initialized recognition model except for the last two layers and the classification head. Unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset at a preset learning rate. During the optimization training process, evaluate the recognition performance of historical categories to ensure that the forgetting rate of historical categories is lower than a preset threshold. The specific implementation process is as follows: S40. After constructing the initial recognition model, the trainable states of each layer in the model need to be configured. Specifically, by traversing all network layers in the model, for each layer, if it does not belong to the last two transformer blocks and is not a classification head, the gradient calculation flag of the layer's parameters is set to "no gradient calculation state". This means that during subsequent backpropagation, the parameters of these layers will not receive gradient information, and the optimizer will not update their values. These frozen parameters include the embedding layer parameters of the input image, the position encoding parameters, and the weights and biases in all multi-head self-attention mechanisms and feedforward neural networks from the first transformer block to the third-to-last transformer block. By freezing these parameters, it can be ensured that the general visual features learned by the model during the pre-training stage are completely preserved and will not change due to training with new data, thereby effectively preventing catastrophic forgetting of historical knowledge.

[0043] S41. Unfreeze the last two layers and the classification head. Corresponding to the freeze operation, unfreezing means setting the gradient calculation flags for all parameters in the last two layers and the classification head to the gradient calculation state. The last two layers in the initialized recognition model refer to the last two transformer blocks counting backwards from the model output, i.e., the penultimate and second-to-last transformer blocks. These two transformer blocks have already learned high-level semantic features relevant to the specific task during the pre-training phase. By fine-tuning them on new data, they can better adapt to new categories while maintaining their discriminative ability for historical categories. The classification head is a linear layer attached to the top of the model; its parameters have not yet been trained and need to be fully updated to learn the classification boundaries of new categories. After unfreezing these layers, only the parameters of these unfrozen layers are added to the optimizer's parameter list during optimizer construction, ensuring that only these parameters are updated during training; the remaining frozen parameters are completely excluded from the optimizer.

[0044] S42, Learning rate set to 10 -4 This is a relatively small value, designed to allow the thawing parameters to move slowly in the parameter space, avoiding drastic interference with existing knowledge. The optimizer uses AdamW, a commonly used adaptive learning rate optimization algorithm that dynamically adjusts the learning rate of each parameter based on the first and second moments of the gradient, contributing to smooth convergence during fine-tuning. In each training iteration, a small batch of samples is randomly drawn from the augmented training dataset of the current period, input into the model, and forward propagation is performed to obtain the predicted probability for each class. Then, the cross-entropy loss function is calculated. ,in, Indicates batch size, This represents the current total number of categories. It is a sample The true label indicator variable, if the sample belongs to the category. The value is 1 if the value is 1, otherwise it is 0. The model predicts the sample. Category The probability of the loss function value is calculated using the backpropagation algorithm to determine the gradient of the unfreezing parameters. The optimizer then calculates the gradient based on the learning rate of 10. -4 These parameters are then updated along with gradients. The entire training process continues for multiple epochs until the model's performance on the validation set no longer improves or the preset maximum number of epochs is reached.

[0045] S43. To monitor the model's retention of historical categories in real time, a historical category validation set needs to be constructed before training begins. This validation set contains only image samples belonging to historical categories, and these samples have never participated in the current training cycle. Before training begins, the currently initialized recognition model is used to infer from this validation set, and the baseline recognition accuracy of the historical categories is calculated. During training, at certain iterations or after each training cycle, the updated model is used again to infer the same historical category validation set to calculate the current recognition accuracy. By comparing the baseline accuracy with the current accuracy, the degree to which the model has forgotten historical knowledge can be quantified.

[0046] S44. The forgetting rate of historical categories is defined as the percentage decrease in historical category recognition accuracy after training relative to the baseline accuracy before training. The calculation formula is as follows: The preset threshold is set at 5%, meaning the model's accuracy in recognizing historical categories after training should not decrease by more than 5%. After each evaluation, the forgetting rate is compared to 5%. If the forgetting rate remains below 5%, training continues normally until completion. If the forgetting rate exceeds 5% in any evaluation, intervention is required to restore model performance. Interventions may include: immediately stopping the current training and rolling back the model parameters to the state below the threshold at the last evaluation; dynamically reducing the learning rate, for example, reducing it to one-tenth of its original value, continuing training for several steps, and then re-evaluating; or adjusting the proportion of historical center samples in the augmented training dataset, increasing the weight of historical samples, and restarting training. Through repeated evaluations and necessary interventions, it is ensured that the final model, while mastering new categories, maintains a strictly controlled forgetting rate for historical categories within the preset threshold, thus achieving knowledge retention in online learning.

[0047] In this context, the last two layers of the initialized recognition model refer to the last two transformer layers in the pre-trained recognition model, arranged from output to input. Taking the DINOv2 model as an example, its main body consists of multiple stacked transformer blocks, each containing a multi-head self-attention mechanism and a feedforward neural network. The last two layers, the penultimate and tertiary transformer blocks, have already learned high-level semantic features during the pre-training phase. By unfreezing and fine-tuning at a small learning rate, the model can better adapt to new data while retaining its ability to abstractly represent historical data. The classification head is not part of the pre-training part of the recognition model and is therefore not included in the last two layers.

[0048] In the initial recognition model, the parameters of all layers except the last two layers and the classification head refer to the parameters of all network layers in the pre-trained recognition model, excluding the last two transformer blocks and the classification head. Specifically, this includes the input embedding layer, positional encoding parameters, and the parameters of all transformer blocks from the first to the third-to-last transformer block. These parameters have already learned general visual features during the pre-training phase. Freezing them prevents drastic changes during training, effectively protecting learned historical knowledge and avoiding catastrophic forgetting. During training, these parameters do not participate in gradient calculation and updates; their values ​​remain at their initial pre-training values.

[0049] Here, "historical categories" refers to the image classifications that have been learned and recognized in all completed training cycles prior to the current cycle. Training samples corresponding to these categories are stored in a historical sample set and used in subsequent cycles to mix with new samples during training, thus maintaining the model's memory of historical categories. During the evaluation phase, the recognition performance of historical categories is tested using an independent validation set, which contains image samples belonging only to historical categories and is not involved in the training process of the current cycle.

[0050] The forgetting rate of historical categories refers to the percentage decrease in the accuracy of the model in identifying historical categories after completing the current cycle of optimization training compared to before training. This metric is used to quantify the degree to which the model forgets old knowledge when learning new data. This technical solution requires the forgetting rate to be lower than a preset threshold, such as 5%, to ensure that the model can balance the learning of new knowledge with the retention of old knowledge.

[0051] S5. Sample representative samples from the newly added training images in the current period and add them to the historical sample set for sample mixing in the next period. Specifically, a preset number of image samples are evenly extracted from the newly added training images in the current period according to the class ratio. The extracted image samples are added to the historical sample set as representative samples to update the historical sample set, which is then used to mix with the newly added samples corresponding to the newly added training images in the next period. The specific implementation process is as follows: S50. First, iterate through all the newly added samples obtained after preprocessing in the current period, read the category label corresponding to each sample, use each category label as a key, and count the number of times the samples appear under each category to obtain the category sample quantity distribution. Suppose that the newly added training images in the current period contain a total of There are several different categories, each category The number of samples is denoted as ,in, .

[0052] S51. The preset quantity is a fixed integer value, denoted as... The number of images to be selected depends on the specific application requirements. For example, if 10 images are selected from each category, then the appropriate number is... For each category Compare their sample sizes With preset quantity The size. If Then the category is planned to be sampled. One sample; if Then all samples of that category are considered representative samples, meaning the actual number drawn is... The number of plans extracted for each category is recorded as follows: Then we have: .

[0053] S52, For each category First, obtain a list of indices for all new samples in that category, or the samples themselves. Then, randomly select from this list. The sample size is determined using random sampling without replacement, ensuring that each sample has an equal probability of being selected and that no two samples from the same category are selected repeatedly. Random sampling can be implemented using a random number generator with a fixed random seed to guarantee repeatability. For samples with insufficient sample size... If the category is clear, then all samples are selected directly. After extraction, a representative set of samples is obtained for each category. These samples are then aggregated to form the representative sample set for the current period. The size of the set is .

[0054] S53. The historical sample set is a sample library used for long-term memory, storing representative samples selected from all previous periods, and representing the current period's sample set. All samples are merged into the historical sample set to form a new historical sample set. When adding samples, it is necessary to record the category label, original image data, and preprocessing information already completed for each sample to ensure it can be directly used as input later. If the historical sample set has a capacity limit, a first-in-first-out (FIFO) or sample importance-based elimination strategy can be used; however, this technical solution does not involve a capacity limit, so only an addition operation is required. The updated historical sample set is denoted as […]. ,in It is the historical sample set before the update.

[0055] S54. At the start of the next cycle, the updated historical sample set will be used. A subset of samples is extracted, and the k-means clustering algorithm is used to select the center samples for each category. These center samples are then mixed with newly added samples from the next cycle to form the augmented training dataset for that cycle. Therefore, the representative samples selected in the current cycle will serve as carriers of historical information in subsequent cycles, participating in model training and helping the model maintain its ability to recognize historical categories. Through this cycle, the historical sample set continuously absorbs representative samples from new cycles, thus achieving long-term maintenance and updating of historical knowledge during online learning.

[0056] Representative samples refer to a predetermined number of image samples uniformly drawn from the newly added training images in the current period, according to the proportion of each category. These samples undergo the same preprocessing steps as the new samples, including image size standardization and pixel value normalization, to ensure that their data format is consistent with the model input requirements. The core role of representative samples is to be added to the historical sample set after the training of the current period, serving as a carrier of historical category knowledge. This knowledge is then used to mix with new samples in subsequent periods, thereby helping the model maintain its ability to recognize historical categories during continuous learning. By selectively retaining a portion of the most representative samples in each period, the feature distribution of each category can be effectively summarized and memorized without storing all historical data, avoiding catastrophic forgetting and controlling storage costs. The process of extracting representative samples ensures that an appropriate number of samples are retained for each category, allowing the historical sample set to reflect the information of all learned categories in a balanced manner.

[0057] In another possible implementation, representative samples are sampled from the newly added training images in the current period, including: 1) Input all newly added training images in the current period into the optimized and trained large recognition model, extract the feature vector output from the penultimate layer of each newly added training image, and simultaneously obtain the set of predicted probability distributions after the model performs multiple random data augmentations on each newly added training image. By calculating the mean of the KL divergence among the multiple predicted probability distributions, the prediction stability index of each newly added training image is obtained. The smaller the prediction stability index, the more stable the model's prediction of that image. Specifically: All newly added training images from the current period are input into the optimized and trained recognition model for forward propagation. The feature vector corresponding to each newly added training image is obtained from the second-to-last layer of the model, the feature representation layer before the classification head, denoted as... ,in For feature dimensions, subscript Indicates the first Each new training image is used independently. To measure the model's prediction stability for that image, a new training image is used. Random data augmentation operation The value is set to 10. Each enhancement uses a combination of transformations such as random horizontal flipping, random rotation from -10 degrees to +10 degrees, and random color jitter to generate an enhanced version of the image. Each enhanced version is input into the model to obtain the predicted probability distribution output by the model, denoted as . ,in, , This represents the current total number of categories. Calculate this... The average distribution of probability distributions Then, the KL divergence between each distribution and the mean distribution is calculated, and the average value is taken as the prediction stability index. ,in, Indicates the first In the enhanced version, the model prediction belongs to the category. The probability, It is a category in the average distribution The probability of. A smaller value indicates that the model's prediction of the image is more stable, meaning that the model has fully learned the image; a larger value indicates that the prediction is less stable, meaning that the model has not yet fully mastered the features of the image.

[0058] 2) For each newly added training image, its feature vector is compared with the feature vectors of all samples in the historical sample set. A density-based clustering algorithm is used to identify dense and sparse regions of feature distribution in the historical sample set. The local density value of the location of the feature vector of the newly added training image is calculated, and this local density value is used as the typicality index of the newly added training image in the feature space. The higher the typicality index, the more representative the image is of the distribution characteristics of its region. Specifically: First, the feature vectors of all samples in the historical sample set are obtained. The historical sample set stores representative samples selected from previous periods. The feature vector of each sample has been extracted using an optimized and trained large-scale recognition model and saved as follows. , The total number of historical samples is represented. The density-based clustering algorithm DBSCAN is used to perform density analysis on the feature vectors of historical samples by setting a neighborhood radius. and minimum sample size This identifies high-density and low-density regions in the feature space. In practical operation, All historical samples can be retrieved and their corresponding... The median of nearest neighbor distance, Typically, a smaller value, such as 5, is chosen. After clustering is complete, for the feature vector of each newly added training image... Calculate its The number of historical samples contained within the neighborhood is used as the local density value. And use this as a typical indicator. The larger the value, the more likely the new image is located in a dense area of ​​historical samples, and its visual features are representative of historical data. The smaller the value, the more likely the image is located in a sparse region, possibly belonging to a new category or a marginal distribution.

[0059] 3) For each new training image, a weighted combined score is constructed by integrating its prediction stability index and typicality index. The weight of the prediction stability index is set to 0.4, and the weight of the typicality index is set to 0.6. This prioritizes images with lower prediction stability (i.e., the model has not yet fully learned) that are located in high-density regions of the feature space. These images are crucial for strengthening the model's understanding of class boundaries. Specifically: To make the prediction stability index and the canonicity index numerically comparable, both indices are first normalized. This is done for the set of prediction stability indices for all newly added training images. Calculate the minimum value and maximum value The normalized instability index is defined as: The larger this value, the less stable the model's prediction for that image. For the typicality index set... Calculate the minimum value and maximum value The typicality index for normalization is defined as follows: A larger value indicates that the image is more representative. Then, a weighted combined score is calculated for each newly added training image: The weights of 0.4 and 0.6 respectively reflect the emphasis on instability and typicality, so that the high-scoring images not only have the characteristics of insufficient model learning, but are also located in the dense area of ​​historical samples, which helps to strengthen the model's cognition of class boundaries.

[0060] 4) Sort all newly added training images in the current period from highest to lowest weighted combined score, select a predetermined number of newly added training images with the highest scores as representative samples, and add these representative samples to the historical sample set for mixing with the newly added samples corresponding to the newly added training images in the next period. This strengthens the supplementary learning of the model's weak points while maintaining historical memory. Specifically: All newly added training images in the current period are scored using a weighted combination. Sort the samples from highest to lowest and select the preset number of samples with the highest scores. One new training image is used as a representative sample. The value of can be determined according to actual needs, such as selecting an average of 10 images per category or setting the total number to 50. These representative samples, along with their image data, feature vectors, and category labels, are added to the historical sample set to form an updated historical sample set, which is then used to mix with the new samples corresponding to the new training images in the next cycle. In this way, the historical sample set not only retains typical samples from the past but also dynamically introduces representative samples that the model has not yet fully learned in the current cycle, thereby effectively maintaining historical memory and specifically strengthening the weak links of the model during continuous online learning.

[0061] Optionally, the above technical solution also includes: S6. After optimizing and training the thawed model part with a preset learning rate, obtain the image to be recognized, input the image to be recognized into the optimized and trained recognition model, and use the optimized and trained recognition model to process the image to be recognized and output the category information corresponding to the image to be recognized. The specific implementation process is as follows: S60. Acquire one or more images to be recognized from an image acquisition device or storage medium. These images may be live footage captured by a camera, user-uploaded image files, or digital images from other sources. The acquired images to be recognized are stored in their original format, such as JPEG or PNG, and their size and pixel value range may vary.

[0062] S61. Preprocess the image to be recognized to meet the input requirements of the optimized and trained recognition model. The preprocessing process is completely consistent with the preprocessing of newly added training images during the training phase to ensure the consistency of data distribution. First, adjust the size of the image to be recognized to 224 pixels × 224 pixels. Use an image scaling algorithm. For non-square images, scale proportionally to make the shorter side reach 224 pixels, and then crop the longer side to obtain the standard size of 224 pixels × 224 pixels. Then, normalize the image pixel values ​​by dividing the integer value of each pixel by 255 to map it to a floating-point number range of 0 to 1. Next, based on pre-calculated statistical information from the large-scale image dataset, standardization was performed by subtracting the corresponding mean from each of the red, green, and blue color channels and dividing by the corresponding standard deviation. The mean for the red channel was 0.485, for the green channel 0.456, and for the blue channel 0.406; the standard deviation for the red channel was 0.229, for the green channel 0.224, and for the blue channel 0.225. The standardized pixel value distribution conforms to a standard normal distribution with a mean of 0 and a standard deviation of 1. Finally, the image data was converted from a three-dimensional array (height × width × number of channels) to a three-dimensional tensor (number of channels × height × width), resulting in an image tensor that can be directly input into the model.

[0063] S62. The optimized recognition model consists of a pre-trained DINOv2 model and a trained, randomly initialized classification head, at which point the model parameters are fixed. Before inference, the model is set to evaluation mode, which disables the random dropout operation used during training, ensuring that the result of each forward propagation is deterministic. The preprocessed image tensor is used as input and passed to the model's input layer.

[0064] S63. The image tensor sequentially passes through all transformer layers and the classification head in the model. First, the image tensor passes through the embedding layer and multiple transformer blocks. Each transformer block contains a multi-head self-attention mechanism and a feedforward neural network to progressively extract high-level semantic features of the image. Finally, the pre-training part outputs a feature vector, denoted as... ,in, This is the feature dimension. This feature vector is then input into the classification head, which is a linear layer, and its calculation process is as follows: ,in It is the weight matrix of the classification head. It is the bias vector of the classification head. This is the original score vector output. This represents the total number of categories that the model can currently recognize.

[0065] S64. Use the Softmax function to convert the original score vector. After converting to probability distributions for each category, the formula for calculating the Softmax function is: ,in, It is the first in the original score vector One element, The model predicts the category to which the image to be identified belongs. The probability of each category is 1. After obtaining the probability distribution, the category with the highest probability is selected as the predicted category, i.e.: , This is the category index to which the image to be identified is determined by the model.

[0066] S65. Index by Category The system retrieves the corresponding category name or category ID from a predefined category name mapping table as the final recognition result. The output can be in the form of text labels, numerical codes, or structured data, for subsequent application scenarios such as image review, content classification, or monitoring alarms. This completes the online recognition processing of the image to be recognized.

[0067] Example 1: Implement automatic classification of user-uploaded images, and learn online as new categories of images continuously emerge, avoiding frequent complete retraining. Specifically: S101. Preprocess the newly added training images within the current period to obtain new samples. Social media platforms collect images uploaded by users within a one-month period that require new categories or supplement existing categories as new training images. For each new training image, first adjust its size to 224 pixels × 224 pixels using an image scaling algorithm. If the image is not square, scale it proportionally to make the shorter side reach 224 pixels, then crop the longer side to the center. Then, normalize the pixel values ​​by dividing the integer value of each pixel by 255 to map it to the range of 0 to 1. Next, subtract the mean values ​​of 0.485, 0.456, and 0.406 from the red, green, and blue color channels respectively, and divide by the standard deviations of 0.229, 0.224, and 0.225. Finally, convert the image data from a three-dimensional array of height × width × number of channels into a three-dimensional tensor of number of channels × height × width to obtain the new samples.

[0068] S102. Extract a subset of samples from the historical sample set. Use the k-means clustering algorithm to select the center samples for each category. Mix the selected center samples with all newly added samples to form the augmented training dataset for the current period. The historical sample set stores representative samples selected from all previous periods. First, based on the list of categories involved in the newly added samples for the current period, extract a certain number of samples from the historical sample set by category, for example, 100 samples from each category. Then, for each sample extracted from a category, input it into the current large-scale recognition model to extract feature vectors. The feature vectors are taken from the output of the second-to-last layer of the model, with dimensions of [missing information]. For each category c, the set of feature vectors of all its samples. Perform k-means clustering, setting the number of clusters k=10, to obtain the cluster centers of each cluster. The sample closest to the center in each cluster is selected as the center sample for that class. Finally, all the selected center samples are merged with all the newly added samples in the current period to form the augmented training dataset.

[0069] S103. Add a classification head to the pre-trained large-scale recognition model and randomly initialize the parameters of the classification head to obtain the initialized recognition model. The large-scale recognition model uses the pre-trained DINOv2 model with 1.03B parameters. The classification head is a single-layer fully connected network, and its input dimension is the feature dimension output by the DINOv2 model. The output dimension is equal to the total number of categories. The parameters of the classification head include the weight matrix. and bias vector The Kaiming initialization method is used to initialize each element in the weight matrix. From a mean of 0 and a standard deviation of Random samples are taken from a normal distribution, and all bias vectors are initialized to 0.

[0070] S104. Freeze the parameters of all layers in the initialized recognition model except for the last two layers and the classification head. Unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the augmented training dataset of the current cycle at a preset learning rate. During the optimization training process, evaluate the recognition performance of historical categories to ensure that the forgetting rate of historical categories is below a preset threshold. In the model configuration, set the gradient calculation flag for all layers except the last two transformer blocks and the classification head to "not calculate gradient state," and these parameters remain unchanged during training. Set the gradient calculation flag for the last two transformer blocks and the classification head to "calculate gradient state." The optimizer is AdamW, and the learning rate is set to 10. -4 Before training begins, the baseline accuracy is calculated using the historical class validation set. During training, a mini-batch of samples is drawn from the augmented training dataset in each iteration, and the cross-entropy loss is calculated. ,in, For batch size, As a true label indicator for the sample, To predict probabilities, only the thawed parameters are updated after backpropagation. The accuracy on the historical class validation set is evaluated after each training round. Calculate the forgetting rate: If the forgetting rate exceeds 5%, stop training and roll back the parameters, or reduce the learning rate and continue training. Ultimately, ensure that the forgetting rate is below 5%.

[0071] S105. From the newly added training images in the current period, a predetermined number of image samples are uniformly extracted according to the category ratio. These extracted image samples are added as representative samples to the historical sample set to update the historical sample set, which is then used to mix with the newly added samples corresponding to the newly added training images in the next period. For each category in the newly added training images of the current period, the number of samples is counted. Each category is planned to be sampled A representative sample, if Then all will be sampled. Random samples will be drawn from each category. From a sample, a representative sample set is obtained. Add it to the historical sample set In this process, a new set of historical samples is formed. .

[0072] S106. After optimizing and training the thawed model part with a preset learning rate, the image to be recognized is obtained and input into the optimized and trained large-scale recognition model. The optimized and trained large-scale recognition model processes the image to be recognized and outputs the category information corresponding to the image to be recognized. When a user uploads an image to be classified, the image is first processed according to the preprocessing method in the first step to obtain an image tensor. Then, the image tensor is input into the optimized and trained large-scale recognition model, and the feature vector is obtained through forward propagation. Then, the raw score is calculated using the classification head. Then, the Softmax function is used to convert the scores into a probability distribution. Finally, select the category with the highest probability. The system predicts the category and outputs the corresponding category label, such as "landscape" or "people". This category information can be used for automatic image archiving, recommendation, or review.

[0073] Example 2: Real-time analysis of surveillance video streams to identify dangerous behaviors or objects, and online updates of the model as new safety requirements emerge.

[0074] S201. Preprocess the newly added training images within the current period to obtain new samples. The factory safety department collects images captured by surveillance cameras within a monthly cycle. These images contain hazard categories that need to be newly identified, such as "not wearing a safety helmet" and "open flame," and are used as new training images. For each new training image, it is first adjusted to 224 pixels × 224 pixels. If the image is not square, it is first scaled proportionally and then cropped in the center. Then, pixel values ​​are normalized by dividing the pixel values ​​by 255 to map them to 0 to 1. Then, the mean values ​​of the red, green, and blue channels are subtracted by 0.485, 0.456, and 0.406 respectively, and divided by the standard deviations of 0.229, 0.224, and 0.225. Finally, it is converted into a three-dimensional tensor of channel number × height × width to obtain the new samples.

[0075] S202. Extract a subset of samples from the historical sample set. Use the k-means clustering algorithm to select the center samples for each category. Mix the selected center samples with all new samples to form the augmented training dataset for the current period. The historical sample set stores representative hazardous scene images selected in previous periods. Based on the category of the new samples, extract samples of the corresponding category from the historical sample set. For each category of extracted samples, input the current large-scale recognition model to extract feature vectors, then perform k-means clustering with a cluster size of 10 to obtain the center of each cluster. Select the sample closest to the center as the center sample. Merge all center samples with the new samples from the current period to form the augmented training dataset.

[0076] S203. Add a classification head to the pre-trained large-scale recognition model and randomly initialize the parameters of the classification head to obtain the initialized recognition model. The large-scale recognition model also uses the pre-trained DINOv2 model, with the classification head being a fully connected layer. The input dimension is the model feature dimension, and the output dimension is the total number of danger categories to be identified. The classification head weights are initialized using Kaiming, and the bias is initialized to 0.

[0077] S204. Freeze all layer parameters in the initialized recognition model except for the last two layers and the classifier head. Unfreeze the last two layers and the classifier head, and optimize the unfrozen model part based on the augmented training dataset of the current cycle at a preset learning rate. During the optimization training process, evaluate the recognition performance of historical categories to ensure that the forgetting rate of historical categories is below a preset threshold. Freeze all layer parameters except for the last two transformer blocks and the classifier head, and unfreeze the last two layers and the classifier head. Use the AdamW optimizer with a learning rate of 10. -4 Before training, a baseline accuracy is calculated using a historical category validation set (e.g., images containing categories such as "helmet" and "flame"). During training, cross-entropy loss is calculated, and the unfrozen parameters are updated. After each training round, the historical category accuracy is evaluated, the forgetting rate is calculated, and it is ensured to be below 5%; otherwise, intervention measures are taken.

[0078] S205. From the newly added training images in the current period, a predetermined number of image samples are evenly extracted according to the category ratio. The extracted image samples are added to the historical sample set as representative samples to update the historical sample set, which is used to mix with the newly added training images corresponding to the next period. For each newly added danger category in the current period, the number of samples is counted. If the number of samples is not less than 10, 10 images are randomly selected; otherwise, all images are selected and added to the historical sample set as representative samples for mixing with new data for training in the next month.

[0079] S206. After optimizing and training the thawed model part with a preset learning rate, acquire the image to be recognized and input it into the optimized and trained recognition model. The optimized and trained recognition model processes the image to be recognized and outputs the category information corresponding to the image to be recognized. In real-time monitoring, key frames are extracted from the camera video stream as images to be recognized. First, each frame undergoes the same preprocessing as in the first step to obtain an image tensor. Then, it is input into the optimized and trained recognition model, and forward propagation yields feature vectors. The classification head outputs the raw score, and Softmax converts it into a probability. The category with the highest probability is taken as the recognition result. If the recognition result is a dangerous category such as "not wearing a safety helmet" or "open flame," an alarm is triggered, and the category information is marked on the monitoring screen, while the time and location are recorded. This alarm information can notify safety management personnel for timely handling.

[0080] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation. The scheme after adjusting the order is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0081] like Figure 2 As shown, an online learning system 200 for image recognition based on large model fine-tuning according to an embodiment of the present invention includes a preprocessing module 201, an extraction and clustering module 202, a model optimization module 203, a model training module 204, and a sampling and addition module 205. The preprocessing module 201 is used to: preprocess the newly added training images in the current period to obtain new samples; The extraction and clustering module 202 is used to: extract a portion of samples from the historical sample set, use a clustering algorithm to select the center samples of each category, and mix the selected center samples with all the newly added samples to form the augmented training dataset for the current period. The model optimization module 203 is used to: add a classification head to the pre-trained recognition model and randomly initialize the parameters of the classification head to obtain an initialized recognition model; The model training module 204 is used to: freeze the parameters of the initial recognition model except for the last two layers and the classification head, unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset at a preset learning rate, and evaluate the recognition performance of historical categories during the optimization training process to ensure that the forgetting rate of historical categories is lower than a preset threshold. The sampling addition module 205 is used to: sample representative samples from the newly added training images in the current period and add them to the historical sample set for sample mixing in the next period.

[0082] Optionally, in the above technical solution, the sampling addition module 205 is specifically used to: uniformly extract a preset number of image samples from the newly added training images in the current period according to the category ratio, and add the extracted image samples as representative samples to the historical sample set to update the historical sample set for mixing with the newly added samples corresponding to the newly added training images in the next period.

[0083] Optionally, in the above technical solution, the extraction clustering module 202 is specifically used to: use the k-means clustering algorithm to perform clustering processing on a portion of the samples extracted from the historical sample set, and select the samples of the cluster centers of each category as the center samples.

[0084] Optionally, the above technical solution also includes a model application module, which is specifically used to: after optimizing and training the thawed model part with a preset learning rate, obtain the image to be recognized, input the image to be recognized into the optimized and trained large recognition model, perform recognition processing on the image to be recognized through the optimized and trained large recognition model, and output the category information corresponding to the image to be recognized.

[0085] It should be noted that the beneficial effects of the online learning system 200 for image recognition based on large model fine-tuning provided in the above embodiments are the same as those of the online learning method for image recognition based on large model fine-tuning described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0086] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned online learning methods for image recognition based on large model fine-tuning.

[0087] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned online learning methods for image recognition based on large model fine-tuning.

[0088] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0089] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An online learning method for image recognition based on large model fine-tuning, characterized in that, include: The newly added training images in the current period are preprocessed to obtain new samples; A portion of samples are extracted from the historical sample set, and a clustering algorithm is used to select the center samples of each category. The selected center samples are then mixed with all the newly added samples to form the augmented training dataset for the current period. A classification head is added to the pre-trained recognition model, and the parameters of the classification head are randomly initialized to obtain an initialized recognition model. Freeze the parameters of the initial recognition model except for the last two layers and the classification head, unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset with a preset learning rate. During the optimization training process, evaluate the recognition performance of historical categories to ensure that the forgetting rate of historical categories is lower than a preset threshold. Representative samples are sampled from the newly added training images in the current period and added to the historical sample set for sample mixing in the next period.

2. The online learning method for image recognition based on large model fine-tuning according to claim 1, characterized in that, Representative samples are sampled from the newly added training images in the current period and added to the historical sample set for sample mixing in the next period, including: A preset number of image samples are uniformly extracted from the newly added training images in the current period according to the category ratio. The extracted image samples are added to the historical sample set as representative samples to update the historical sample set, which is then used to mix with the newly added samples corresponding to the newly added training images in the next period.

3. The online learning method for image recognition based on large model fine-tuning according to claim 1, characterized in that, Clustering algorithms are used to select the center samples for each category, including: The k-means clustering algorithm was used to cluster a subset of samples extracted from the historical sample set, and the samples at the cluster centers of each category were selected as the center samples.

4. An online learning method for image recognition based on large model fine-tuning according to any one of claims 1 to 3, characterized in that, Also includes: After optimizing and training the thawed model part with a preset learning rate, the image to be recognized is obtained, and the image to be recognized is input into the optimized and trained recognition model. The optimized and trained recognition model performs recognition processing on the image to be recognized and outputs the category information corresponding to the image to be recognized.

5. An online learning system for image recognition based on large model fine-tuning, characterized in that, It includes a preprocessing module, an extraction and clustering module, a model optimization module, a model training module, and a sampling and insertion module; The preprocessing module is used to: preprocess newly added training images in the current period to obtain new samples; The extraction and clustering module is used to: extract a portion of samples from the historical sample set, use a clustering algorithm to filter the center samples of each category, and mix the filtered center samples with all the newly added samples to form the augmented training dataset for the current period. The model optimization module is used to: add a classification head to the pre-trained recognition model and randomly initialize the parameters of the classification head to obtain an initialized recognition model; The model training module is used to: freeze the parameters of the layers other than the last two layers and the classification head in the initialized recognition model, unfreeze the last two layers and the classification head, and optimize the unfrozen model part based on the current period's augmented training dataset with a preset learning rate, and evaluate the recognition performance of historical categories during the optimization training process to ensure that the forgetting rate of historical categories is lower than a preset threshold. The sampling addition module is used to: sample representative samples from the newly added training images in the current period and add them to the historical sample set for sample mixing in the next period.

6. The online learning system for image recognition based on large model fine-tuning according to claim 5, characterized in that, The sampling addition module is specifically used to: uniformly extract a preset number of image samples from the newly added training images in the current period according to the category ratio, and add the extracted image samples as representative samples to the historical sample set to update the historical sample set, which is used to mix with the newly added samples corresponding to the newly added training images in the next period.

7. The online learning system for image recognition based on large model fine-tuning according to claim 5, characterized in that, The extraction and clustering module is specifically used to: use the k-means clustering algorithm to cluster a portion of the samples extracted from the historical sample set, and select the samples that are the cluster centers of each category as the center samples.

8. An online learning system for image recognition based on large model fine-tuning according to any one of claims 5 to 7, characterized in that, It also includes a model application module, which is specifically used to: after optimizing and training the thawed model part with a preset learning rate, obtain the image to be recognized, input the image to be recognized into the optimized and trained large recognition model, perform recognition processing on the image to be recognized through the optimized and trained large recognition model, and output the category information corresponding to the image to be recognized.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an online learning method for image recognition based on large model fine-tuning as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements an online learning method for image recognition based on large model fine-tuning as described in any one of claims 1 to 4.

Images