Model updating method and apparatus

Abstract

This invention provides a model update method and apparatus, belonging to the field of machine learning technology. The model update method includes: after training a product recognition model using a first training dataset, calculating the uncertainty and typical representativeness of each sample data in the first training dataset, wherein the typical representativeness is the cosine distance between the sample data and a first vector, and the first vector is the standard vector corresponding to the category to which the sample data belongs; selecting a first dataset from the first training dataset based on the uncertainty and typical representativeness of each sample data; obtaining a second training dataset; and retraining the product recognition model using the first dataset and the second training dataset. The technical solution of this invention can reduce the training time for model updates.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a model update method and apparatus. Background Technology

[0002] Deep learning-based product recognition models are widely used in business scenarios such as smart shelves and smart freezers due to their high accuracy. In practical applications, when product categories increase or packaging / product appearance is updated, digital image processing methods often need to be retrained to support new samples, and iterative updates of the model have high time and computing power costs. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a model update method and apparatus that can reduce the training time for model updates.

[0004] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0005] On the one hand, a model update method is provided, including:

[0006] After training the product recognition model using the first training dataset, the uncertainty and typical representativeness of each sample data in the first training dataset are calculated respectively. The first training dataset includes sample data of multiple categories. The typical representativeness is the cosine distance between the sample data and the first vector, which is the standard vector corresponding to the category to which the sample data belongs.

[0007] The first data set is selected from the first training data set based on the uncertainty and typical representativeness of each sample data set;

[0008] Obtain a second training data set, and use the first data set and the second training data set to train the product recognition model again. The second training data set includes newly added sample data of one or more categories, or newly added sample data belonging to at least one category among the multiple categories of the first training data set.

[0009] In some embodiments, selecting the first data set from the first training data set based on the uncertainty and typical representativeness of each sample data includes:

[0010] The sample data of each category in the first training data set are sorted from high to low according to the uncertainty, and the first N sample data are selected, where N is an integer greater than 1.

[0011] The sample data of each category in the first training data set are sorted from high to low according to the typical representativeness, and the first M sample data are selected, where M is an integer greater than 1.

[0012] The first data set is formed by using the N sample data and the M sample data of each category.

[0013] In some embodiments, the uncertainty in calculating sample data includes:

[0014] The sample data is augmented, and the data augmentation includes at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color jitter, random color depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion.

[0015] The product recognition model is used to predict the category of the data-enhanced sample data;

[0016] The uncertainty of the sample data is calculated based on the data augmented sample data and the predicted category.

[0017] In some embodiments, the method further includes the step of obtaining the first vector, which includes:

[0018] For each category, a feature queue of length S is established, which includes the feature vectors of the latest S training sample data, where S is an integer greater than 20;

[0019] Calculate the first mean of the feature vectors of the S training sample data;

[0020] After new training sample data is added to the feature queue, the second mean of the feature vectors of the S training sample data is recalculated.

[0021] The weighted sum of the first mean and the second mean is calculated as the first vector.

[0022] In some embodiments, retraining the product recognition model using the first dataset and the second training dataset includes:

[0023] The training data in the first dataset and the second training dataset are augmented and then input into the product recognition model.

[0024] During training, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of a first loss function and a second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and a first vector.

[0025] In some embodiments, N equals M.

[0026] Embodiments of the present invention also provide a model update apparatus, the model update apparatus comprising:

[0027] The processing module is used to calculate the uncertainty and typical representativeness of each sample data in the first training data set after training the commodity recognition model using the first training data set. The first training data set includes sample data of multiple categories, and the typical representativeness is the cosine distance between the sample data and the first vector, where the first vector is the standard vector corresponding to the category to which the sample data belongs.

[0028] The filtering module is used to filter out the first data set from the first training data set based on the uncertainty and typical representativeness of each sample data.

[0029] The training module is used to acquire a second training data set and to retrain the product recognition model using the first data set and the second training data set. The second training data set includes newly added sample data of one or more categories, or newly added sample data belonging to at least one category among the multiple categories in the first training data set.

[0030] In some embodiments, the filtering module includes:

[0031] The first screening unit is used to sort the sample data of each category in the first training data set according to the uncertainty from high to low, and select the first N sample data, where N is an integer greater than 1.

[0032] The second filtering unit is used to sort the sample data of each category in the first training data set from high to low according to the typical representativeness, and select the first M sample data, where M is an integer greater than 1.

[0033] The third processing unit is used to form the first data set using the N sample data and the M sample data of each category.

[0034] In some embodiments, the processing module is specifically used to perform data augmentation on the sample data, the data augmentation including at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color dithering, random color bit depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion; predicting the category of the data-augmented sample data using the product recognition model; and calculating the uncertainty of the sample data based on the data-augmented sample data and the predicted category.

[0035] In some embodiments, the apparatus further includes:

[0036] The acquisition module is used to establish a feature queue of length S for each category, the feature queue including the feature vectors of the latest S training sample data, where S is an integer greater than 20; calculate the first mean of the feature vectors of the S training sample data; after adding new training sample data to the feature queue, recalculate the second mean of the feature vectors of the S training sample data; and calculate the weighted sum of the first mean and the second mean as the first vector.

[0037] In some embodiments, the training module is specifically used to perform data augmentation on the training data in the first dataset and the second training dataset before inputting it into the product recognition model; during the training process, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer, wherein the loss function is a weighted sum of a first loss function and a second loss function, the first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and the first vector.

[0038] Embodiments of the present invention also provide a model update apparatus, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described above.

[0039] The embodiments of the present invention have the following beneficial effects:

[0040] In the above scheme, the sample data in the existing first training dataset is screened based on the uncertainty and typical representativeness of the sample data. The screened sample data and the new training data, namely the second training dataset, are used together to train the model. The catastrophic forgetting of the model is avoided by using sample replay. By selecting appropriate sample data, the model accuracy and training cost are balanced, which can ensure the training accuracy of the model, reduce the training time of the model, and reduce the update cost of the model. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the model update method according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the acquisition of training data in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the training of the model according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the process for updating the model according to an embodiment of the present invention;

[0045] Figure 5 This is a structural block diagram of the model update device according to an embodiment of the present invention. Detailed Implementation

[0046] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0047] The technical problem to be solved by the present invention is to provide a model update method and apparatus that can reduce the training time for model updates.

[0048] Embodiments of the present invention provide a model update method, such as... Figure 1 As shown, it includes:

[0049] Step 101: After training the product recognition model using the first training dataset, calculate the uncertainty and typical representativeness of each sample data in the first training dataset. The first training dataset includes sample data of multiple categories. The typical representativeness is the cosine distance between the sample data and the first vector. The first vector is the standard vector corresponding to the category to which the sample data belongs.

[0050] In this embodiment, the product recognition model can identify the category of a product based on its image. The input of the product recognition model is the image of the product, and the output is the probability value of each category.

[0051] In practical applications, on the one hand, the appearance of target categories such as goods will change over time, such as short-term packaging design changes and long-term product appearance design updates; on the other hand, new categories are added due to application scenarios or model misidentification optimization. Therefore, it is believed that the update requirements of the product recognition model mainly come from two aspects: 1. The addition of new product categories (class increment), such as newly added product categories such as potato chips, cola, spicy strips, etc.; 2. Changes in the data domain (domain increment), including short-term packaging design changes and long-term product appearance design updates.

[0052] When new categories are added to the sample data or the data domain changes, the product recognition model needs to be updated. To avoid catastrophic forgetting of the product recognition model, it can be fine-tuned by combining existing training data with new training data. However, when the amount of old training data is huge, the cost of updating the model is high. Therefore, the old training data can be filtered based on its uncertainty and representativeness, and the filtered sample data can be used in conjunction with the new training data to train the product recognition model.

[0053] In this embodiment, the first training dataset is the old training data, meaning the current product recognition model is the model obtained after training using the first training dataset. The first training dataset includes multiple sample data, each of which includes an image and label of the product. The second training dataset is the new training data, including images and labels of newly added product categories, images and labels of products with changed packaging designs, images and labels of products with changed product shapes, etc.

[0054] Since training a product recognition model multiple times on simple sample data cannot effectively improve its recognition accuracy, a certain number of samples with high uncertainty are selected and combined with new training data for training. In this embodiment, the Regression-Raphson (RM) algorithm can be used to filter out samples with high uncertainty. The RM algorithm believes that after sample data is transformed, such as by occlusion or rotation, the larger the variance of the model's prediction result, the more difficult the sample data is, and the higher the uncertainty. However, if noisy samples and difficult samples account for a large proportion of the training data, it will cause instability in model learning. Therefore, in this embodiment, highly representative samples are also selected from the old training data and combined with new training data for training. In this embodiment, a prototype vector (i.e., the first vector) is constructed for each target category. Sample data whose features are closer to the prototype vector are considered to be more representative of the category. In this embodiment, for each category of sample data in the old training data, samples with high uncertainty and samples with high representativeness can be selected and combined with new training data for training. In each category, the number of samples with high certainty can be equal to the number of samples with high representativeness.

[0055] In some embodiments, the uncertainty in calculating sample data includes:

[0056] The sample data is augmented, and the data augmentation includes at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color jitter, random color depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion.

[0057] The product recognition model is used to predict the category of the data-enhanced sample data;

[0058] The uncertainty of the sample data is calculated based on the data augmented sample data and the predicted category.

[0059] A stable sample dataset should consistently produce correct results after different data augmentation transformations. The more errors, the more unstable the sample data is considered, and the greater the classification difficulty. In this embodiment, the Rainbow-memory algorithm is used for 12 data augmentation transformations. The uncertainty of the sample data is calculated by predicting the number of errors using an existing product recognition model. The sample data is then sorted from highest to lowest uncertainty for further selection. The formula for calculating the uncertainty U(x) is as follows:

[0060]

[0061] Where T = 12 represents the number of data augmentation transformations, I c The function returns 1 when the input equals the true class c of the sample data, and 0 otherwise. This represents the input image after the t-th data augmentation transformation of the sample data. To be The category output by the current product recognition model after inputting the current product recognition model. The data augmentation method used in this embodiment can be autoAugment, which is an accumulation of 14 data transformation methods, including horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color dithering, random color bit depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion.

[0062] To obtain the typical representativeness of the sample data, a prototype vector can be maintained for each known category. Sample data within a category that is closer to the prototype vector is considered to have higher typical representativeness. Cosine similarity can be used to measure the distance between sample features and the prototype vector. The typical representativeness of the sample data is R(x) = 1 - dist(f). c (x),β i ), where β i For the prototype vector, the sample feature f c (x) represents the features extracted after inputting sample data x into the current product recognition model. The feature dimension is determined by the structure of the product recognition model. dist() calculates the sample features f. c (x) and prototype vector β i The cosine distance between them.

[0063] In this embodiment, the product recognition model uses multiple convolutional layers to extract features from the images in the input sample data, obtaining feature vectors. These vectors are then input into a classification head to predict the category probability value. For each known category i, a prototype vector β is maintained using the final features extracted by the backbone network. i .

[0064] To ensure the reliability of the prototype vectors, it is necessary to guarantee class segmentation in the feature space, that is, the distance between samples of the same class should be as close as possible, while the distance between samples of dissimilar classes should be as far apart as possible. To achieve this, the contrastive loss should be minimized during training.

[0065]

[0066]

[0067] This can encourage the product recognition model to learn in the direction of reducing intra-class distance and increasing inter-class distance, especially to make the inter-class distance greater than the threshold θ.

[0068] Theoretically, the prototype vector β of category i i The first vector should be the mean of all sample features within category i after the product recognition model has been trained. However, during model training, the model weights are constantly updated, and the calculated feature vector is unstable. Therefore, in this embodiment, obtaining the first vector includes:

[0069] For each category, a feature queue of length S is established, which includes the feature vectors of the latest S training sample data, where S is an integer greater than 20; the first mean of the feature vectors of the S training sample data is calculated; after new training sample data is added to the feature queue, the second mean of the feature vectors of the S training sample data is recalculated; the weighted sum of the first mean and the second mean is calculated as the first vector.

[0070] Specifically, during model training, a fixed-length feature queue F = [q1, q2, ..., q] is maintained for each category. s To store the feature vectors of the latest S sample data for each category, the mean β of the feature vectors for that category in the feature queue is then taken. i_new In terms of momentum β i =ηβ i +(1-η)β i_new Update the prototype vector, where η is a pre-set parameter that can be adjusted as needed. In the early stages of product recognition model training, the model is unstable. Therefore, after the product recognition model has reached a certain number of iterations N1, the prototype vector is calculated for the first time based on the feature queue F, and the calculation of L begins. cont Loss (otherwise L) cont=0), and then every N2 iterations, update the prototype vector β. i .

[0071] This embodiment constructs prototype vectors for each category and selects representative sample data from old training data. This can solve the problem of unstable model training caused by a large proportion of noisy and difficult samples, and improve the training accuracy of the model.

[0072] Step 102: Select a first data set from the first training data set based on the uncertainty and typical representativeness of each sample data set;

[0073] Specifically, after calculating the uncertainty of each sample data, the sample data of each category in the first training data set can be sorted from high to low according to the uncertainty, and the first N sample data can be selected, where N is an integer greater than 1 and N is a preset value that can be adjusted as needed.

[0074] After calculating the typical representativeness of each sample data, the sample data of each category in the first training data set can be sorted from high to low according to the typical representativeness, and the first M sample data are selected, where M is an integer greater than 1.

[0075] The first data set is composed of the N sample data and the M sample data of each category, and the first data set can be stored in a local buffer.

[0076] In some embodiments, N can be equal to M; of course, N may not be equal to M. When N equals M, both the uncertainty and representativeness of the sample data can be considered. Specifically, when N equals M, after setting the number of sample data K that the buffer can store, the average is applied across all categories C, i.e., the same number of sample data k = K / C is selected for each existing category. The k / 2 = N sample data points with the highest uncertainty and the k / 2 = M sample data points with the highest representativeness are selected from each category. If the number of sample data points for a certain category is less than k, then repeated sampling is performed.

[0077] Step 103: Obtain a second training data set, and retrain the product recognition model using the first data set and the second training data set. The second training data set includes newly added sample data of one or more categories, or newly added sample data belonging to at least one category among the multiple categories in the first training data set.

[0078] The second training dataset consists of newly added training data, including images and labels of products in newly added categories, products with changed packaging designs, and products with changed product shapes. The second training dataset may include sample data from newly added categories, or it may include new data from existing categories, where the existing categories are the categories of the sample data in the first training dataset.

[0079] When retraining the product recognition model, to balance model accuracy and training time, the number of sample data in the first dataset is equal to the number of sample data in the second training dataset. That is, in each batch of training data, such as... Figure 2 As shown, half of the training data comes from newly added data, and half comes from previously used training data. Of course, the ratio between the number of samples in the first dataset and the number of samples in the second training dataset can be adjusted according to the actual situation, and the balance between new data and existing old data needs to be considered.

[0080] Specifically, when training the product recognition model, the training data in the first dataset and the second training dataset can be augmented and then input into the product recognition model. During the training process, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of the first loss function and the second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and the first vector.

[0081] In this embodiment, as Figure 3 As shown, the product recognition model can specifically adopt ReXNet (Rank Expansion Network). ReXNet is a lightweight model, which is conducive to terminal deployment. Specifically, ReXNet addresses the bottleneck of feature representation (intermediate layers such as pooling layers compress feature dimensions, causing feature loss) by expanding the number of input channels and designing multiple expansion layers (layers with more output channels than input channels), so that the number of channels gradually increases, avoiding a large difference between the rank of the input dimension and the rank of the output dimension.

[0082] For sample data of goods or commodities, the features within the category are relatively fixed (goods have relatively fixed appearance and shape), and the main variations come from occlusion, lighting, deformation, and collection angle. Therefore, in order to enrich the training data, data augmentation methods such as random cropping, random horizontal flipping, color jittering, and CutMix can be used to augment the training data. Then, the data augmented training data is input into the commodity recognition model for training.

[0083] During training, the parameters of the product recognition model are adjusted based on feedback from the loss function and the optimizer.

[0084] Wherein, the loss function L total The calculation formulas for cross-entropy loss and contrastive loss are as follows:

[0085]

[0086] Where y i The label for the sample data is p, where p equals 1 if the true class is i, and 0 otherwise. i Let α be the predicted probability that the sample data belongs to category i, and let α be the hyperparameter of the balanced contrast loss. It is worth noting that the above formula calculates the loss of a single sample data.

[0087] During the initial training of the product recognition model, the optimizer uses the ASAM algorithm. The ASAM optimizer finds a region with lower loss, rather than a point, through two backward calculations. The ASAM algorithm is not prone to overfitting, so extending the training period can achieve higher accuracy. However, for the model fine-tuning stage, the SGD algorithm is used to control the training time.

[0088] In this embodiment, the sample data in the existing first training dataset is screened based on the uncertainty and typical representativeness of the sample data. The screened sample data and the new training data, namely the second training dataset, are used together to train the model. The catastrophic forgetting of the model is avoided by using sample replay. By selecting appropriate sample data, the model accuracy and training cost are balanced, which can ensure the training accuracy of the model, reduce the training time of the model, and reduce the update cost of the model.

[0089] In one specific embodiment, such as Figure 4 As shown, model updates can specifically include the following steps:

[0090] Step 1: Obtain the first training dataset;

[0091] The first training dataset can be the training data used for the first training of the product recognition model. The first training dataset includes multiple sample data, each of which includes an image of the product and its label.

[0092] Step 2: Train the product recognition model using the first training dataset;

[0093] Step 3: Determine if the model needs to be updated;

[0094] When a new product category is added or the data domain changes, the product recognition model needs to be updated.

[0095] Step 4: Determine if the local buffer is empty. If it is, proceed to step 5; otherwise, proceed to step 10.

[0096] The Buffer is used to store the filtered sample data, which is used to calculate the uncertainty and typicality of the old training data, and is selected from the old training data based on the uncertainty and typicality.

[0097] Step 5: Import the current product recognition model and use the product recognition model to calculate the uncertainty and typical representativeness of each sample data in the first training dataset;

[0098] Step 6: Filter the sample data in the first training dataset and put the filtered sample data into the Buffer;

[0099] If the Buffer is empty, calculate the uncertainty and typicality of the sample data in the old training data (i.e. the first training data set). Based on the uncertainty and typicality, select sample data with high uncertainty and sample data with high typicality from the old training data to form the first data set and put it into the Buffer.

[0100] Step 10: If the Buffer is not empty, read the data in the local Buffer and proceed to step 7;

[0101] Step 7: Update the product recognition model using the second training dataset;

[0102] The second training dataset consists of newly added training data, including images and labels of newly added product categories, images and labels of products with changed packaging designs, and images and labels of products with changed product shapes.

[0103] The product recognition model is trained and updated using training data from the first dataset and the second training dataset.

[0104] Step 8: After updating the product recognition model, the first data set and the second training data set become the new old training data. Calculate the uncertainty and typical representativeness of the sample data in the first data set and the second training data set, and select sample data from the first data set and the second training data set based on the uncertainty and typical representativeness.

[0105] Step 9: Update the data in the Buffer using the sample data selected in Step 8.

[0106] Specifically, the existing data in the Buffer can be cleared, and the sample data selected in step 8 can be stored in the Buffer.

[0107] In this embodiment, the product recognition model is trained and updated using the old training data and new training data stored in the buffer. This reduces the training time for model updates, combats catastrophic forgetting, and lowers the time and computing cost of model updates.

[0108] Embodiments of the present invention also provide a model update apparatus, such as Figure 5 As shown, the model update device includes:

[0109] Processing module 21 is used to calculate the uncertainty and typical representativeness of each sample data in the first training data set after training the commodity recognition model using the first training data set. The first training data set includes sample data of multiple categories. The typical representativeness is the cosine distance between the sample data and the first vector, and the first vector is the standard vector corresponding to the category to which the sample data belongs.

[0110] In this embodiment, the product recognition model can identify the category of a product based on its image. The input of the product recognition model is the image of the product, and the output is the probability value of each category.

[0111] In practical applications, on the one hand, the appearance of target categories such as goods will change over time, such as short-term packaging design changes and long-term product appearance design updates; on the other hand, new categories are added due to application scenarios or model misidentification optimization. Therefore, it is believed that the update requirements of the product recognition model mainly come from two aspects: 1. The addition of new product categories (class increment), such as newly added product categories such as potato chips, cola, spicy strips, etc.; 2. Changes in the data domain (domain increment), including short-term packaging design changes and long-term product appearance design updates.

[0112] When new categories are added to the sample data or the data domain changes, the product recognition model needs to be updated. To avoid catastrophic forgetting of the product recognition model, it can be fine-tuned by combining existing training data with new training data. However, when the amount of old training data is huge, the cost of updating the model is high. Therefore, the old training data can be filtered based on its uncertainty and representativeness, and the filtered sample data can be used in conjunction with the new training data to train the product recognition model.

[0113] In this embodiment, the first training dataset is the old training data, meaning the current product recognition model is the model obtained after training using the first training dataset. The first training dataset includes multiple sample data, each of which includes an image and label of the product. The second training dataset is the new training data, including images and labels of newly added product categories, images and labels of products with changed packaging designs, images and labels of products with changed product shapes, etc.

[0114] Since training a product recognition model multiple times on simple sample data cannot effectively improve its recognition accuracy, a certain number of samples with high uncertainty are selected and combined with new training data for training. In this embodiment, the Regression-Raphson (RM) algorithm can be used to filter out samples with high uncertainty. The RM algorithm believes that after sample data is transformed, such as by occlusion or rotation, the larger the variance of the model's prediction result, the more difficult the sample data is, and the higher the uncertainty. However, if noisy samples and difficult samples account for a large proportion of the training data, it will cause instability in model learning. Therefore, in this embodiment, highly representative samples are also selected from the old training data and combined with new training data for training. In this embodiment, a prototype vector (i.e., the first vector) is constructed for each target category. Sample data whose features are closer to the prototype vector are considered to be more representative of the category. In this embodiment, for each category of sample data in the old training data, samples with high uncertainty and samples with high representativeness can be selected and combined with new training data for training. In each category, the number of samples with high certainty can be equal to the number of samples with high representativeness.

[0115] In some embodiments, the processing module 21 is specifically used to perform data augmentation on the sample data, the data augmentation including at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color dithering, random color bit depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion; predicting the category of the data-augmented sample data using the product recognition model; and calculating the uncertainty of the sample data based on the data-augmented sample data and the predicted category.

[0116] A stable sample dataset should consistently produce correct results after different data augmentation transformations. The more errors, the more unstable the sample data is considered, and the greater the classification difficulty. In this embodiment, the Rainbow-memory algorithm is used for 12 data augmentation transformations. The uncertainty of the sample data is calculated by predicting the number of errors using an existing product recognition model. The sample data is then sorted from highest to lowest uncertainty for further selection. The formula for calculating the uncertainty U(x) is as follows:

[0117]

[0118] Where T = 12 represents the number of data augmentation transformations, I c The function returns 1 when the input equals the true class c of the sample data, and 0 otherwise. This represents the input image after the t-th data augmentation transformation of the sample data. To be The category output by the current product recognition model after inputting the current product recognition model. The data augmentation method used in this embodiment can be autoAugment, which is an accumulation of 14 data transformation methods, including horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color dithering, random color bit depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion.

[0119] To obtain the typical representativeness of the sample data, a prototype vector can be maintained for each known category. Sample data within a category that is closer to the prototype vector is considered to have higher typical representativeness. Cosine similarity can be used to measure the distance between sample features and the prototype vector. The typical representativeness of the sample data is R(x) = 1 - dist(f). c (x),β i ), where β i For the prototype vector, the sample feature f c (x) represents the features extracted after inputting sample data x into the current product recognition model. The feature dimension is determined by the structure of the product recognition model. dist() calculates the sample features f. c (x) and prototype vector β i The cosine distance between them.

[0120] In this embodiment, the product recognition model uses multiple convolutional layers to extract features from the images in the input sample data, obtaining feature vectors. These vectors are then input into a classification head to predict the category probability value. For each known category i, a prototype vector β is maintained using the final features extracted by the backbone network. i .

[0121] To ensure the reliability of the prototype vectors, it is necessary to guarantee class segmentation in the feature space, that is, the distance between samples of the same class should be as close as possible, while the distance between samples of dissimilar classes should be as far apart as possible. To achieve this, the contrastive loss should be minimized during training.

[0122]

[0123]

[0124] This can encourage the product recognition model to learn in the direction of reducing intra-class distance and increasing inter-class distance, especially to make the inter-class distance greater than the threshold θ.

[0125] Theoretically, the prototype vector β of category i i This should be the mean of all sample features within category i after the product recognition model has been trained. However, during model training, the model weights are constantly updated, and the calculated feature vectors are unstable. Therefore, the model update device in this embodiment further includes:

[0126] The acquisition module is used to establish a feature queue of length S for each category, the feature queue including the feature vectors of the latest S training sample data, where S is an integer greater than 20; calculate the first mean of the feature vectors of the S training sample data; after adding new training sample data to the feature queue, recalculate the second mean of the feature vectors of the S training sample data; and calculate the weighted sum of the first mean and the second mean as the first vector.

[0127] Specifically, during model training, a fixed-length feature queue F = [q1, q2, ..., q] is maintained for each category. s To store the feature vectors of the latest S sample data for each category, the mean β of the feature vectors for that category in the feature queue is then taken. i_new In terms of momentum β i =ηβ i +(1-η)β i_new Update the prototype vector, where η is a pre-set parameter that can be adjusted as needed. In the early stages of product recognition model training, the model is unstable. Therefore, after the product recognition model has reached a certain number of iterations N1, the prototype vector is calculated for the first time based on the feature queue F, and the calculation of L begins. cont Loss (otherwise L) cont =0), and then every N2 iterations, update the prototype vector β. i .

[0128] This embodiment constructs prototype vectors for each category and selects representative sample data from old training data. This can solve the problem of unstable model training caused by a large proportion of noisy and difficult samples, and improve the training accuracy of the model.

[0129] The filtering module 22 is used to filter out the first data set from the first training data set based on the uncertainty and typical representativeness of each sample data.

[0130] In some embodiments, the filtering module 22 includes:

[0131] The first screening unit is used to sort the sample data of each category in the first training data set according to the uncertainty from high to low, and select the first N sample data, where N is an integer greater than 1.

[0132] The second filtering unit is used to sort the sample data of each category in the first training data set from high to low according to the typical representativeness, and select the first M sample data, where M is an integer greater than 1.

[0133] The third processing unit is used to form the first data set using the N sample data and the M sample data of each category.

[0134] Specifically, after calculating the uncertainty of each sample data, the sample data of each category in the first training data set can be sorted from high to low according to the uncertainty, and the first N sample data are selected, where N is an integer greater than 1 and is a preset value that can be adjusted as needed; after calculating the typical representativeness of each sample data, the sample data of each category in the first training data set can be sorted from high to low according to the typical representativeness, and the first M sample data are selected, where M is an integer greater than 1; the first data set is composed of the N sample data and the M sample data of each category, and the first data set can be stored in a local buffer.

[0135] In some embodiments, N can be equal to M; of course, N may not be equal to M. When N equals M, both the uncertainty and representativeness of the sample data can be considered. Specifically, when N equals M, after setting the number of sample data K that the buffer can store, the average is applied across all categories C, i.e., the same number of sample data k = K / C is selected for each existing category. The k / 2 = N sample data points with the highest uncertainty and the k / 2 = M sample data points with the highest representativeness are selected from each category. If the number of sample data points for a certain category is less than k, then repeated sampling is performed.

[0136] Training module 23 is used to acquire a second training data set and retrain the product recognition model using the first data set and the second training data set. The second training data set includes newly added sample data of one or more categories, or newly added sample data belonging to at least one category among the multiple categories in the first training data set.

[0137] In some embodiments, the training module 23 is specifically used to perform data augmentation on the training data in the first dataset and the second training dataset and then input it into the product recognition model; during the training process, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of a first loss function and a second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and the first vector.

[0138] The second training dataset consists of newly added training data, including images and labels of newly added product categories, images and labels of products with changed packaging designs, and images and labels of products with changed product shapes.

[0139] When retraining the product recognition model, to balance model accuracy and training time, the number of sample data in the first dataset is equal to the number of sample data in the second training dataset. That is, in each batch of training data, such as... Figure 2 As shown, half of the training data comes from newly added data, and half comes from previously used training data.

[0140] Specifically, when training the product recognition model, the training data in the first dataset and the second training dataset can be augmented and then input into the product recognition model. During the training process, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of the first loss function and the second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and the first vector.

[0141] In this embodiment, as Figure 3 As shown, the product recognition model can specifically adopt ReXNet (Rank Expansion Network). ReXNet is a lightweight model, which is conducive to terminal deployment. Specifically, ReXNet addresses the bottleneck of feature representation (intermediate layers such as pooling layers compress feature dimensions, causing feature loss) by expanding the number of input channels and designing multiple expansion layers (layers with more output channels than input channels), so that the number of channels gradually increases, avoiding a large difference between the rank of the input dimension and the rank of the output dimension.

[0142] For sample data of goods or commodities, the features within the category are relatively fixed (goods have relatively fixed appearance and shape), and the main variations come from occlusion, lighting, deformation, and collection angle. Therefore, in order to enrich the training data, data augmentation methods such as random cropping, random horizontal flipping, color jittering, and CutMix can be used to augment the training data. Then, the data augmented training data is input into the commodity recognition model for training.

[0143] During training, the parameters of the product recognition model are adjusted based on feedback from the loss function and the optimizer.

[0144] Wherein, the loss function L total The calculation formulas for cross-entropy loss and contrastive loss are as follows:

[0145]

[0146] Where y i The label for the sample data is p, where p equals 1 if the true class is i, and 0 otherwise. i Let α be the predicted probability that the sample data belongs to category i, and let α be the hyperparameter of the balanced contrast loss. It is worth noting that the above formula calculates the loss of a single sample data.

[0147] During the initial training of the product recognition model, the optimizer uses the ASAM algorithm. The ASAM optimizer finds a region with lower loss, rather than a point, through two backward calculations. The ASAM algorithm is not prone to overfitting, so extending the training period can achieve higher accuracy. However, for the model fine-tuning stage, the SGD algorithm is used to control the training time.

[0148] In this embodiment, the sample data in the existing first training dataset is screened based on the uncertainty and typical representativeness of the sample data. The screened sample data and the new training data, namely the second training dataset, are used together to train the model. The catastrophic forgetting of the model is avoided by using sample replay. By selecting appropriate sample data, the model accuracy and training cost are balanced, which can ensure the training accuracy of the model, reduce the training time of the model, and reduce the update cost of the model.

[0149] Embodiments of the present invention also provide a model update apparatus, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described above.

[0150] In the various method embodiments of the present invention, the sequence numbers of each step are not intended to limit the order of the steps. For those skilled in the art, any changes in the order of the steps without creative effort are also within the scope of protection of the present invention.

[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0153] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model update method, characterized in that, The model update method includes: After training the product recognition model using the first training dataset, the uncertainty and typical representativeness of each sample data in the first training dataset are calculated respectively. The first training dataset includes sample data of multiple categories. The typical representativeness is the cosine distance between the sample data and the first vector. The first vector is the standard vector corresponding to the category to which the sample data belongs. The sample data in the first training dataset includes product images and labels. The product recognition model is used to identify the category of the product based on the product image. The first data set is selected from the first training data set based on the uncertainty and typical representativeness of each sample data set; Obtain a second training data set, and use the first data set and the second training data set to train the product recognition model again. The second training data set includes sample data of one or more newly added categories, or newly added sample data of at least one category among the multiple categories of the first training data set. The training data in the second training data set includes images and labels of products of newly added categories, images and labels of products with changed packaging designs, and images and labels of products with changed product shapes. The retrained product recognition model is used to identify the category of a product based on its image. The input of the product recognition model is the image of the product, and the output is the probability value of each category. Among them, the calculation uncertainty The formula is as follows: T represents the number of data augmentation transformations. The function returns 1 when the input equals the true class c of the sample data, and 0 otherwise. This represents the input image after the t-th data augmentation transformation of the sample data. To be The category output by the current product recognition model after inputting the current product recognition model.

2. The model update method according to claim 1, characterized in that, The step of selecting the first data set from the first training data set based on the uncertainty and representativeness of each sample data includes: The sample data of each category in the first training data set are sorted from high to low according to the uncertainty, and the first N sample data are selected, where N is an integer greater than 1. The sample data of each category in the first training data set are sorted from high to low according to the typical representativeness, and the first M sample data are selected, where M is an integer greater than 1. The first data set is formed by using the N sample data and the M sample data of each category.

3. The model update method according to claim 1, characterized in that, The uncertainty in calculating sample data includes: The sample data is augmented, and the data augmentation includes at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color jitter, random color depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion. The product recognition model is used to predict the category of the data-enhanced sample data; The uncertainty of the sample data is calculated based on the data augmented sample data and the predicted category.

4. The model update method according to claim 1, characterized in that, The method further includes a step of obtaining the first vector, wherein obtaining the first vector includes: For each category, a feature queue of length S is established, which includes the feature vectors of the latest S training sample data, where S is an integer greater than 20; Calculate the first mean of the feature vectors of the S training sample data; After new training sample data is added to the feature queue, the second mean of the feature vectors of the S training sample data is recalculated. The weighted sum of the first mean and the second mean is calculated as the first vector.

5. The model update method according to claim 1, characterized in that, The step of retraining the product recognition model using the first dataset and the second training dataset includes: The training data in the first dataset and the second training dataset are augmented and then input into the product recognition model. During training, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of a first loss function and a second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and a first vector.

6. The model update method according to claim 2, characterized in that, N equals M.

7. A model update device, characterized in that, The model update device includes: The processing module is used to calculate the uncertainty and typical representativeness of each sample data in the first training data set after training the product recognition model using the first training data set. The first training data set includes sample data of multiple categories. The typical representativeness is the cosine distance between the sample data and the first vector. The first vector is the standard vector corresponding to the category to which the sample data belongs. The sample data in the first training data set includes product images and labels. The product recognition model is used to identify the category of the product based on the product image. The filtering module is used to filter out the first data set from the first training data set based on the uncertainty and typical representativeness of each sample data. The training module is used to acquire a second training data set and retrain the product recognition model using the first data set and the second training data set. The second training data set includes sample data of one or more newly added categories, or newly added sample data of at least one category among the multiple categories in the first training data set. The training data in the second training data set includes images and labels of products of newly added categories, images and labels of products with changed packaging designs, and images and labels of products with changed product shapes. The retrained product recognition model is used to identify the category of a product based on its image. The input of the product recognition model is the image of the product, and the output is the probability value of each category. Among them, the calculation uncertainty The formula is as follows: T represents the number of data augmentation transformations. The function returns 1 when the input equals the true class c of the sample data, and 0 otherwise. This represents the input image after the t-th data augmentation transformation of the sample data. To be The category output by the current product recognition model after inputting the current product recognition model.

8. The model update apparatus according to claim 7, characterized in that, The filtering module includes: The first screening unit is used to sort the sample data of each category in the first training data set according to the uncertainty from high to low, and select the first N sample data, where N is an integer greater than 1. The second filtering unit is used to sort the sample data of each category in the first training data set from high to low according to the typical representativeness, and select the first M sample data, where M is an integer greater than 1. The third processing unit is used to form the first data set using the N sample data and the M sample data of each category.

9. The model update apparatus according to claim 7, characterized in that, The processing module is specifically used to perform data augmentation on the sample data. The data augmentation includes at least one of the following: horizontal cropping, vertical cropping, horizontal translation, vertical translation, random rotation, color dithering, random color depth adjustment, random exposure adjustment, random contrast adjustment, random sharpening, random brightness adjustment, contrast maximization, histogram equalization, and pixel value inversion; predicting the category of the data-augmented sample data using the product recognition model; and calculating the uncertainty of the sample data based on the data-augmented sample data and the predicted category.

10. The model update apparatus according to claim 7, characterized in that, The device further includes: The acquisition module is used to establish a feature queue of length S for each category, the feature queue including the feature vectors of the latest S training sample data, where S is an integer greater than 20; calculate the first mean of the feature vectors of the S training sample data; after adding new training sample data to the feature queue, recalculate the second mean of the feature vectors of the S training sample data; and calculate the weighted sum of the first mean and the second mean as the first vector.

11. The model update apparatus according to claim 7, characterized in that, The training module is specifically used to perform data augmentation on the training data in the first dataset and the second training dataset, and then input the data into the product recognition model. During the training process, the parameters of the product recognition model are adjusted using the loss function and the output of the optimizer. The loss function is a weighted sum of the first loss function and the second loss function. The first loss function is calculated based on the labels of the sample data and the output of the product recognition model, and the second loss function is calculated based on the cosine distance between the output of the product recognition model and the first vector.

12. A model update device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-6.

Images