Model adaptive updating method and device during model reasoning

Abstract

The invention provides a model adaptive updating method and device during model reasoning, and the method comprises the steps: a database stores data samples obtained during image classification model reasoning, and the data samples comprise data features and corresponding category probabilities; neighbors of the current data sample are obtained by retrieving the cosine distance between the data features of the current data sample and the data features of the other data samples, the probability of each category of all the neighbors is averaged to obtain the average probability of each category, and the category with the highest average probability is taken as a pseudo label of the current data sample; and constructing a loss function according to the data samples and the corresponding pseudo labels to update and train the image classification model, and performing image classification on the to-be-classified images by using the updated and trained image classification model to obtain an image classification result of the to-be-classified images. Compared with an existing adaptive technology during model reasoning, model adjustment is more accurate and effective, and the reasoning accuracy of the model in different scenes is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of edge computing and terminal-side model training, and in particular to a model adaptive updating method, device, electronic device, computer-readable storage medium and computer program product during model reasoning. Background Art

[0002] Deep neural networks (DNNs) show strong performance when training and test data share similar distributions. However, due to different scenarios, sensor differences, or weather changes, the actual data encountered after the model is deployed often has a large difference from the distribution of training data. In this case, the model generally has a significant decrease in accuracy. Collecting and annotating data for new scenarios and retraining the model requires a lot of manpower and time costs, which hinders the expansion and adaptation of the model in different scenarios.

[0003] As a prominent paradigm to alleviate this problem, time-to-reasoning adaptation (TTA) has received considerable attention. After the model infers the test data, the TTA method adjusts the model online while outputting the results, without the need to collect and label a large amount of data in the new domain. However, the current methods have limited accuracy improvement and need to be improved and enhanced.

[0004] Most TTA methods focus on the self-training mode, which uses the source model to perform reasoning on real data to generate pseudo labels (PL) and use them to guide the update of the model itself. Since pseudo labels inevitably contain noise, incorporating them into model updates will lead to error accumulation within the model. Some methods use the maximum softmax probability or the entropy of the predicted probability to evaluate the reliability of pseudo labels, and set a higher threshold to filter out unreliable pseudo labels.

[0005] This type of method has the following problems. First, the softmax score tends to combine the feature norm and angular distance. In addition to adjusting the angular feature embedding and classifier weights between the two, increasing the norm of the output feature can also improve the softmax score. This feature makes the model overconfident, so that samples with a high risk of misprediction are included in the model update, resulting in poor accuracy after model adjustment. Secondly, the existing methods only use the global feature information obtained from the source model training, without considering the local feature information of the actual data, which is not conducive to the model adapting to the actual data distribution. Finally, the reliability of pseudo-labels in the existing methods is coupled with the difficulty of samples. The screened samples are all samples that are easy for the model to predict, which is of little help to the model update and is not conducive to the model to quickly adapt to the actual data. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the above-mentioned prior art has poor adaptive effect during model reasoning due to noise pseudo-labels, and proposes a framework for adaptive update during model reasoning based on neighborhood information.

[0007] In view of the deficiencies in the prior art, the present invention Figure 2 As shown, a method for adaptively updating a model during model reasoning is proposed, and the device includes:

[0008] Initial step, obtain the pre-trained image classification model;

[0009] The storage step is to store the data sample obtained when the image classification model is inferred in the database, and the data sample includes: data features and their corresponding category probabilities;

[0010] The label generation step is to retrieve the cosine distance between the data features of the current data sample and the data features of the remaining data samples to obtain the neighbors of the current data sample, and to average the probabilities of each category of all neighbors to obtain the average probability of each category, and to take the category with the highest average probability as the pseudo label of the current data sample;

[0011] The model updating step constructs a loss function based on the data sample and its corresponding pseudo-label to update the image classification model, and uses the updated and trained image classification model to classify the image to be classified to obtain the image classification result of the image to be classified.

[0012] The method for adaptively updating the model during model reasoning comprises the following initial steps:

[0013] The image classification model is decomposed into a backbone network for extracting data features and a classification layer for predicting class probabilities, and the classification layer calculates the output probability based on the angular distance:

[0014] The angular distance A(w,z) is calculated by the following formula:

[0015]

[0016] The predicted probability p of the kth class k for:

[0017]

[0018] Where w is the weight matrix of the classification layer h, w i and w k are the i-th and k-th row vectors of the weight matrix respectively, and z is the data feature output by the backbone network.

[0019] The method for adaptively updating the model during model reasoning, wherein the storage step comprises:

[0020] Scoring the data sample based on the storage time and reliability of the data sample, and removing data samples with scores below a threshold when updating the database;

[0021] The calculation process of the score α of this data sample is:

[0022] α=min(t0-t,1 / exp(H(p)-H0)

[0023] Where t is the storage time, s is the reliability; p is the category probability output by the image classification model;

[0024] H(p)=-∑ i p i logp i , H0=logC, t0=2C; C is the total number of image categories that can be output by the image classification model; p i is the probability of the i-th category output by the image classification model.

[0025] The model adaptive updating method during model inference, wherein the loss function in the model updating step is for:

[0026]

[0027] x is a data sample, is a pseudo label, w is a weight;

[0028] w represents the following:

[0029]

[0030] p i is the probability of the i-th category output by the image classification model; w is the weight matrix of the classification layer of the image classification model, w i and w k are the i-th and k-th row vectors of the weight matrix respectively.

[0031] like Figure 3 As shown, the present invention also proposes a model adaptive updating device during model reasoning, the device comprising:

[0032] Initial device, obtains the pre-trained image classification model;

[0033] The storage device stores the data samples obtained when the image classification model is inferred in the database, and the data samples include: data features and their corresponding category probabilities;

[0034] The label generating device retrieves the cosine distance between the data feature of the current data sample and the data features of the remaining data samples to obtain the neighbors of the current data sample, averages the probabilities of each category of all neighbors to obtain the average probability of each category, and takes the category with the highest average probability as the pseudo label of the current data sample;

[0035] The model updating device constructs a loss function according to the data sample and its corresponding pseudo-label to update the image classification model, and uses the updated and trained image classification model to classify the image to be classified to obtain the image classification result of the image to be classified.

[0036] The model adaptive updating device during model reasoning, the initial device comprises:

[0037] The image classification model is decomposed into a backbone network for extracting data features and a classification layer for predicting class probabilities, and the classification layer calculates the output probability based on the angular distance:

[0038] The angular distance A(w,z) is calculated by the following formula:

[0039]

[0040] The predicted probability p of the kth class k for:

[0041]

[0042] Where w is the weight matrix of the classification layer h, w i and w k are the i-th and k-th row vectors of the weight matrix respectively, and z is the data feature output by the backbone network.

[0043] The model adaptive updating device during model reasoning, the storage device comprises:

[0044] Scoring the data sample based on the storage time and reliability of the data sample, and removing data samples with scores below a threshold when updating the database;

[0045] The calculation process of the score α of this data sample is:

[0046] α=min(t0-t, 1 / exp(H(p)-H0)

[0047] Where t is the storage time, s is the reliability; p is the category probability output by the image classification model;

[0048] H(p)=-∑ i p i logp i , H0=logC, t0=2C; C is the total number of image categories that can be output by the image classification model; p i is the probability of the i-th category output by the image classification model;

[0049] The loss function in the model updating device for:

[0050]

[0051] x is a data sample, is a pseudo label, w is a weight;

[0052] w represents the following:

[0053]

[0054] p i is the probability of the i-th category output by the image classification model; w is the weight matrix of the classification layer of the image classification model, w i and w k are the i-th and k-th row vectors of the weight matrix respectively.

[0055] The present invention also proposes an electronic device, which includes a model adaptive updating device during model inference. The electronic device may be connected to an information display device, and the information display device is used to display the image classification results according to display parameters and attributes set by the user or through an artificial intelligence model.

[0056] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the model adaptive updating method during model reasoning are implemented.

[0057] The present invention also proposes a computer program product, including a computer program, which implements the steps of the model adaptive updating method during model reasoning when the computer program is executed by a processor.

[0058] It can be seen from the above scheme that the advantages of the present invention are:

[0059] This invention proposes a framework for adaptive updating during model inference based on neighborhood information. This framework adjusts the model online while the model is inferring, without the need to manually collect and annotate a large amount of new data. Compared with existing adaptive technologies during model inference, model adjustment is more accurate and effective, greatly improving the model's inference accuracy in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a framework flow chart of the present invention;

[0061] Figure 2 is a flow chart of the method of the present invention;

[0062] Figure 3 It is a module diagram of the device of the present invention;

[0063] Figure 4 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0064] Figure 5 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0065] Figure 6 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0066] Reference numerals:

[0067] A-First electronic device;

[0068] B-Model adaptive updating device during model reasoning;

[0069] C-data acquisition equipment;

[0070] D-information display device;

[0071] 1000 - second electronic device;

[0072] Ⅰ-computational unit;

[0073] II-ROM;

[0074] III-RAM;

[0075] IV-bus;

[0076] V-interface;

[0077] VI - input unit;

[0078] VII-output unit;

[0079] VIII- Storage medium;

[0080] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0081] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0082] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0083] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0084] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0085] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0086] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0087] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0088] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0089] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0090] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0091] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0095] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0096] When the inventor was working on a model deployment project, he found that after the model was deployed, the effect was often not as expected because the data distribution of the actual scene was different from the training data. It is impractical to collect labeled data for each application scenario to retrain the model, so the inventor tried to study the adaptive method of model reasoning so that it can be automatically adjusted after the model is deployed. After studying the self-training technology of the model with pseudo labels, the inventor proposed an adaptive framework for model reasoning based on neighborhood information. The prediction results given by the model on new data in turn guide the update of the model to adapt to the data distribution of the new scene. The framework first constructs a feature storage to cache a certain amount of new data features and model prediction results to characterize the feature distribution of the new data, and then uses the nearest neighbor method to generate pseudo labels using local information to guide model updates. In order to reduce the impact of pseudo-label noise, the framework replaces the original softmax output of the model with the prediction probability based on the angular distance, and weights the samples based on the confidence of the label and the difficulty of the data sample, and updates the model to adapt to the new data distribution. The framework can adaptively update model parameters while model reasoning to improve the prediction accuracy of the model.

[0097] The process of generating pseudo labels using local information takes advantage of the smoothing assumption: data with similar features have the same label. We use the features of the current sample to calculate the cosine similarity with the features in the repository, take the 5 samples with the highest similarity, use their probability average as the predicted probability of the current sample, and further obtain the onehot label. Local information: information about a few samples with the most similar features, relative to global information Global information: for example, the category center calculated using the data features of the entire dataset.

[0098] In order to achieve the above technical effects, the present invention proposes the following key technical points:

[0099] Key point 1: Pseudo-labeling technology based on the nearest neighbor algorithm; by building a feature repository to cache the data feature vectors and prediction results during inference, when inferring new data, similar samples are searched from the repository, and the prediction results of these samples are combined to generate more accurate labels.

[0100] Key point 2: Output probability calculation method based on angular distance; using angular distance to calculate the output category probability of the model eliminates the influence of the feature vector norm on the prediction confidence, and can also be used to calculate the sample difficulty.

[0101] Key point 3, sample weighting method based on pseudo-label confidence and sample difficulty; the framework differentiates the contribution of samples to model updates through sample weighting, which can better utilize the information of new data to adjust the model and improve the accuracy of the model. Confidence is used to measure the reliability (accuracy) of the pseudo-label of the present invention. Here, the entropy of the average probability prediction calculated in the pseudo-label generation process is used to measure it. The smaller the entropy, the more reliable it is. Difficulty is used to measure the difficulty of classifying the current sample by the model (for example, samples close to the classification surface are more difficult to classify accurately, and samples far away from the classification surface are easier). The present invention uses the ADH formula proposed in the previous section to measure. The larger the value, the more difficult the sample.

[0102] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0103] The overall structure of this framework is as follows Figure 1 As shown in the figure. When the adaptive function is turned on, for each batch of new data, the framework first obtains the features extracted by the backbone network and the probability output of the classification layer, where the classification layer has been replaced with angular distance predictions. Then, the framework updates the features and probability outputs of its current data to the repository, and retrieves neighboring samples in the repository based on the features extracted from the current data, and uses the average of the outputs stored by these neighboring samples to generate pseudo labels. Finally, the framework uses the angular distance and average probability output to calculate the weighting coefficients and updates the model by minimizing the weighted loss.

[0104] Process 1: Classification layer replacement

[0105] In the present invention, given a pre-trained model f, we decompose the model into a backbone network g and a classification layer h, i.e., f = h·g. The backbone network g maps unlabeled test data x to feature representation z = g(x)∈R D , h represents the classifier mapping z to the predicted probability output p = h(z)∈R C , D and C represent the feature dimension and the number of categories. Usually the model uses softmax to calculate the probability of category k:

[0106]

[0107] where w k is the kth row vector of the weight matrix in the classifier h. The present invention uses the model output based on angular distance instead of the traditional softmax probability, because the angular distance is a more accurate measure of the uncertainty of the prediction, and the angular distance is calculated by the following formula:

[0108]

[0109] A smaller angular distance means a higher probability that the data belongs to that category, so the k-th category prediction probability can be calculated as:

[0110]

[0111] We replace the classification layer with predictions based on angular distance.

[0112] Process 2: Repository Construction

[0113] The present invention maintains a storage repository with a capacity of N, which stores N / C samples for each class. Specifically, the storage body stores the features and predicted probabilities of new data encountered after deployment j is the sample number, z j is the feature of the jth sample and p j is the classification probability of the jth sample, and the dimension of the classification probability is C, corresponding to the probability of each class. The repository is initialized with the classifier weight. When updating the memory library, we consider the timeliness of the data and the reliability of the predicted probability. Since data with high uncertainty predictions are prone to provide incorrect category information, the repository only retains data with reliable predictions. In addition, outdated data will weaken the ability of the model to adapt to new distributions, so it will be replaced first. The present invention uses a heuristic strategy to assign a score α(t,s) to each data in the repository, where t is the storage time of the sample, initialized to 0, and increased by 1 for each update, and s is the reliability of the entropy calculation predicted by the model. After the new sample reasoning is completed, it is updated to the repository. If it is not full, it is directly stored. If the repository is full, some samples need to be replaced. If the current sample score is lower than the sample scores in the repository, it will not be replaced, and the current sample will not be stored in the repository; if there is data in the repository with a lower score than the current sample, the low-scoring data will be replaced first, and the current data with a high score will be stored first (there may be multiple data in a batch), and the data in the repository will be guaranteed to have a higher score after the update. The score of the data can be calculated by combining the timeline and reliability:

[0114] α(t,s)=min(t0-t,1 / exp)(H(p)-H0)

[0115] Where p is the model output, entropy can be calculated as H(p) = -∑ i p i logp i , H0=logC, t0=2C are hyperparameters that control the sample storage time.

[0116] Process 3: Pseudo-label generation

[0117] The present invention uses the local neighboring features of the data sample as a template, and gradually guides the feature space to the target domain, and realizes pseudo label generation through nearest neighbor soft voting. Neighbors are retrieved by calculating the cosine distance between the feature vector z of the current data and the features stored in the library, and K data with the lowest cosine distance in the storage volume are selected as neighbors. Then, the probability of each category is calculated by averaging the probability output of the selected neighbors, and the category with the highest probability is taken as the pseudo label of the data. By aggregating the knowledge of nearby samples, the present invention can obtain more accurate label estimation and further update the model through self-supervised training.

[0118] Step 4: Sample Weighting

[0119] Pseudo labels cannot be completely regarded as true labels. Blindly trusting pseudo labels may cause incorrect model updates. The present invention weights the loss by the reliability of pseudo labels and the difficulty of the data. Since we have changed the model output, the present invention proposes angular distance loss, which is defined as the NLL loss on the model output based on angular distance:

[0120]

[0121] If the category of a data sample is difficult to predict and the pseudo-label is reliable, then improving the model's prediction of the data can effectively improve the model's generalization ability. Weighted loss Different data contribute differently to the model update. Formally, we express the weight w as follows, by dividing the sum of the first and second terms of the following formula by 2, so that the value of w ranges from 0 to 1:

[0122]

[0123] The first term assigns greater weight to the probability output with low entropy, which is obtained by averaging the predictions of neighboring samples in process 3. If the predicted probabilities of neighbors are consistent, the derived pseudo-label is more reliable and has low entropy. The second term measures the difficulty of the model to predict the current data. It is calculated by the normalized angular distance between the continuously updated classifier weights and the extracted data features:

[0124]

[0125] A high value means that the model does not predict the current data well.

[0126] Process 5: Model update

[0127] During the operation of the model, the intermediate features and model outputs calculated in the forward process are used to minimize the weighted loss calculated in process 4, and the gradient is calculated and the model parameters are updated through a reverse calculation process. In order to minimize the additional computational burden, the framework does not repeatedly calculate the probability output of the current data, but outputs the model prediction results before the reverse calculation, and the updated model is directly used for the prediction of new data.

[0128] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0129] like Figure 3 As shown, the present invention also proposes a model adaptive updating device during model reasoning, the device comprising:

[0130] Initial device, obtains the pre-trained image classification model;

[0131] The storage device stores the data samples obtained when the image classification model is inferred in the database, and the data samples include: data features and their corresponding category probabilities;

[0132] The label generating device retrieves the cosine distance between the data feature of the current data sample and the data features of the remaining data samples to obtain the neighbors of the current data sample, averages the probabilities of each category of all neighbors to obtain the average probability of each category, and takes the category with the highest average probability as the pseudo label of the current data sample;

[0133] The model updating device constructs a loss function according to the data sample and its corresponding pseudo-label to update the image classification model, and uses the updated and trained image classification model to classify the image to be classified to obtain the image classification result of the image to be classified.

[0134] The model adaptive updating device during model reasoning, the initial device comprises:

[0135] The image classification model is decomposed into a backbone network for extracting data features and a classification layer for predicting class probabilities, and the classification layer calculates the output probability based on the angular distance:

[0136] The angular distance A(w,z) is calculated by the following formula:

[0137]

[0138] The predicted probability p of the kth class k for:

[0139]

[0140] Where w is the weight matrix of the classification layer h, w i and w k are the i-th and k-th row vectors of the weight matrix respectively, and z is the data feature output by the backbone network.

[0141] The model adaptive updating device during model reasoning, the storage device comprises:

[0142] Scoring the data sample based on the storage time and reliability of the data sample, and removing data samples with scores below a threshold when updating the database;

[0143] The calculation process of the score α of this data sample is:

[0144] α=min(t0-t, 1 / exp(H(p)-H0)

[0145] Where t is the storage time, s is the reliability; p is the category probability output by the image classification model;

[0146] H(p)=-∑ i p i logp i , H0=logC, t0=2C; C is the total number of image categories that can be output by the image classification model; p i is the probability of the i-th category output by the image classification model;

[0147] The loss function in the model updating device for:

[0148]

[0149] x is a data sample, is a pseudo label, w is a weight;

[0150] w represents the following:

[0151]

[0152] p i is the probability of the i-th category output by the image classification model; w is the weight matrix of the classification layer of the image classification model, w i and w k are the i-th and k-th row vectors of the weight matrix respectively.

[0153] like Figure 4 As shown, the present invention further proposes a first electronic device A in another embodiment, including the model adaptive updating device during model reasoning.

[0154] like Figure 5 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain the image classification model and the image to be classified, and the information display device D is used to display the image classification results obtained by the analysis of the present invention.

[0155] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, such as the specified image classification results, etc. The user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0156] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the model adaptive update method during model reasoning provided by the above methods.

[0157] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the model adaptive updating method when executing the model reasoning. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0158] Figure XX shows a schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0159] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0160] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0162] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for adaptively updating a model during model inference, characterized in that: include: Initial step, obtain the pre-trained image classification model; The storage step is to store the data sample obtained when the image classification model is inferred in the database, and the data sample includes: data features and their corresponding category probabilities; The label generation step is to retrieve the cosine distance between the data features of the current data sample and the data features of the remaining data samples to obtain the neighbors of the current data sample, and to average the probabilities of each category of all neighbors to obtain the average probability of each category, and to take the category with the highest average probability as the pseudo label of the current data sample; The model updating step constructs a loss function based on the data sample and its corresponding pseudo-label to update the image classification model, and uses the updated and trained image classification model to classify the image to be classified to obtain the image classification result of the image to be classified.

2. The method for adaptively updating a model during model inference according to claim 1, characterized in that: This initial step includes: The image classification model is decomposed into a backbone network for extracting data features and a classification layer for predicting class probabilities, and the classification layer calculates the output probability based on the angular distance: The angular distance A(w,z) is calculated by the following formula: The predicted probability p of the kth class k for: Where w is the weight matrix of the classification layer h, w i and w k are the i-th and k-th row vectors of the weight matrix respectively, and z is the data feature output by the backbone network.

3. The method for adaptively updating a model during model inference according to claim 1, characterized in that: The storage step includes: Scoring the data sample based on the storage time and reliability of the data sample, and removing data samples with scores below a threshold when updating the database; The calculation process of the score α of this data sample is: α=min(t0-t,1 / exp(H(p)-H0) Where t is the storage time, s is the reliability; p is the category probability output by the image classification model; H(p)=-∑ i p i logp i , H0=logC, t0=2C; C is the total number of image categories that can be output by the image classification model; p i is the probability of the i-th category output by the image classification model.

4. The method for adaptively updating a model during model reasoning according to claim 2 or 3, characterized in that: The loss function in the model update step for: x is a data sample, is a pseudo label, w is a weight; w represents the following: p i is the probability of the i-th category output by the image classification model; w is the weight matrix of the classification layer of the image classification model, w i and w k are the i-th and k-th row vectors of the weight matrix respectively.

5. A model adaptive updating device during model inference, characterized in that: include: Initial device, obtains the pre-trained image classification model; The storage device stores the data samples obtained when the image classification model is inferred in the database, and the data samples include: data features and their corresponding category probabilities; The label generating device retrieves the cosine distance between the data feature of the current data sample and the data features of the remaining data samples to obtain the neighbors of the current data sample, averages the probabilities of each category of all neighbors to obtain the average probability of each category, and takes the category with the highest average probability as the pseudo label of the current data sample; The model updating device constructs a loss function according to the data sample and its corresponding pseudo-label to update the image classification model, and uses the updated and trained image classification model to classify the image to be classified to obtain the image classification result of the image to be classified.

6. The model adaptive updating device during model reasoning according to claim 5, characterized in that: The initial set-up includes: The image classification model is decomposed into a backbone network for extracting data features and a classification layer for predicting class probabilities, and the classification layer calculates the output probability based on the angular distance: The angular distance A(w,z) is calculated by the following formula: The predicted probability p of the kth class k for: Where w is the weight matrix of the classification layer h, w i and w k are the i-th and k-th row vectors of the weight matrix respectively, and z is the data feature output by the backbone network.

7. The model adaptive updating device during model inference according to claim 5, characterized in that: The storage device comprises: Scoring the data sample based on the storage time and reliability of the data sample, and removing data samples with scores below a threshold when updating the database; The calculation process of the score α of this data sample is: α=min(t0-t,1 / exp(H(p)-H0) Where t is the storage time, s is the reliability; p is the category probability output by the image classification model; H(p)=-∑ i p i logp i , H0=logC, t0=2C; C is the total number of image categories that can be output by the image classification model; p i is the probability of the i-th category output by the image classification model; The loss function in the model updating device for: x is a data sample, is a pseudo label, w is a weight; w represents the following: p i is the probability of the i-th category output by the image classification model; w is the weight matrix of the classification layer of the image classification model, w i and w k are the i-th and k-th row vectors of the weight matrix respectively.

8. An electronic device, characterized in that: It includes a device as described in claims 5-7, and the electronic device is connected to an information display device, and the information display device is used to display the image classification result according to the display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images