Model training, image recognition method, device, equipment and storage medium

Abstract

The present disclosure relates to a model training method and device, an image recognition method and device, equipment and a storage medium. The present disclosure extracts historical knowledge from a historical image set by a machine learning model to be trained; and spreads the historical knowledge to feature mean values of various sample images in a first sub-set in the historical image set, and spreads the historical knowledge to feature information of each sample image in a second sub-set in the historical image set; and trains the machine learning model according to a similarity between the updated feature information of each sample image in the second sub-set and the updated feature mean values of various sample images in the first sub-set, so that the trained machine learning model can spread the historical knowledge to new category images, thereby enhancing or supplementing feature information of the new category images, and according to the enhanced or supplemented feature information of the new category images, the recognition accuracy of the new category images can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a model training, image recognition method, apparatus, device, and storage medium. Background Technology

[0002] Supported by massive amounts of data and powerful computing capabilities, machine learning models, such as deep learning models, have achieved tremendous success in the field of image recognition. Currently, to avoid catastrophic forgetting and overfitting problems after training, machine learning models are typically frozen, meaning their parameters are fixed.

[0003] However, freezing a trained machine learning model can cause it to become inaccurate in recognizing new categories of images. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a model training method, image recognition method, apparatus, device, and storage medium to improve the recognition accuracy of new categories of images.

[0005] In a first aspect, embodiments of this disclosure provide a model training method, including:

[0006] Extracting historical knowledge from a collection of historical images using a machine learning model to be trained;

[0007] Using the machine learning model to be trained, the feature mean of each type of sample image in the first subset of the historical image set is calculated, and the feature information of each sample image in the second subset of the historical image set is calculated.

[0008] The historical knowledge is propagated to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, and the historical knowledge is propagated to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset.

[0009] The machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the average updated feature values ​​of each type of sample image in the first subset.

[0010] Secondly, embodiments of this disclosure provide an image recognition method, including:

[0011] Obtain the mean value of the first feature corresponding to each historical category among multiple historical categories. The mean value of the first feature corresponding to each historical category is the mean value of the image features of multiple historical images under the historical category.

[0012] The machine learning model is trained to calculate the target image features of the images to be identified in the test set, and to calculate the mean of the image features of multiple sample images under each image category in the training set, thereby obtaining the mean of the third feature corresponding to each image category in the training set. The machine learning model is trained using the method described in the first aspect.

[0013] The target image features are updated based on the mean value of the first feature corresponding to each historical category to obtain the updated target image features.

[0014] Based on the mean value of the first feature corresponding to each historical category, the mean value of the third feature corresponding to each image category in the training set is updated to obtain the updated mean value of the third feature corresponding to each image category in the training set.

[0015] The target category corresponding to the content in the image to be identified is determined based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set.

[0016] Thirdly, embodiments of this disclosure provide a few-sample incremental image recognition method based on knowledge propagation, including:

[0017] Extracting historical knowledge from historical image collections;

[0018] Calculate the feature mean of each type of sample image in the training set, and calculate the target image features of the image to be identified in the test set;

[0019] The historical knowledge is propagated to the feature mean of various sample images in the training set to obtain the updated feature mean of various sample images in the training set, and the historical knowledge is propagated to the target image feature of the image to be identified to obtain the updated target image feature of the image to be identified.

[0020] The target category corresponding to the content in the image to be identified is determined based on the similarity between the updated target image features of the image to be identified and the updated mean features of various sample images in the training set.

[0021] Fourthly, embodiments of this disclosure provide an electronic device, including:

[0022] Memory;

[0023] Processor; and

[0024] Computer programs;

[0025] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first, second, or third aspect.

[0026] Fifthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in the first, second, or third aspects.

[0027] The model training, image recognition method, apparatus, device, and storage medium provided in this disclosure extract historical knowledge from a historical image set through a machine learning model to be trained; calculate the feature mean of each type of sample image in a first subset of the historical image set, and calculate the feature information of each sample image in a second subset of the historical image set through the machine learning model to be trained; propagate the historical knowledge to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, and propagate the historical knowledge to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset; train the machine learning model based on the similarity between the updated feature information of each sample image in the second subset and the updated feature mean of each type of sample image in the first subset, so that the trained machine learning model can propagate historical knowledge to new category images, thereby enhancing or supplementing the feature information of new category images, and improving the recognition accuracy of new category images based on the enhanced or supplemented feature information of new category images. Therefore, even if the machine learning model is frozen, the accuracy of recognizing new categories of images can still be improved by using the machine learning model that has been trained to do so when new categories of images appear. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0029] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart of the model training method provided in this embodiment of the disclosure;

[0031] Figure 2A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0032] Figure 3 A schematic diagram illustrating model training provided in an embodiment of this disclosure;

[0033] Figure 4 A flowchart of a model training method provided in another embodiment of this disclosure;

[0034] Figure 5 A schematic diagram of model reasoning provided for another embodiment of this disclosure;

[0035] Figure 6 A schematic diagram of a knowledge dissemination module provided in another embodiment of this disclosure;

[0036] Figure 7 This is a schematic diagram of the structure of the model training device provided in the embodiments of this disclosure;

[0037] Figure 8 This is a schematic diagram of the structure of the image recognition device provided in the embodiments of this disclosure;

[0038] Figure 9 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0039] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0040] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0041] It should be noted that the image data involved in this application (including but not limited to historical images, sample images, images to be identified, etc.) is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0042] In addition, the model training method or image recognition method provided in this application involves the following explanations of terms, detailed below:

[0043] Lifelong learning: Lifelong learning is an advanced machine learning paradigm that accumulates knowledge from past tasks through continuous learning and uses this knowledge to aid future learning.

[0044] Incremental learning: In incremental learning, new categories are constantly emerging, and the model needs to correctly classify the input into its corresponding category. There is no overlap between the categories contained in different tasks.

[0045] Few-sample incremental class learning: Incremental class learning involves very few new class training samples, such as only 5 samples.

[0046] Catastrophic forgetting: After the model learns new knowledge, it almost completely forgets the features and information learned during previous training.

[0047] Overfitting: When there are too few training samples, the model overfits the training samples, resulting in poor classification performance on test samples.

[0048] Image recognition: Assigning corresponding category labels to different image content.

[0049] Prototype features: usually refers to the mean of features of all samples in each category.

[0050] Supported by massive amounts of data and powerful computing capabilities, machine learning models, such as deep learning models, have achieved tremendous success in image recognition. However, the performance of a trained machine learning model (e.g., the number of image categories it can recognize) is limited by the image categories contained in the training set. In real-world image recognition scenarios, images of new categories (e.g., categories the machine learning model has not yet recognized) constantly emerge, posing a challenge to the model. This challenge is often referred to as class incremental learning. Specifically, class incremental learning requires not only that the machine learning model can recognize images of new categories, but also that it prevents the model from forgetting the recognition ability of images of old categories. However, due to privacy concerns, device storage limitations, and high storage costs, images of old categories (also known as old category data) are often unavailable, or only a small number can be retained. This makes machine learning models prone to catastrophic forgetting problems (i.e., a significant decrease in the ability to recognize images of old categories). Existing class incremental learning methods typically assume that images of new categories are sufficient. However, in some scenarios, such as rare bird recognition or automotive part defect recognition, the number of new bird images or automotive part defect images is often very small, making some existing class incremental learning methods ineffective in such scenarios. Therefore, few-shot incremental learning emerged. However, in few-shot incremental learning, the limited number of new category images not only exacerbates the catastrophic forgetting problem but also easily leads to overfitting of the machine learning model to the training data. To avoid catastrophic forgetting and overfitting, the trained machine learning model is usually frozen, i.e., its parameters are fixed. However, freezing the trained machine learning model can cause it to become inaccurate in recognizing new category images.

[0051] To address this problem, this disclosure provides a model training method, which will be described below with reference to specific embodiments.

[0052] Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this disclosure. The method can be executed by a model training device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Furthermore, this model training method is applicable to... Figure 2 The application scenarios shown are as follows: Figure 2 As shown, this application scenario includes a terminal 21 and a server 22. The server 22 can train a machine learning model using the model training method described in this embodiment. Furthermore, the trained machine learning model is deployed to the terminal 21, enabling the terminal 21 to perform image recognition based on the trained model. Alternatively, the terminal 21 can send the image to be recognized to the server 22, allowing the server 22 to perform image recognition. Or, the server 22 can perform image recognition on images from local storage, other terminals, or other servers. Figure 1 As shown, the specific steps of this method are as follows:

[0053] S101. Extract historical knowledge from the historical image set using a machine learning model to be trained.

[0054] For example, a collection of historical images is like... Figure 3 The old training data shown is used to extract relevant knowledge from the historical image set through the machine learning model to be trained. This relevant knowledge can be recorded as historical knowledge.

[0055] Optionally, the historical image set includes multiple historical images under each of multiple historical categories; extracting historical knowledge from the historical image set using a machine learning model to be trained includes: for each of the multiple historical categories, calculating the mean of image features of multiple historical images under the historical category using the machine learning model to be trained, to obtain the first feature mean corresponding to the historical category.

[0056] like Figure 3 As shown, the machine learning model includes a pre-trained backbone network and a knowledge propagation module. The pre-trained backbone network can be an encoder, used to encode the image into a low-dimensional representation vector. Specifically, the pre-trained backbone network is obtained by pre-training a backbone network, such as a deep residual network like ResNet-18, using traditional image classification methods. Before training the machine learning model, it is necessary to perform pre-training as follows: Figure 3As shown in step 1, library construction is performed. Specifically, relevant knowledge is extracted from the old class training data through a pre-trained backbone network. The old class training data can be a large number of images from old categories, which can be denoted as historical categories. There can be multiple historical categories, and each historical category can contain multiple historical images. For example, this embodiment involves 100 historical categories, each containing 10 historical images. Therefore, as shown in step 1... Figure 3 The training data shown consists of 1000 historical images. The pre-trained backbone network can calculate the image features of each of these 1000 historical images. The image features of each historical image can include not only the feature information of the target object (e.g., a horse, a tiger, etc.) in the historical image (e.g., the tiger's stripe information, the horse's body structure information), but also some feature information of the historical image itself, such as the number of target objects in the historical image, background features, etc. In this embodiment, the image feature of each historical image can be a low-dimensional representation vector, such as a 256-dimensional representation vector. Furthermore, the pre-trained backbone network can also calculate the average image feature of multiple historical images under each historical category, thereby obtaining the first feature mean corresponding to that historical category. For example, a certain historical category includes 10 historical images. Each of these 10 historical images corresponds to one image feature, meaning there are 10 image features for each of the 10 historical images. Further, the mean of these 10 image features is calculated; this mean is also a 256-dimensional representation vector. This mean is denoted as the first feature mean corresponding to that historical category. Therefore, in this embodiment, the relevant knowledge extracted by the pre-trained backbone network from the old class training data can include not only the image features of each of the 1000 historical images, but also the first feature mean corresponding to each of the 100 historical categories. Figure 3 As shown, a knowledge base can be constructed, and the relevant knowledge can be stored in the knowledge base. This embodiment can also initialize a matching library, which is used to store prototype features involved in subsequent embodiments.

[0057] S102. Using the machine learning model to be trained, calculate the feature mean of each type of sample image in the first subset of the historical image set, and calculate the feature information of each sample image in the second subset of the historical image set.

[0058] Optionally, the first subset contains the same image categories as the second subset, but the sample images contained in the first subset are different from those contained in the second subset.

[0059] After building the library, you can do so by... Figure 3As shown in Figure 2, knowledge propagation training is performed. Specifically, several tasks can be extracted from the old training data. Each task contains a support set S and a query set Q. Each task can be denoted as a meta-task, the support set S as the first subset, and the query set Q as the second subset. For example, the old training data includes 100 historical categories. Five historical categories are randomly selected from these 100 historical categories. These five historical categories are denoted as historical category A1, historical category A2, historical category A3, historical category A4, and historical category A5. The 10 historical images under historical category A1 are denoted as A101-A110, the 10 historical images under historical category A2 are denoted as A201-A210, the 10 historical images under historical category A3 are denoted as A301-A310, the 10 historical images under historical category A4 are denoted as A401-A410, and the 10 historical images under historical category A5 are denoted as A501-A510. Furthermore, the five historical images under historical category A1, such as A101-A105, the five historical images under historical category A2, such as A201-A205, the five historical images under historical category A3, such as A301-A305, the five historical images under historical category A4, such as A401-A405, and the five historical images under historical category A5, such as A501-A505, constitute a first subset. The five historical images under historical category A1, such as A106-A110, the five historical images under historical category A2, such as A206-A210, the five historical images under historical category A3, such as A306-A310, the five historical images under historical category A4, such as A406-A410, and the five historical images under historical category A5, such as A506-A510, constitute a second subset. Furthermore, in this embodiment, each historical image included in the first subset is designated as a sample image, and each historical image included in the second subset is also designated as a sample image. Similarly, each historical category included in the first subset is designated as an image category, and each historical category included in the second subset is also designated as an image category. This ensures that the image categories included in the first subset and the second subset are the same, but the sample images included in the first subset and the second subset are different. In other words, for the same image category, the sample images included in the first subset and the sample images included in the second subset do not overlap.

[0060] Furthermore, it is understood that, for the same image category, the number of sample images contained in the first subset and the number of sample images contained in the second subset may be different. For example, for historical category A1, the first subset may contain 3 historical images, such as A101-A103, while the second subset may contain 7 historical images, such as A104-A110. Moreover, for different image categories, such as historical category A1 and historical category A2, the number of historical images under historical category A1 contained in the first subset and the number of historical images under historical category A2 contained in the first subset may be different.

[0061] Optionally, calculating the feature mean of each type of sample image in the first subset of the historical image set includes: calculating the image feature mean of multiple sample images under each image category in the first subset to obtain the second feature mean corresponding to each image category in the first subset.

[0062] Optionally, calculating the feature information of each sample image in the second subset of the historical image set includes: calculating the image features of each sample image in the second subset.

[0063] like Figure 3 As shown, the image features of each sample image in the second subset can be calculated using a pre-trained backbone network. For example, if the second subset includes 5 historical categories, and each historical category includes 5 historical images, then the second subset includes a total of 25 historical images. The image features of each of these 25 historical images can be calculated using a pre-trained backbone network, thus obtaining 25 image features.

[0064] Simultaneously, the pre-trained backbone network can calculate the mean of image features for multiple sample images under each image category in the first subset, thus obtaining the mean of the second feature corresponding to each image category in the first subset. For example, the first subset includes 5 historical categories, and each historical category includes 5 historical images. For each of the 5 historical categories, the mean of the image features corresponding to the 5 historical images under that historical category is calculated. This mean can be used as the mean of the second feature corresponding to that historical category, thereby obtaining the mean of the second feature corresponding to historical categories A1, A2, A3, A4, and A5 in the first subset.

[0065] S103. The historical knowledge is propagated to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, and the historical knowledge is propagated to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset.

[0066] like Figure 3 As shown, the pre-trained backbone network can feed the mean of the second feature corresponding to each image category in the first subset into the knowledge propagation module, so that the knowledge propagation module propagates this historical knowledge to the mean feature of each type of sample image in the first subset, thus obtaining the updated mean feature of each type of sample image in the first subset. Similarly, as... Figure 3 As shown, the pre-trained backbone network can send the image features of each sample image in the second subset to the knowledge propagation module, so that the knowledge propagation module can propagate the historical knowledge to the feature information of each sample image in the second subset, and obtain the updated feature information of each sample image in the second subset.

[0067] Optionally, the historical knowledge is propagated to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, including: updating the second feature mean of each image category in the first subset according to the first feature mean of each historical category to obtain the updated second feature mean of each image category in the first subset.

[0068] like Figure 3 As shown, the pre-trained backbone network can feed the mean of the second feature corresponding to each image category in the first subset into the knowledge propagation module, enabling the knowledge propagation module to update the mean of the second feature corresponding to each image category in the first subset. Specifically, the knowledge propagation module can use the knowledge stored in the knowledge base (as described above) to update the mean of the second feature corresponding to each image category in the first subset, obtaining the updated mean of the second feature corresponding to each image category in the first subset. Specifically, the knowledge propagation module can use the mean of the first feature corresponding to each of the 100 historical categories stored in the knowledge base, as described above, to update the mean of the second feature corresponding to each image category in the first subset, obtaining the updated mean of the second feature corresponding to each image category in the first subset.

[0069] Optionally, based on the first feature mean corresponding to each historical category, the second feature mean corresponding to each image category in the first subset is updated to obtain the updated second feature mean corresponding to each image category in the first subset. This includes: weighting the first feature mean corresponding to each historical category based on the similarity between the second feature mean corresponding to any image category in the first subset and the first feature mean corresponding to each historical category, to obtain a first weighted result; and adding the first weighted result to the second feature mean corresponding to any image category to obtain the updated second feature mean corresponding to any image category.

[0070] For example, taking historical category A1 in the first subset as an example, the similarity (also called similarity) between the second feature mean corresponding to historical category A1 and the first feature mean corresponding to each of the 100 historical categories is calculated, thus obtaining 100 similarities. Further, based on these 100 similarities, the first feature mean corresponding to each of the 100 historical categories is weighted and summed to obtain a first weighted result. Here, the 100 similarities can be 100 weight values ​​in the weighted summation process; a similarity can be used as a weight value, or a similarity can be processed using a preset algorithm to obtain a weight value. Alternatively, in some other embodiments, several similarities with similarities greater than or equal to a preset threshold are selected from the 100 similarities. Further, based on these several similarities, the first feature mean corresponding to each of the several historical categories corresponding to these several similarities is weighted and summed to obtain a first weighted result. It is understood that when the first feature mean is a 256-dimensional representation vector, the first weighted result is also a 256-dimensional representation vector.

[0071] Furthermore, the first weighted result is added to the mean of the second feature corresponding to the historical category A1 to obtain the updated mean of the second feature corresponding to the historical category A1. Similarly, the updated mean of the second feature corresponding to other image categories in the first subset can be obtained. The updated mean of the second feature is also a 256-dimensional representation vector.

[0072] Optionally, the historical knowledge is propagated to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset, including: updating the image features of each sample image in the second subset according to the first feature mean corresponding to each historical category to obtain the updated image features of each sample image in the second subset.

[0073] like Figure 3 As shown, the pre-trained backbone network can feed the image features of each sample image in the second subset into the knowledge propagation module, enabling the knowledge propagation module to update the image features of each sample image in the second subset. Specifically, the knowledge propagation module can use the knowledge stored in the knowledge base (as described above) to update the image features of each sample image in the second subset, obtaining the updated image features of each sample image in the second subset. Specifically, the knowledge propagation module can use the mean of the first feature corresponding to each of the 100 historical categories stored in the knowledge base, as described above, to update the image features of each sample image in the second subset, obtaining the updated image features of each sample image in the second subset.

[0074] Optionally, the image features of each sample image in the second subset are updated according to the mean of the first feature corresponding to each historical category to obtain the updated image features of each sample image in the second subset. This includes: weighting the mean of the first feature corresponding to each historical category according to the similarity between the image features of any sample image in the second subset and the mean of the first feature corresponding to each historical category to obtain a second weighted result; and adding the second weighted result to the image features of any sample image to obtain the updated image features of any sample image.

[0075] For example, taking any sample image in the second subset as an example, the similarity (also called similarity) between the image features of this sample image and the mean of the first feature corresponding to each of the 100 historical categories is calculated, thus obtaining 100 similarities. Further, based on these 100 similarities, the mean of the first feature corresponding to each of the 100 historical categories is weighted and summed to obtain a second weighted result. Here, the 100 similarities can be 100 weight values ​​in the weighted summation process; one similarity can be used as a weight value, or a similarity can be processed using a preset algorithm to obtain a weight value. Alternatively, in some other embodiments, several similarities with similarities greater than or equal to a preset threshold are selected from the 100 similarities. Further, based on these several similarities, the mean of the first feature corresponding to each of the several historical categories corresponding to these several similarities is weighted and summed to obtain a second weighted result. It is understood that when the mean of the first feature is a 256-dimensional representation vector, the second weighted result is also a 256-dimensional representation vector.

[0076] Furthermore, the second weighted result is added to the image features of any sample image to obtain the updated image features of that sample image. Similarly, the updated image features of other sample images in the second subset can be obtained. The updated image features are also 256-dimensional representation vectors.

[0077] S104. The machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the updated feature mean of each type of sample image in the first subset.

[0078] For example, the first subset includes 5 historical categories, i.e., image categories. After updating the mean of the second feature corresponding to each of these 5 image categories, a total of 5 updated mean values ​​of the second feature are obtained. The second subset includes 25 sample images. After updating the image features of each of these 25 sample images, a total of 25 updated image features are obtained. Specifically, for any one of the 25 updated image features, the similarity between that updated image feature and each of the 5 updated mean values ​​of the second feature is calculated, resulting in a total of 5 similarity scores. Each of these 5 similarity scores represents the probability that the image category of the sample image corresponding to that updated image feature is the image category corresponding to the updated mean value of the second feature used when calculating that similarity score. Assuming that these 5 similarity scores constitute a column of data, it means that each updated image feature corresponds to such a column of data, and the 25 updated image features correspond to a total of 25 columns of data, which can form a 5*25 matrix. Here, this 5*25 matrix is ​​denoted as the first matrix. In other words, after such Figure 3 The similarity calculation shown yields the first matrix, which can then be used as the output of the machine learning model to be trained; that is, the first matrix can serve as the prediction result of the machine learning model. Based on this prediction result and the actual category of each sample image in the second subset, the machine learning model is trained.

[0079] Optionally, the machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the updated feature mean of each type of sample image in the first subset, including: training the machine learning model based on the similarity between the updated second feature mean corresponding to each image category in the first subset and the updated image features of each sample image in the second subset, and the actual category of each sample image in the second subset.

[0080] For the second subset, during the training phase of the machine learning model, the image category of each sample image in the second subset can be pre-labeled, thus identifying the actual category of each sample image in the second subset. Since the first subset contains the same image categories as the second subset—for example, both the first and second subsets each contain 5 image categories (e.g., A1-A5)—if the actual category of any sample image in the second subset is one of the 5 categories, such as A1, then the actual category of that sample image cannot be any of A2-A5. In this case, a column vector can be constructed, where the elements in the first row are 1, and the elements in the second through fifth rows are 0. Similarly, a similar column vector can be constructed for each other sample image in the second subset, resulting in a total of 25 column vectors. These 25 column vectors form a 5*25 matrix. Here, this 5*25 matrix is ​​denoted as the second matrix.

[0081] Understandably, since several tasks can be extracted from the old training data, each task contains a support set S and a query set Q, a first matrix and a second matrix can be obtained through the calculation process described above, meaning that a first matrix and a second matrix can be obtained for each task. If we consider the first matrix and the second matrix corresponding to the same task as a set of data, then N tasks will correspond to N sets of data, and a cross-entropy loss function value can be calculated based on the first matrix and the second matrix in the same set of data, resulting in a total of N cross-entropy loss function values. Further, these N cross-entropy loss function values ​​are used in conjunction with the parameters of the machine learning model to be trained, i.e., training. Specifically, based on a cross-entropy loss function value and its corresponding task, the parameters of the machine learning model to be trained can be optimized once, thus performing N iterative optimizations on the parameters of the machine learning model to be trained. This enables the trained machine learning model to recognize the historical category corresponding to each of the N tasks.

[0082] The trained machine learning model includes a trained backbone network and a trained knowledge propagation module. After freezing the model parameters, if multiple sample images of a new category appear, the trained backbone network can encode each sample image of the new category, obtaining the image features of each sample image of the new category, as well as the average image feature value of multiple sample images of the new category. Furthermore, the trained knowledge propagation module can update (e.g., enhance or supplement) the average image feature value corresponding to the new category using knowledge stored in the knowledge base. When an image to be identified appears, the trained backbone network can also encode the image features of the image to be identified. The trained knowledge propagation module can further update (e.g., enhance or supplement) the image features of the image to be identified using knowledge stored in the knowledge base. If the updated image features of the image to be identified have a high similarity to the updated average image feature value corresponding to the new category, then the new category can be adopted as the image category of the image to be identified.

[0083] This embodiment extracts historical knowledge from a historical image set using a machine learning model to be trained. The machine learning model calculates the mean features of various sample images within a first subset of the historical image set, and calculates the feature information of each sample image within a second subset of the historical image set. The historical knowledge is propagated to the mean features of various sample images within the first subset to obtain updated mean features, and the historical knowledge is also propagated to the feature information of each sample image within the second subset to obtain updated feature information. Based on the similarity between the updated feature information of each sample image within the second subset and the updated mean features of various sample images within the first subset, the machine learning model is trained. This allows the trained machine learning model to propagate historical knowledge to new categories of images, thereby enhancing or supplementing the feature information of these new categories. Based on the enhanced or supplemented feature information of the new categories, the recognition accuracy of these new categories of images can be improved. Therefore, even if the machine learning model is frozen, the recognition accuracy of new categories of images can still be improved when new categories of images appear using the trained machine learning model.

[0084] It is understandable that a trained machine learning model can not only propagate historical knowledge to new category images, but also to old category images. Therefore, the method described in this embodiment can not only improve the recognition accuracy of new category images, but also further improve the recognition accuracy of old category images.

[0085] Figure 4This is a flowchart illustrating an image recognition method according to another embodiment of the present disclosure. The method can be as follows: Figure 5 The model inference process shown here refers to the machine learning model trained using the model training method described above. Figure 5 As shown, the trained machine learning model includes a trained backbone network and a trained knowledge propagation module. It can be understood that the model's training process and its inference process (i.e., usage process) can be performed by the same entity or by different entities. For example, Figure 2 The server 22 shown can execute model training methods to obtain a trained machine learning model. The trained machine learning model can be stored on server 22, allowing server 22 to execute image recognition methods based on the trained machine learning model. Alternatively, server 22 can deploy the trained machine learning model to other devices, enabling those devices to execute image recognition methods using the trained machine learning model. Furthermore, in this embodiment, the trained machine learning model can be frozen, that is, its parameters can be fixed, so that the frozen machine learning model can accurately identify images of new categories. Specifically, the image recognition method includes the following steps:

[0086] S401. Obtain the mean value of the first feature corresponding to each historical category among multiple historical categories. The mean value of the first feature corresponding to each historical category is the mean value of the image features of multiple historical images under the historical category.

[0087] For example, the mean of the first feature corresponding to each of the 100 historical categories can be obtained from the knowledge base mentioned above. The mean of the first feature corresponding to each historical category is the mean of the image features of multiple historical images under that historical category.

[0088] Furthermore, it is understood that since the trained machine learning model includes a trained backbone network and a trained knowledge propagation module, after obtaining the trained backbone network, the image features of each of the 1000 historical images can be recalculated using this trained backbone network, thereby recalculating the mean of the first feature corresponding to each of the 100 historical categories. Therefore, the mean of the first feature corresponding to each of the 100 historical categories obtained in this embodiment can be the mean of the first feature corresponding to each of the 100 historical categories recalculated by the trained backbone network.

[0089] S402. Using the trained machine learning model, calculate the target image features of the images to be identified in the test set, and calculate the mean value of the image features of multiple sample images under each image category in the training set, to obtain the mean value of the third feature corresponding to each image category in the training set. The machine learning model is trained using the model training method described above.

[0090] Optionally, the test set contains the same image categories as the training set, but the images to be identified in the test set are different from the sample images in the training set.

[0091] For example, during the model inference phase, a new task arises that requires the trained machine learning model to recognize not only old-class images but also new-class images. In this case, for this new task, training and test sets can be constructed. The training set can include both new-class and old-class images, and similarly, the test set also includes both. The only difference is that the class of each image in the training set is known, while the class of each image in the test set is unknown, i.e., to be identified. In some embodiments, the relationship between the training and test sets is similar to the relationship between the support set and query set as described above.

[0092] In this embodiment, for each image category in the training set, the mean image features of multiple sample images under that image category can be calculated using the trained backbone network, and this mean image feature is used as the mean third feature corresponding to that image category. For example, the training set includes several images of zebras, tigers, and horses, where "zebra" is a new category, and "tiger" and "horse" are old categories. Using the trained backbone network, the image features of each image in the zebra images can be calculated. Furthermore, the mean image features of each image in the zebra images are averaged to obtain the mean third feature corresponding to "zebra". Similarly, the mean third feature corresponding to "tiger" and the mean third feature corresponding to "horse" can be calculated. Here, the mean third feature corresponding to "zebra", the mean third feature corresponding to "tiger", and the mean third feature corresponding to "horse" can be recorded as prototype features. Simultaneously, using the trained backbone network, the image features of any image to be identified in the test set can also be calculated. These image features can be recorded as target image features. The target image features can be as follows: Figure 5 The test features are shown.

[0093] S403. Update the target image features according to the mean value of the first feature corresponding to each historical category to obtain the updated target image features.

[0094] For example Figure 5As shown, the trained backbone network can feed the target image features into the trained knowledge propagation module, enabling the knowledge propagation module to update the target image features. Specifically, the trained knowledge propagation module can utilize the knowledge stored in the knowledge base, such as the mean of the first feature corresponding to each of the 100 historical categories mentioned above, to update the target image features, thus obtaining the updated target image features.

[0095] Optionally, the target image features are updated based on the mean of the first feature corresponding to each historical category to obtain updated target image features, including: weighting the mean of the first feature corresponding to each historical category based on the similarity between the target image features and the mean of the first feature corresponding to each historical category to obtain a third weighted result; adding the third weighted result to the target image features to obtain the updated target image features.

[0096] For example, the target image feature is a 256-dimensional representation vector, and the mean of the first feature corresponding to each of the 100 historical categories is also a 256-dimensional representation vector. The similarity between the target image feature and the mean of the first feature corresponding to each of the 100 historical categories is calculated, resulting in 100 similarity scores. Further, based on these 100 similarity scores, the mean of the first feature corresponding to each of the 100 historical categories is weighted and summed to obtain a third weighted result. Finally, this third weighted result is added to the target image feature to obtain the updated target image feature.

[0097] S404. Based on the mean value of the first feature corresponding to each historical category, update the mean value of the third feature corresponding to each image category in the training set to obtain the updated mean value of the third feature corresponding to each image category in the training set.

[0098] For example Figure 5 As shown, the trained backbone network can feed the prototype features into the trained knowledge propagation module, enabling the knowledge propagation module to update the prototype features. Specifically, the trained knowledge propagation module can utilize the knowledge stored in the knowledge base, such as the mean of the first feature corresponding to each of the 100 historical categories mentioned above, to update the mean of the third feature corresponding to each image category in the training set, thus obtaining the updated mean of the third feature for each image category in the training set.

[0099] For example, the trained knowledge propagation module can update the mean of the third feature corresponding to "zebra," "tiger," and "horse" based on the mean of the first feature corresponding to each of the 100 historical categories as described above, resulting in updated mean values ​​for the third features of "zebra," "tiger," and "horse," respectively. Further, these updated mean values ​​are stored in a matching library; that is, the updated prototype features are stored in the matching library.

[0100] Optionally, the average third feature corresponding to each image category in the training set is updated based on the average first feature corresponding to each historical category to obtain the updated average third feature corresponding to each image category in the training set. This includes: weighting the average first feature corresponding to each historical category based on the similarity between the average third feature corresponding to any image category in the training set and the average first feature corresponding to each historical category to obtain a fourth weighting result; and adding the fourth weighting result to the average third feature corresponding to any image category to obtain the updated average third feature corresponding to any image category.

[0101] For example, taking the mean of the third feature corresponding to "zebra" as an example, the similarity between the mean of the third feature corresponding to "zebra" and the mean of the first feature corresponding to each of the 100 historical categories is calculated, resulting in 100 similarity scores. Further, based on these 100 similarity scores, the mean of the first feature corresponding to each of the 100 historical categories is weighted and summed to obtain a fourth weighted result. Then, this fourth weighted result is added to the mean of the third feature corresponding to "zebra" to obtain the updated mean of the third feature corresponding to "zebra". Similarly, the updated mean of the third feature corresponding to "tiger" and "horse" are calculated.

[0102] S405. Based on the updated target image features and the average of the updated third features corresponding to each image category in the training set, determine the target category corresponding to the content in the image to be identified.

[0103] For example, based on the updated target image features and the average of the updated third features corresponding to "zebra", "tiger", and "horse", the target category corresponding to the content in the image to be identified can be determined as "zebra", "tiger", or "horse".

[0104] Optionally, determining the target category corresponding to the content in the image to be identified based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set includes: determining the target image category from the training set based on the similarity between the updated target image features and the updated mean of the third feature corresponding to each image category in the training set, wherein the updated target image features have the highest similarity to the updated mean of the third feature corresponding to the target image category; and determining the target image category as the target category corresponding to the content in the image to be identified.

[0105] For example, the following steps are taken: calculating the first similarity between the updated target image features and the mean of the updated third features corresponding to "zebra"; calculating the second similarity between the updated target image features and the mean of the updated third features corresponding to "tiger"; and calculating the third similarity between the updated target image features and the mean of the updated third features corresponding to "horse". Further, the maximum value is selected from the first, second, and third similarities. For example, assuming the first similarity is the largest, "zebra" is determined as the target image category in the training set, and "zebra" is determined as the target category corresponding to the content in the image to be identified. That is, "zebra" is the recognition result for the image to be identified.

[0106] Since the updated prototype features, such as the updated mean of the third feature corresponding to "zebra," "tiger," and "horse," are stored in the matching library, in other embodiments, the updated prototype features stored in the matching library can also be copied to obtain matching features; that is, the matching features are the updated prototype features. When calculating the similarity between the updated target image features and each updated mean of the third feature, the similarity between the updated target image features and each matching feature can be specifically calculated. Furthermore, different labels can be pre-labeled for each image category in the training set. For example, the new category "zebra" can be labeled as label 1, the old category "tiger" as label 2, and the old category "horse" as label 3. When the first similarity, as described above, is the highest, it indicates that the probability of the image to be identified being of the category "zebra" is the highest. At this time, label 3 can be assigned to the image to be identified, thereby determining that the content in the image to be identified is a zebra.

[0107] Understandably, freezing the trained machine learning model is equivalent to freezing the trained backbone network and the trained knowledge propagation module. When a new category, such as "zebra," appears in the training set, the trained backbone network, having not encoded zebra images, has limited ability to recognize them, resulting in incomplete image features for the calculated zebra images. Since there are multiple zebra images in the training set, averaging the image features of each image in these zebra images to obtain the mean of the third feature for "zebra" is also incomplete or inaccurate. However, the trained knowledge propagation module can update the mean of the third feature for "zebra" based on historical knowledge stored in the knowledge base (e.g., the mean of the first feature for each of 100 historical categories), thereby enhancing or supplementing the mean of the third feature for "zebra," making the updated mean of the third feature for "zebra" complete and accurate. For example, this historical knowledge includes the mean of the first feature corresponding to the old category "tiger" and the mean of the first feature corresponding to the old category "horse". Since the mean of the first feature corresponding to "tiger" is the mean of image features from a large number of historical images of "tiger", the mean of the first feature corresponding to "tiger" can include species-specific information of tigers (the tiger's unique appearance), such as stripe information. Similarly, the mean of the first feature corresponding to "horse" can include species-specific information of horses (the horse's unique appearance), such as body structure information. When updating the mean of the third feature corresponding to "zebra" based on historical knowledge, the stripe information of tigers and the body structure information of horses can be propagated to the mean of the third feature corresponding to "zebra", that is, the stripe information of tigers and the body structure information of horses are supplemented to the mean of the third feature corresponding to "zebra", so that the mean of the third feature corresponding to "zebra" is enhanced or supplemented. Therefore, the updated mean of the third feature corresponding to "zebra" can include relatively complete species-specific information of zebras (such as stripes and body structure). Similarly, the completed knowledge dissemination module can enhance or supplement the mean of the third feature corresponding to the old category "tiger" in the training set, and enhance or supplement the mean of the third feature corresponding to the old category "horse" in the training set, based on historical knowledge.

[0108] Similarly, the trained knowledge propagation module, based on historical knowledge stored in the knowledge base (e.g., the average of the first features corresponding to each of the 100 historical categories), can enhance or supplement the image features of the image to be recognized when updating them, ensuring that the updated target image features are complete and accurate. Therefore, when calculating the similarity between the updated target image features and the average of the updated third features corresponding to each image category in the training set, the accuracy of each similarity can be guaranteed. Thus, based on the accurate similarity scores, the target category corresponding to the content in the image to be recognized can be accurately determined from the old and new categories included in the training set.

[0109] In other words, this embodiment, from the perspective of knowledge dissemination, helps the frozen machine learning model adapt to the image recognition process of new categories by disseminating historical knowledge. This method acquires historical knowledge and then uses a knowledge dissemination module to propagate this knowledge to images of new categories, enhancing or supplementing the content of these images, such as the specificity of the target object. For example, historical knowledge includes information about tiger stripes and horse body structure. When images of zebras appear in the new category, this stripe and body structure information is disseminated to enhance or supplement the zebra's specificity (e.g., its unique appearance). Since the method described in this embodiment can be applied to few-shot incremental learning, this embodiment provides a few-shot incremental learning method based on knowledge dissemination. This method improves the performance of few-shot incremental image recognition while maintaining a small memory footprint; that is, it can improve the recognition accuracy not only for images of new categories but also for images of old categories.

[0110] This embodiment uses a trained machine learning model to calculate the target image features of the image to be identified in the test set, and calculates the mean value of image features of multiple sample images under each image category in the training set, thus obtaining the mean value of the third feature corresponding to each image category in the training set. Further, based on the mean value of the first feature corresponding to each historical category, the target image features are updated to obtain updated target image features, making the updated target image features complete and accurate. Based on the mean value of the first feature corresponding to each historical category, the mean value of the third feature corresponding to each image category in the training set is updated to make the mean value of the third feature corresponding to each image category in the training set complete and accurate. Further, based on the similarity between the updated target image features and the updated mean value of the third feature corresponding to each image category in the training set, the target category corresponding to the content in the image to be identified can be accurately determined from the image categories included in the training set.

[0111] The following is combined Figure 6 The knowledge dissemination module described above will be introduced. For example... Figure 6 As shown, assume the knowledge base stores the mean of the first feature for each of 100 historical categories. The feature to be updated can be the mean of the second feature, the mean of the third feature, or the target image feature of the image to be identified, as described above. The knowledge propagation module can input the feature to be updated into the knowledge base, which can calculate the similarity between the feature to be updated and the mean of the first feature for each of the 100 historical categories, obtaining 100 similarity scores. Further, the knowledge base weights the mean of the first feature for each of the 100 historical categories based on these 100 similarity scores, obtaining a weighted result, and feeds this weighted result back to the feature enhancement unit in the knowledge propagation module. The feature enhancement unit can add the feature to be updated and the weighted result to obtain the updated feature.

[0112] Figure 7 This is a schematic diagram of the structure of a model training apparatus provided in an embodiment of this disclosure. The model training apparatus provided in this embodiment of the disclosure can execute the processing flow provided in the model training method embodiment, such as... Figure 7 As shown, the model training device 70 includes:

[0113] Extraction module 71 is used to extract historical knowledge from a collection of historical images through a machine learning model to be trained;

[0114] The calculation module 72 is used to calculate the feature mean of each type of sample image in the first subset of the historical image set through the machine learning model to be trained, and to calculate the feature information of each sample image in the second subset of the historical image set.

[0115] The knowledge dissemination module 73 is used to disseminate the historical knowledge to the feature mean of various sample images in the first subset, to obtain the updated feature mean of various sample images in the first subset, and to disseminate the historical knowledge to the feature information of each sample image in the second subset, to obtain the updated feature information of each sample image in the second subset.

[0116] The training module 74 is used to train the machine learning model based on the similarity between the updated feature information of each sample image in the second subset and the updated feature mean of each type of sample image in the first subset.

[0117] Optionally, the historical image set includes multiple historical images under each of multiple historical categories; when the extraction module 71 extracts historical knowledge from the historical image set through the machine learning model to be trained, it is specifically used to: for each of the multiple historical categories, calculate the average image feature of multiple historical images under the historical category through the machine learning model to be trained, and obtain the first feature average value corresponding to the historical category.

[0118] Optionally, when the calculation module 72 calculates the feature mean of each type of sample image in the first subset of the historical image set, it is specifically used to: calculate the image feature mean of multiple sample images under each image category in the first subset to obtain the second feature mean corresponding to each image category in the first subset; when the calculation module 72 calculates the feature information of each sample image in the second subset of the historical image set, it is specifically used to: calculate the image feature of each sample image in the second subset.

[0119] Optionally, when the knowledge propagation module 73 propagates the historical knowledge to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, it is specifically used to: update the second feature mean corresponding to each image category in the first subset according to the first feature mean corresponding to each historical category to obtain the updated second feature mean corresponding to each image category in the first subset; when the knowledge propagation module 73 propagates the historical knowledge to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset, it is specifically used to: update the image features of each sample image in the second subset according to the first feature mean corresponding to each historical category to obtain the updated image features of each sample image in the second subset.

[0120] Optionally, the first subset contains the same image categories as the second subset, but the sample images contained in the first subset are different from those contained in the second subset.

[0121] Optionally, when training the machine learning model based on the similarity between the updated feature information of each sample image in the second subset and the updated feature mean of each type of sample image in the first subset, the training module 74 specifically trains the machine learning model based on the similarity between the updated second feature mean corresponding to each image category in the first subset and the updated image features of each sample image in the second subset, as well as the actual category of each sample image in the second subset.

[0122] Optionally, when the knowledge dissemination module 73 updates the second feature mean corresponding to each image category in the first subset based on the first feature mean corresponding to each historical category, and obtains the updated second feature mean corresponding to each image category in the first subset, it is specifically used for:

[0123] Based on the similarity between the mean of the second feature corresponding to any image category in the first subset and the mean of the first feature corresponding to each historical category, the mean of the first feature corresponding to each historical category is weighted to obtain a first weighted result.

[0124] The first weighted result is added to the mean of the second feature corresponding to any image category to obtain the updated mean of the second feature corresponding to any image category.

[0125] Optionally, when the knowledge dissemination module 73 updates the image features of each sample image in the second subset based on the mean of the first feature corresponding to each historical category, and obtains the updated image features of each sample image in the second subset, it is specifically used for:

[0126] Based on the similarity between the image features of any sample image in the second subset and the mean value of the first feature corresponding to each historical category, the mean value of the first feature corresponding to each historical category is weighted to obtain a second weighted result.

[0127] The second weighted result is added to the image features of any sample image to obtain the updated image features of any sample image.

[0128] Figure 7 The model training apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0129] Figure 8 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this disclosure. The image recognition device provided in this embodiment can execute the processing flow provided in the image recognition method embodiment, such as... Figure 8 As shown, the image recognition device 80 includes:

[0130] The acquisition module 81 is used to acquire the mean value of the first feature corresponding to each historical category among multiple historical categories, wherein the mean value of the first feature corresponding to each historical category is the mean value of the image features of multiple historical images under the historical category;

[0131] The calculation module 82 is used to calculate the target image features of the image to be identified in the test set and the mean value of the image features of multiple sample images under each image category in the training set through the trained machine learning model, so as to obtain the mean value of the third feature corresponding to each image category in the training set. The machine learning model is trained using the model training method described above.

[0132] The first update module 83 is used to update the target image features according to the first feature mean value corresponding to each historical category, so as to obtain the updated target image features;

[0133] The second update module 84 is used to update the third feature mean of each image category in the training set according to the first feature mean of each historical category, so as to obtain the updated third feature mean of each image category in the training set.

[0134] The determination module 85 is used to determine the target category corresponding to the content in the image to be identified based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set.

[0135] Optionally, when the first update module 83 updates the target image features based on the mean value of the first feature corresponding to each historical category to obtain the updated target image features, it is specifically used for:

[0136] Based on the similarity between the target image features and the mean value of the first feature corresponding to each historical category, the mean value of the first feature corresponding to each historical category is weighted to obtain a third weighted result.

[0137] The third weighted result is added to the target image features to obtain the updated target image features.

[0138] Optionally, when the second update module 84 updates the third feature mean corresponding to each image category in the training set based on the first feature mean corresponding to each historical category, and obtains the updated third feature mean corresponding to each image category in the training set, it is specifically used for:

[0139] Based on the similarity between the mean of the third feature corresponding to any image category in the training set and the mean of the first feature corresponding to each historical category, the mean of the first feature corresponding to each historical category is weighted to obtain a fourth weighted result.

[0140] The fourth weighted result is added to the mean of the third feature corresponding to any image category to obtain the updated mean of the third feature corresponding to any image category.

[0141] Optionally, when determining the target category corresponding to the content in the image to be identified based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set, the determining module 85 is specifically used for:

[0142] Based on the similarity between the updated target image features and the mean of the updated third features corresponding to each image category in the training set, the target image category is determined from the training set, wherein the updated target image features have the greatest similarity to the mean of the updated third features corresponding to the target image category;

[0143] The target image category is determined as the target category corresponding to the content in the image to be identified.

[0144] Optionally, the test set contains the same image categories as the training set, but the images to be identified in the test set are different from the sample images in the training set.

[0145] Figure 8 The image recognition device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0146] In addition, this embodiment also provides a few-sample incremental image recognition method based on knowledge propagation, which includes the following steps:

[0147] S1. Obtain historical knowledge from the historical image collection.

[0148] Optionally, the historical image set includes multiple historical images under each historical category; obtaining historical knowledge from the historical image set includes: obtaining the average value of a first feature corresponding to each historical category, wherein the average value of the first feature corresponding to each historical category is the average value of image features of multiple historical images under the historical category.

[0149] S2. Calculate the feature mean of each type of sample image in the training set, and calculate the target image features of the image to be identified in the test set.

[0150] Optionally, calculating the mean features of various sample images in the training set includes: calculating the mean image features of multiple sample images under each image category in the training set to obtain the mean third feature corresponding to each image category in the training set. Alternatively, calculating the mean image features of multiple sample images under each image category in the training set to obtain the mean third feature corresponding to each image category in the training set includes: using a trained machine learning model, calculating the mean image features of multiple sample images under each image category in the training set to obtain the mean third feature corresponding to each image category in the training set. The machine learning model is trained using the model training method described above.

[0151] Optionally, the target image features of the images to be identified in the test set are calculated, including: calculating the target image features of the images to be identified in the test set using a trained machine learning model.

[0152] Optionally, the test set contains the same image categories as the training set, but the images to be identified in the test set are different from the sample images in the training set.

[0153] S3. The historical knowledge is propagated to the feature mean of each type of sample image in the training set to obtain the updated feature mean of each type of sample image in the training set, and the historical knowledge is propagated to the target image feature of the image to be identified to obtain the updated target image feature of the image to be identified.

[0154] Optionally, propagating the historical knowledge to the target image features of the image to be identified to obtain updated target image features of the image to be identified includes: updating the target image features according to the mean value of the first feature corresponding to each historical category to obtain updated target image features.

[0155] Optionally, based on the similarity between the target image features and the mean of the first features corresponding to each historical category, a weighted average of the first features corresponding to each historical category is calculated to obtain a third weighted result;

[0156] The third weighted result is added to the target image features to obtain the updated target image features.

[0157] Optionally, the historical knowledge is propagated to the feature mean of each type of sample image in the training set to obtain the updated feature mean of each type of sample image in the training set, including: updating the third feature mean of each image category in the training set according to the first feature mean of each historical category to obtain the updated third feature mean of each image category in the training set.

[0158] Optionally, the average third feature corresponding to each image category in the training set is updated based on the average first feature corresponding to each historical category to obtain the updated average third feature corresponding to each image category in the training set. This includes: weighting the average first feature corresponding to each historical category based on the similarity between the average third feature corresponding to any image category in the training set and the average first feature corresponding to each historical category to obtain a fourth weighting result; and adding the fourth weighting result to the average third feature corresponding to any image category to obtain the updated average third feature corresponding to any image category.

[0159] S4. Determine the target category corresponding to the content in the image to be identified based on the similarity between the updated target image features of the image to be identified and the updated mean features of various sample images in the training set.

[0160] Optionally, the target category corresponding to the content in the image to be identified is determined based on the similarity between the updated target image features of the image to be identified and the updated mean features of each type of sample image in the training set. This includes: determining the target image category from the training set based on the similarity between the updated target image features and the updated mean third feature corresponding to each image category in the training set, wherein the updated target image features have the highest similarity to the updated mean third feature corresponding to the target image category; and determining the target image category as the target category corresponding to the content in the image to be identified.

[0161] Specifically, the implementation methods and specific principles of S1-S4 are as described above, and will not be repeated here.

[0162] The above describes the internal functions and structure of the model training and image recognition device, which can be implemented as an electronic device. Figure 9 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. (See attached diagram.) Figure 9 As shown, the electronic device includes a memory 91 and a processor 92.

[0163] Memory 91 is used to store programs. In addition to the programs described above, memory 91 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0164] The memory 91 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0165] The processor 92 is coupled to the memory 91 and executes the program stored in the memory 91 for:

[0166] Extracting historical knowledge from a collection of historical images using a machine learning model to be trained;

[0167] Using the machine learning model to be trained, the feature mean of each type of sample image in the first subset of the historical image set is calculated, and the feature information of each sample image in the second subset of the historical image set is calculated.

[0168] The historical knowledge is propagated to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, and the historical knowledge is propagated to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset.

[0169] The machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the average updated feature values ​​of each type of sample image in the first subset.

[0170] Alternatively, the processor can also be used for:

[0171] Extracting historical knowledge from historical image collections;

[0172] Calculate the feature mean of each type of sample image in the training set, and calculate the target image features of the image to be identified in the test set;

[0173] The historical knowledge is propagated to the feature mean of various sample images in the training set to obtain the updated feature mean of various sample images in the training set, and the historical knowledge is propagated to the target image feature of the image to be identified to obtain the updated target image feature of the image to be identified.

[0174] The target category corresponding to the content in the image to be identified is determined based on the similarity between the updated target image features of the image to be identified and the updated mean features of various sample images in the training set.

[0175] Alternatively, the processor can also be used for:

[0176] Obtain the mean value of the first feature corresponding to each historical category among multiple historical categories. The mean value of the first feature corresponding to each historical category is the mean value of the image features of multiple historical images under the historical category.

[0177] The machine learning model is trained to calculate the target image features of the images to be identified in the test set, and to calculate the mean of the image features of multiple sample images under each image category in the training set, thereby obtaining the mean of the third feature corresponding to each image category in the training set. The machine learning model is trained using the model training method described above.

[0178] The target image features are updated based on the mean value of the first feature corresponding to each historical category to obtain the updated target image features.

[0179] Based on the mean value of the first feature corresponding to each historical category, the mean value of the third feature corresponding to each image category in the training set is updated to obtain the updated mean value of the third feature corresponding to each image category in the training set.

[0180] The target category corresponding to the content in the image to be identified is determined based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set.

[0181] Furthermore, such as Figure 9 As shown, the electronic device may also include other components such as a communication component 93, a power supply component 94, an audio component 95, and a display 96. Figure 9 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 9 The components shown.

[0182] Communication component 93 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 93 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0183] Power supply component 94 provides power to various components of the electronic device. Power supply component 94 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0184] Audio component 95 is configured to output and / or input audio signals. For example, audio component 95 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 91 or transmitted via communication component 93. In some embodiments, audio component 95 also includes a speaker for outputting audio signals.

[0185] Display 96 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0186] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the model training method or image recognition method described in the above embodiments.

[0187] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0188] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, wherein, The method includes: Historical knowledge is extracted from a set of historical images using a machine learning model to be trained. The set of historical images includes multiple historical images under each of multiple historical categories. The historical knowledge is used to represent the mean of image features of the multiple historical images under each historical category. Using the machine learning model to be trained, the feature mean of each type of sample image in the first subset of the historical image set is calculated, and the feature information of each sample image in the second subset of the historical image set is calculated. The historical knowledge is propagated to the feature mean of each type of sample image in the first subset to obtain the updated feature mean of each type of sample image in the first subset, and the historical knowledge is propagated to the feature information of each sample image in the second subset to obtain the updated feature information of each sample image in the second subset. The machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the average updated feature values ​​of each type of sample image in the first subset.

2. The method according to claim 1, wherein, The extraction of historical knowledge from the historical image set using a machine learning model to be trained includes: For each of the multiple historical categories, the average image features of multiple historical images under each historical category are calculated using the machine learning model to be trained, thereby obtaining the first feature mean corresponding to each historical category.

3. The method according to claim 2, wherein, Calculating the feature mean of each type of sample image within the first subset of the historical image set includes: Calculate the mean value of image features of multiple sample images under each image category in the first subset to obtain the mean value of the second feature corresponding to each image category in the first subset; Calculating the feature information of each sample image within the second subset of the historical image set includes: Calculate the image features of each sample image in the second subset.

4. The method according to claim 3, wherein, The historical knowledge is propagated to the feature mean of each type of sample image within the first subset, resulting in updated feature mean values ​​for each type of sample image within the first subset, including: Based on the mean value of the first feature corresponding to each historical category, the mean value of the second feature corresponding to each image category in the first subset is updated to obtain the updated mean value of the second feature corresponding to each image category in the first subset; The historical knowledge is propagated to the feature information of each sample image within the second subset, resulting in updated feature information for each sample image within the second subset, including: Based on the mean value of the first feature corresponding to each historical category, the image features of each sample image in the second subset are updated to obtain the updated image features of each sample image in the second subset.

5. The method according to claim 1, wherein, The first subset contains the same image categories as the second subset, but the sample images contained in the first subset are different from those contained in the second subset.

6. The method according to claim 1, wherein, The machine learning model is trained based on the similarity between the updated feature information of each sample image in the second subset and the average updated feature values ​​of each type of sample image in the first subset, including: The machine learning model is trained based on the similarity between the updated second feature mean corresponding to each image category in the first subset and the updated image features of each sample image in the second subset, as well as the actual category of each sample image in the second subset.

7. The method according to claim 4, wherein, Based on the mean value of the first feature corresponding to each historical category, the mean value of the second feature corresponding to each image category in the first subset is updated to obtain the updated mean value of the second feature corresponding to each image category in the first subset, including: Based on the similarity between the mean of the second feature corresponding to any image category in the first subset and the mean of the first feature corresponding to each historical category, the mean of the first feature corresponding to each historical category is weighted to obtain a first weighted result. The first weighted result is added to the mean of the second feature corresponding to any image category to obtain the updated mean of the second feature corresponding to any image category.

8. The method according to claim 4, wherein, Based on the mean value of the first feature corresponding to each historical category, the image features of each sample image in the second subset are updated to obtain the updated image features of each sample image in the second subset, including: Based on the similarity between the image features of any sample image in the second subset and the mean value of the first feature corresponding to each historical category, the mean value of the first feature corresponding to each historical category is weighted to obtain a second weighted result. The second weighted result is added to the image features of any sample image to obtain the updated image features of any sample image.

9. A few-sample incremental image recognition method based on knowledge propagation, wherein, The method includes: Historical knowledge is obtained from a set of historical images, wherein the set of historical images includes multiple historical images under each of multiple historical categories, and the historical knowledge is used to represent the mean image feature of the multiple historical images under each historical category; Calculate the feature mean of each type of sample image in the training set, and calculate the target image features of the image to be identified in the test set; The historical knowledge is propagated to the feature mean of various sample images in the training set to obtain the updated feature mean of various sample images in the training set, and the historical knowledge is propagated to the target image feature of the image to be identified to obtain the updated target image feature of the image to be identified. The target category corresponding to the content in the image to be identified is determined based on the similarity between the updated target image features of the image to be identified and the updated mean features of various sample images in the training set.

10. An image recognition method, wherein, The method includes: Obtain the mean value of the first feature corresponding to each historical category among multiple historical categories. The mean value of the first feature corresponding to each historical category is the mean value of the image features of multiple historical images under each historical category. The machine learning model is trained to calculate the target image features of the images to be identified in the test set, and to calculate the mean of the image features of multiple sample images under each image category in the training set, so as to obtain the mean of the third feature corresponding to each image category in the training set. The machine learning model is trained using the method described in any one of claims 1-8. The target image features are updated based on the mean value of the first feature corresponding to each historical category to obtain the updated target image features. Based on the mean value of the first feature corresponding to each historical category, the mean value of the third feature corresponding to each image category in the training set is updated to obtain the updated mean value of the third feature corresponding to each image category in the training set. The target category corresponding to the content in the image to be identified is determined based on the updated target image features and the updated mean of the third feature corresponding to each image category in the training set.

11. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images