Image processing and image classification method, device, equipment and storage medium
Through iterative training and correction of predicted values, the classification model is corrected, which solves the problem of imbalance between predicted values in the old and new categories, and improves the efficiency and accuracy of incremental learning.
Patent Information
- Application Number
- CN202210294934.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-03-23
AI Technical Summary
During the incremental learning process, due to the imbalance of the training data volume of new and old categories, the prediction values output by the target model on the new and old categories are unbalanced, affecting the prediction accuracy of the model.
By iteratively training the original classification model by using the newly added sample image set and historical sample image set, the prediction value of the corrected sample image is obtained, the target offset is determined, and the candidate classification model is output and corrected, and the target classification model is obtained, so that the new and old are new The predicted values on the category are balanced.
Improve the prediction accuracy of the model in the old and new categories, and reduce the resource consumption of incremental learning, avoiding the need for repeated training.
Smart Images

Figure CN114693993B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image processing and image classification method, apparatus, device and storage medium. Background Art
[0002] With the development of artificial intelligence technology, various machine learning models have been widely used in life. In related technologies, the machine learning model is trained using training data collected over a period of time to obtain the original model. However, as time goes by, new training data that can be used to train the machine learning model will be generated. Related technologies use incremental learning to iteratively train the original model using the newly added training data to obtain the target model, thereby updating the original model.
[0003] In the specific incremental learning process, the original model is iteratively trained using the training data corresponding to the new category and the training data corresponding to the old category. Since the amount of training data corresponding to the new category is different from the amount of training data corresponding to the old category, the prediction values output by the target model on the new and old categories are unbalanced.
[0004] For example, when the amount of training sample image data corresponding to a new category is greater than that of the old category, the target model tends to output a higher prediction value for the new category and a lower prediction value for the old category, thereby affecting the accuracy of the model prediction. Summary of the invention
[0005] The embodiments of the present application provide an image processing and image classification method, apparatus, device and storage medium for improving the accuracy of model prediction.
[0006] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:
[0007] Iteratively train the original classification model using the newly added sample image set and the historical sample image set until the iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category, and the historical sample image set corresponds to multiple old categories;
[0008] Acquire multiple corrected sample images, and input each corrected sample image into the candidate classification model to obtain a first new category prediction value corresponding to the at least one new category and a first old category prediction value corresponding to the multiple old categories of the corresponding corrected sample image, wherein each corrected sample image corresponds to one new category or one old category;
[0009] A target offset is determined based on the obtained first new category prediction value and the first old category prediction value, and the target offset is used to perform output correction on the candidate classification model to obtain a target classification model.
[0010] On the one hand, an embodiment of the present application provides an image classification method, the method comprising:
[0011] Inputting the image to be processed into a target classification model to obtain initial prediction values of the image to be processed corresponding to each of the multiple old categories and a corrected target prediction value corresponding to at least one new category, wherein the target classification model is obtained by using the above-mentioned image processing method;
[0012] Based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category, a target category corresponding to the image to be processed is determined from the multiple old categories and the at least one new category.
[0013] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:
[0014] A model training module, used to iteratively train the original classification model using a newly added sample image set and a historical sample image set until an iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category, and the historical sample image set corresponds to multiple old categories;
[0015] A first prediction module, used for acquiring a plurality of corrected sample images, and inputting each of the corrected sample images into the candidate classification model, to obtain a first new category prediction value corresponding to the at least one new category, and a first old category prediction value corresponding to the plurality of old categories of the corresponding corrected sample image, wherein each of the corrected sample images corresponds to one of the new categories or one of the old categories;
[0016] A correction module is used to determine a target offset based on the obtained first new category prediction value and the first old category prediction value, and use the target offset to perform output correction on the candidate classification model to obtain a target classification model.
[0017] Optionally, the correction module is specifically used for:
[0018] Determine a first average energy value corresponding to the at least one new category based on the obtained first new category prediction value;
[0019] Determining second average energy values corresponding to the plurality of old categories based on the obtained first old category prediction value;
[0020] The difference between the first average energy value and the second average energy value is used as the target offset.
[0021] Optionally, the correction module is specifically used for:
[0022] The target offset is added to the output layer of the candidate classification model to obtain the target classification model, wherein the target offset is used to correct the new category prediction value corresponding to the at least one new category.
[0023] Optionally, the historical sample image set is selected from historical training data used to train the original classification model;
[0024] The correction module is also used for:
[0025] The target offset is used to perform output correction on the candidate classification model. After obtaining the target classification model, the newly added sample image set is added as a historical sample to the historical training data, and the at least one new category is added as an old category to the multiple old categories.
[0026] Optionally, the model training module is also used to:
[0027] Before iteratively training the original classification model using the newly added sample image set and the historical sample image set, the original classification model is determined in any of the following ways: using the initialized classification model as the original classification model;
[0028] Using the classification model obtained by training with historical training data as the original classification model;
[0029] The classification model obtained by training with historical training data and output correction with historical offsets is used as the original classification model.
[0030] Optionally, the first prediction module is specifically used to:
[0031] The plurality of corrected sample images are respectively selected from the newly added sample image set and the historical sample image set; or,
[0032] The plurality of corrected sample images are obtained by selecting from other sample sets other than the newly added sample image set and the historical sample image set.
[0033] Optionally, the model training module is specifically used for:
[0034] Input each newly added sample image in the set of newly added sample images into the original classification model to obtain a second new category prediction value corresponding to the at least one new category and a second old category prediction value corresponding to the plurality of old categories of the corresponding newly added sample image;
[0035] Input each historical sample image in the historical sample image set into the original classification model to obtain a third new category prediction value corresponding to the at least one new category and a third old category prediction value corresponding to the plurality of old categories of the corresponding historical sample image;
[0036] Based on the obtained second new category prediction value, second old category prediction value, third new category prediction value and third old category prediction value, a target loss value is determined, and the target loss value is used to adjust the parameters of the original classification model.
[0037] Optionally, the model training module is specifically used for:
[0038] Determine a cross entropy loss value based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value, and the third old category prediction value;
[0039] Determine a knowledge distillation loss value based on the obtained second oldest category prediction value and the third oldest category prediction value;
[0040] The target loss value is determined based on the cross entropy loss value and the knowledge distillation loss value.
[0041] Optionally, the original classification model is obtained by correcting the output of the corresponding historical classification model using the historical offset;
[0042] The model training module is specifically used for:
[0043] Inputting each newly added sample image in the set of newly added sample images into the historical classification model to obtain a first candidate prediction value corresponding to the plurality of old categories of the corresponding newly added sample image;
[0044] Inputting each historical sample image in the historical sample image set into the historical classification model to obtain a second candidate prediction value corresponding to the plurality of old categories of the corresponding historical sample image;
[0045] A knowledge distillation loss value is determined based on the obtained first candidate prediction value, the second candidate prediction value, the second old category prediction value, and the third old category prediction value.
[0046] On the one hand, an embodiment of the present application provides an image classification device, the device comprising:
[0047] A second prediction module is used to input the image to be processed into a target classification model to obtain initial prediction values of the image to be processed corresponding to each of the multiple old categories and a corrected target prediction value corresponding to at least one new category, wherein the target classification model is obtained by using the above-mentioned image processing device;
[0048] A classification module is used to determine the target category corresponding to the image to be processed from the multiple old categories and the at least one new category based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category.
[0049] Optionally, the second prediction module is specifically used to:
[0050] By using the target classification model, feature extraction is performed on the image to be processed to obtain a target feature image, and based on the target feature image, initial prediction values of the image to be processed corresponding to each of the at least one new category and the plurality of old categories are determined;
[0051] The target classification model is used to correct the initial prediction value corresponding to each of the at least one new category using the target offset to obtain the target prediction value corresponding to each of the at least one new category.
[0052] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image processing and image classification methods when executing the program.
[0053] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned image processing and image classification methods.
[0054] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the above-mentioned image processing and image classification methods.
[0055] In the embodiment of the present application, the original classification model is iteratively trained using the new and old sample image sets to obtain a candidate classification model, so the candidate classification model has learned the classification ability of the new category while not forgetting the classification ability of the old category. The candidate classification model is used to perform classification prediction on multiple corrected sample images, and based on the obtained prediction values of the multiple corrected sample images on the new and old categories, the target offset is determined, and then the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, so that the prediction values output by the target classification model on the new and old categories are balanced, thereby improving the classification accuracy of the classification model. In addition, the target offset is directly used to correct the output of the candidate separation model, and there is no need to achieve output correction of the candidate separation model through repeated training, thereby improving the efficiency of incremental learning and reducing training resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0057] Figure 1a A schematic diagram of the correlation between the amount of category training data and the category energy value provided in an embodiment of the present application;
[0058] Figure 1b A schematic diagram of the correlation between the amount of category training data and the category energy value provided in an embodiment of the present application;
[0059] Figure 1c A schematic diagram of a system architecture provided in an embodiment of the present application;
[0060] Figure 2 A schematic diagram of a search interface provided in an embodiment of the present application;
[0061] Figure 3 A schematic diagram of an image to be searched provided in an embodiment of the present application;
[0062] Figure 4 A schematic diagram of a search result interface provided in an embodiment of the present application;
[0063] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;
[0064] Figure 6 A flowchart of a model training method provided in an embodiment of the present application;
[0065] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;
[0066] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application;
[0067] Figure 9a A schematic diagram of an effect comparison result provided in an embodiment of the present application;
[0068] Figure 9b A schematic diagram of an effect comparison result provided in an embodiment of the present application;
[0069] Fig.10 A flowchart of an image classification method provided in an embodiment of the present application;
[0070] Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0071] Fig.12 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application;
[0072] Fig.13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0074] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.
[0075] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0076] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0077] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The image processing method in the embodiment of the present application is applicable to the training and reasoning of any machine learning model, and is particularly suitable for the training and reasoning of any machine learning model based on a softmax classification head.
[0078] Energy function model: (energy-based model, referred to as EBM), considers two sets of random variables and in and represent the entire distribution space of variables x and y respectively. The energy function E θ (x, y)) maps a set of matching pairs (x, y) into a scalar energy value, where θ represents a trainable parameter. Usually, the smaller the energy value, the more compatible the matching pair (x, y), while the larger the energy value, the less compatible (x, y).
[0079] In practical applications, energy values need to be normalized for use in different scenarios. The most common approach is to use Boltzmann distribution to convert the energy value into a probability density, as shown in formula (1):
[0080]
[0081] Energy Alignment (EA) means that the energy values corresponding to different categories are the same.
[0082] LwF: learning without forgetting (learning without forgetting algorithm, an incremental learning algorithm).
[0083] iCaRL: incremental classifier and representation learning (incremental classifier and representation learning algorithm, an incremental learning algorithm).
[0084] EEIL: end-to-end incremental learning (end-to-end incremental learning algorithm, an incremental learning algorithm).
[0085] BiC: bias correction (bias correction algorithm, an incremental learning algorithm).
[0086] RPS: random path selection (random path selection algorithm, an incremental learning algorithm).
[0087] WA: weight aligning (weight alignment algorithm, an incremental learning algorithm).
[0088] The design concept of the embodiments of the present application is introduced below.
[0089] In the specific incremental learning process, the original model is iteratively trained using the training data corresponding to the new category and the training data corresponding to the old category. Since the amount of training data corresponding to the new category is different from the amount of training data corresponding to the old category, the prediction values output by the target model on the new and old categories are unbalanced. For example, when the amount of training sample image data corresponding to the new category is greater than the amount of training sample image data corresponding to the old category, the target model tends to output a higher prediction value on the new category and a lower prediction value on the old category, thereby affecting the accuracy of the model prediction.
[0090] Through analysis, it is found that the classification model f θ (x) generally maps the input x to C logits outputs, where logits output refers to the output of the fully connected layer and θ represents the trainable network parameters in the classification model. The C logits outputs are mapped to the predicted probabilities of each category through the softmax function, as shown in the following formula (2):
[0091]
[0092] Among them, P θ (y|x) represents the conditional probability of y under input x, and also represents the predicted probability of category y under input x, f θ (x)[y] represents the logits output of category y under input x.
[0093] By comparing the above formula (1) and formula (2), it can be seen that without changing the network parameters, the logits output of the classification model can be directly used as the negative energy value of the matching pair (x, y), which specifically satisfies the following formula (3):
[0094] -E θ (x, y) = f θ(x)[y]……………(3)
[0095] From a physical sense, it can be found that the lower the energy value, the higher the logit output of the classification model, which means that the network tends to judge that the input x corresponds to category y; conversely, the higher the energy value, the lower the logit output of the classification model, which means that the network does not tend to judge that the input x corresponds to category y.
[0096] Furthermore, based on the above formula (1), the joint probability of (x, y) can be obtained, as shown in the following formula (4):
[0097]
[0098] in,
[0099] At the same time, the marginal probability of y can also be obtained, as shown in the following formula (5):
[0100]
[0101] Among them, E θ (y) is the energy value of variable y, which specifically satisfies the following formula (6):
[0102]
[0103] Taking the logarithm of both sides of the above formula (5), we can get the following formula (7):
[0104] logp θ (y) = -E θ (y)-logZ θ …………(7)
[0105] From the above formula (7), we can see that the negative energy of variable y -E θ (y) and logP θ (y) is linearly positively correlated. That is to say, when training a classification model, if the distribution of training data in different categories is unbalanced, the marginal probabilities corresponding to different categories will also be different, and accordingly, the energy values corresponding to different categories will also be different, which will lead to unbalanced prediction values of the classification model corresponding to different categories.
[0106] In order to ensure that the predicted values of the classification model output corresponding to different categories are balanced when using imbalanced distribution training data for incremental learning of the classification model, that is, the marginal probabilities corresponding to different categories are the same, it is necessary to achieve the ideal state that the energy values corresponding to different categories are the same, as shown in the following formula (8):
[0107] -E θ(y=i)=-E θ (y=j)………………(8)
[0108] Among them, 1≤i≤C, 1≤j≤C, i≠j.
[0109] However, the above formula (8) can only be achieved when the distribution of training data in different categories is balanced. If the distribution of training data in different categories is unbalanced, it is necessary to align the energy values corresponding to different categories through an offset, as shown in the following formula (9):
[0110] -E θ (y=i)=-E θ (y=j)+α j …………(9)
[0111] Among them, α j Indicates the offset.
[0112] In the case of energy alignment, the energy value of variable y and the classification model f can be obtained by combining the above formula (3) and formula (6) θ The relationship between (.) is shown in the following formula (10):
[0113]
[0114] Furthermore, the above formula (10) can be rewritten as the following formula (11):
[0115]
[0116] It can be seen from the above formula (11) that after specifying a category i as the base category, for any other category j, an offset α can be calculated j , using the offset α j Correct the output of the classification model, that is, the logit output of category j predicted by the classification model With offset α j Add them together to obtain the corrected logit output of category j. At this point, even if the distribution of training data in category i and category j is unbalanced, the energy values corresponding to category i and category j can be made the same, and thus the prediction values of the classification model corresponding to category i and category j are balanced.
[0117] Based on the above formula (11), the offset α can be further obtained j The calculation formula is as shown in formula (12):
[0118]
[0119] It can be seen from the above formula (12) that based on the logit output of category i predicted by the classification model and the logit output of category j predicted by the classification model, the offset α can be obtained j .
[0120] Corresponding to the incremental learning scenario in the present application, multiple old categories can be used as a base class cluster, and at least one new category can be used as a class cluster to be corrected. For at least one new category in the class cluster to be corrected, the above formula (12) is used to calculate an offset, and then the offset is used to perform output correction on the classification model before correction to obtain the classification model after correction.
[0121] In order to verify the accuracy of the above derivation process, this application conducts the following experiments using categories 1 to 10 as examples, where categories 1 to 5 are new categories, and categories 6 to 10 are old categories. The amount of training data for the new categories is much larger than that for the old categories, and the training samples are images.
[0122] For the classification model before correction, Figure 1a The relationship between the amount of category training data and the category energy value is shown as an example. Figure 1a It can be seen that the amount of training data for the new category is much larger than that for the old category. Correspondingly, the negative energy value output by the classification model before correction on the new category is also much larger than the negative energy value output on the old category. That is, when the amount of training data corresponding to the new and old categories is different, the energy values output by the classification model before correction on the new and old categories will be different, which will lead to an imbalance in the predicted values output by the classification model before correction on the new and old categories.
[0123] For the corrected classification model, Figure 1b The relationship between the amount of category training data and the category energy value is shown as an example. Figure 1b It can be seen that the amount of training data for the new category is much larger than that for the old category. At this time, the negative energy value output by the corrected classification model on the new category is basically close to the negative energy value output on the old category. In other words, even if the amount of training data corresponding to the new and old categories is different, the energy value output by the corrected classification model on the new and old categories is the same, which makes the prediction value output by the corrected classification model on the new and old categories balanced.
[0124] Based on the derivation process, an embodiment of the present application provides an image processing method, which includes:
[0125] The original classification model is iteratively trained using a newly added sample image set and a historical sample image set until the iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category and the historical sample image set corresponds to multiple old categories. Then, multiple corrected sample images are obtained, and each corrected sample image is input into the candidate classification model to obtain the first new category prediction value corresponding to the corresponding corrected sample image in at least one new category, and the first old category prediction value corresponding to multiple old categories, each corrected sample image corresponding to a new category or an old category. Then, the target offset is determined based on the obtained first new category prediction value and the first old category prediction value, and the target offset is used to perform output correction on the candidate classification model to obtain a target classification model.
[0126] In the embodiment of the present application, the original classification model is iteratively trained using the new and old sample image sets to obtain a candidate classification model, so the candidate classification model has learned the classification ability of the new category while not forgetting the classification ability of the old category. The candidate classification model is used to perform classification prediction on multiple corrected sample images, and based on the obtained prediction values of the multiple corrected sample images on the new and old categories, the target offset is determined, and then the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, so that the prediction values output by the target classification model on the new and old categories are balanced, thereby improving the classification accuracy of the classification model. In addition, the target offset is directly used to correct the output of the candidate separation model, and there is no need to achieve output correction of the candidate separation model through repeated training, thereby improving the efficiency of incremental learning and reducing training resource consumption.
[0127] See also Figure 1c , which is a system architecture diagram applicable to the embodiment of the present application. The architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and servers 102.
[0128] The terminal device 101 may be installed with a target application, wherein the target application may be a client application, a web application, a small program application, etc. In actual application scenarios, the target application may be any application with a text detection function. The terminal device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a smart speaker, a smart watch, a smart car device, etc., but is not limited thereto. The embodiments of the present application may be applied to various scenarios, including but not limited to the map field, car-mounted scenarios, cloud technology, artificial intelligence, smart transportation, and assisted driving.
[0129] The server 102 may be a background server of the target application, providing corresponding services for the target application. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.
[0130] The image processing method and the image classification method in the embodiments of the present application may be executed by the terminal device 101, or by the server 102, or may be executed interactively by the terminal device 101 and the server 102.
[0131] The following is a specific description of the image processing method performed by the server 102 as an example:
[0132] The terminal device 101 sends the newly added sample image set to the server 102. The server 102 uses the newly added sample image set and the historical sample image set to iteratively train the original classification model until the iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category and the historical sample image set corresponds to multiple old categories. Then, multiple corrected sample images are obtained, and each corrected sample image is input into the candidate classification model to obtain the first new category prediction value corresponding to the corresponding corrected sample image in at least one new category, and the first old category prediction value corresponding to multiple old categories, each corrected sample image corresponds to a new category or an old category. Then, the target offset is determined based on the obtained first new category prediction value and the first old category prediction value, and the target offset is used to perform output correction on the candidate classification model to obtain a target classification model.
[0133] After the target classification model is obtained, the target classification model is stored in the server 102 .
[0134] The following is a specific description of the image classification method performed by the server 102 as an example:
[0135] The terminal device 101 sends the image to be processed to the server 102. The server 102 inputs the image to be processed into the target classification model to obtain the initial prediction values corresponding to each of the multiple old categories and the corrected target prediction value corresponding to at least one new category of the image to be processed. Then, based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category, the target category corresponding to the image to be processed is determined from the multiple old categories and the at least one new category.
[0136] The server 102 sends the target category corresponding to the image to be processed to the terminal device 101. The terminal device 101 displays the target category corresponding to the image to be processed on the display interface.
[0137] In practical applications, the image processing method in the embodiment of the present application is applicable to the training of any image classification model. The image classification method in the embodiment of the present application is applicable to scenarios including but not limited to: intelligent video analysis, traffic scene object recognition, vehicle counting, license plate detection, image retrieval, automatic album classification, etc. The following is an example of an image retrieval scenario:
[0138] Image classification model training phase:
[0139] The terminal device 101 sends the newly added sample image set to the server 102, wherein the newly added sample image set corresponds to two new categories, namely "sofa" and "dining table". The server 102 has previously stored a historical sample image set, which corresponds to three old categories, namely "tableware", "carpet", and "storage box".
[0140] The server 102 uses the newly added sample image set and the historical sample image set to iteratively train the original classification model until the iteration stop condition is met to obtain a candidate classification model. Then, multiple corrected sample images are obtained, and each corrected sample image is input into the candidate classification model to obtain the first new category prediction value corresponding to the corresponding corrected sample image in at least one new category, and the first old category prediction value corresponding to multiple old categories, each corrected sample image corresponding to a new category or an old category. Then, based on the obtained first new category prediction value and the first old category prediction value, the target offset is determined, and the target offset is used to perform output correction on the candidate classification model to obtain the target classification model. After obtaining the target classification model, the target classification model is saved in the server 102.
[0141] Image retrieval stage:
[0142] The terminal device 101 obtains the image to be searched in response to the search operation triggered in the search interface, wherein the search interface is as follows: Figure 2 As shown, the image to be searched is Figure 3As shown. The terminal device 101 sends the image to be searched to the server 102. The server 102 inputs the image to be searched into the target classification model, obtains the initial prediction values corresponding to the three old categories and the corrected target prediction values corresponding to the two new categories. Then, based on the initial prediction values corresponding to the three old categories and the target prediction values corresponding to the two new categories, it is determined that the target category corresponding to the image to be searched is the new category "sofa" from the three old categories and the two new categories.
[0143] The server 102 obtains the search result image related to the new category "sofa" from the search library, and sends the search result image to the terminal device 101. The terminal device 101 displays the search result image in the search result interface, as shown in FIG. Figure 4 shown.
[0144] based on Figure 1c The system architecture diagram shown in the figure, the embodiment of the present application provides a process of an image processing method, such as Figure 5 As shown, the process of this method can be Figure 1c The terminal device 101 or the server 102 shown is executed, or the terminal device 101 and the server 102 can be interactively executed, including the following steps:
[0145] Step S501, using the newly added sample image set and the historical sample image set, iteratively train the original classification model until the iteration stop condition is met to obtain a candidate classification model.
[0146] Specifically, the newly added sample image set and the historical sample image set are non-overlapping sample sets. The newly added sample image set corresponds to at least one new category, and the historical sample image set corresponds to multiple old categories, and the at least one new category does not overlap with the multiple old categories.
[0147] In the newly added sample image set, each new category corresponds to at least one newly added sample image. In the historical sample image set, each old category corresponds to at least one historical sample image.
[0148] In some implementations, the iteration stopping condition may be that the number of iterations reaches a preset number, or that the loss value satisfies a preset convergence condition, or other conditions, which are not specifically limited in this application.
[0149] Step S502, obtaining multiple corrected sample images, and inputting each corrected sample image into a candidate classification model to obtain a first new category prediction value corresponding to at least one new category and a first old category prediction value corresponding to multiple old categories of the corresponding corrected sample image.
[0150] Specifically, each corrected sample image corresponds to a new category or an old category. Among multiple corrected sample images, a new category corresponds to at least one corrected sample image, and an old category corresponds to at least one corrected sample image. The corrected sample image corresponds to a first new category prediction value in each new category, and the first new category prediction value represents the predicted logits output of the corrected sample image in a new category; the corrected sample image corresponds to a first old category prediction value in each old category, and the first old category prediction value represents the predicted logits output of the corrected sample image in an old category.
[0151] In some embodiments, the total number of corrected sample images corresponding to at least one new category is the same as the total number of corrected sample images corresponding to the plurality of old categories.
[0152] Step S503, determining a target offset based on the obtained first new category prediction value and the first old category prediction value, and using the target offset to perform output correction on the candidate classification model to obtain a target classification model.
[0153] Specifically, using the target offset to correct the output of the candidate classification model may refer to using the same target offset to correct the new category prediction value corresponding to at least one new category predicted by the candidate classification model, and then outputting the corrected new category prediction value; it may also refer to using the same target offset to correct the old category prediction value corresponding to multiple old categories predicted by the candidate classification model, and then outputting the corrected old category prediction value.
[0154] In the embodiment of the present application, the original classification model is iteratively trained using the new and old sample image sets to obtain a candidate classification model, so the candidate classification model has learned the classification ability of the new category while not forgetting the classification ability of the old category. The candidate classification model is used to perform classification prediction on multiple corrected sample images, and based on the obtained prediction values of the multiple corrected sample images on the new and old categories, the target offset is determined, and then the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, so that the prediction values output by the target classification model on the new and old categories are balanced, thereby improving the classification accuracy of the classification model. In addition, the target offset is directly used to correct the output of the candidate separation model, and there is no need to achieve output correction of the candidate separation model through repeated training, thereby improving the efficiency of incremental learning and reducing training resource consumption.
[0155] Optionally, in the above step S501, before the original classification model is iteratively trained using the newly added sample image set and the historical sample image set, the original classification model is determined by any of the following methods:
[0156] Method 1: Use the initialized classification model as the original classification model.
[0157] Method 2: The classification model obtained by training with historical training data is used as the original classification model.
[0158] Specifically, the original classification model is obtained by iteratively training the initialized classification model using historical training data. At this time, the network parameters in the original classification model are not the initialized network parameters.
[0159] The original classification model may also be obtained by iteratively training a trained classification model using historical training data. In this case, the network parameters in the trained classification model and the original classification model are not initialized network parameters.
[0160] Method 3: The classification model obtained by training with historical training data and output correction with historical offset is used as the original classification model.
[0161] Specifically, the original classification model is obtained by iteratively training the initialized classification model using historical training data, and performing output correction using historical offsets after the training. At this time, the network parameters in the original classification model are not the initialized network parameters.
[0162] The original classification model can also be obtained by iteratively training a trained classification model using historical training data, and then correcting the output using historical offsets after the training. At this time, the network parameters in the trained classification model and the original classification model are not initialized network parameters.
[0163] The image processing method in the embodiment of the present application is applicable to various classification models, such as the original classification model being an initialized classification model, a trained classification model, a trained classification model with output correction, etc. It has a wide range of applicability and high versatility.
[0164] Optionally, in the above step S501, when the original classification model is iteratively trained, each iteration process includes the following steps: Figure 6 As shown:
[0165] Step S601, input each newly added sample image in the newly added sample image set into the original classification model to obtain a second new category prediction value corresponding to at least one new category and a second old category prediction value corresponding to multiple old categories of the corresponding newly added sample image.
[0166] Specifically, a newly added sample image corresponds to a second new category prediction value in each new category, and the second new category prediction value represents: the predicted logits output of the newly added sample image in a new category output by the original classification model; a newly added sample image corresponds to a second old category prediction value in each old category, and the second old category prediction value represents: the predicted logits output of the newly added sample image in an old category output by the original classification model.
[0167] For each newly added sample image, after obtaining the second new category prediction value corresponding to at least one new category and the second old category prediction value corresponding to multiple old categories of the newly added sample image, the above formula (2) is used to determine the prediction probability of the newly added sample image corresponding to at least one new category and multiple old categories based on the obtained second new category prediction values and second old category prediction values.
[0168] For each newly added sample image, the real category corresponding to the newly added sample image is marked in advance, and the real probabilities corresponding to the newly added sample image in at least one new category and multiple categories are marked accordingly.
[0169] For example, set at least one new category to be new category 1 and new category 2, and multiple old categories to be old category 1, old category 2, and old category 3, and the true category corresponding to the newly added sample A is new category 1, then the true probability of the newly added sample image A corresponding to the new category 1 is set to 1, the true probability corresponding to the new category 2 is set to 0, and the true probabilities corresponding to the old category 1, old category 2, and old category 3 are all set to 0.
[0170] Step S602, input each historical sample image in the historical sample image set into the original classification model to obtain a third new category prediction value corresponding to at least one new category and a third old category prediction value corresponding to multiple old categories of the corresponding historical sample image.
[0171] Specifically, a historical sample image corresponds to a third new category prediction value in each new category, and the third new category prediction value represents: the predicted logits output of the historical sample image in a new category output by the original classification model; a historical sample image corresponds to a third old category prediction value in each old category, and the third old category prediction value represents: the predicted logits output of the historical sample image in an old category output by the original classification model.
[0172] For each historical sample image, after obtaining the third new category prediction value corresponding to at least one new category and the third old category prediction value corresponding to multiple old categories of the historical sample image, the above formula (2) is used to determine the prediction probability corresponding to at least one new category and multiple old categories of the historical sample image based on the obtained third new category prediction values and third old category prediction values.
[0173] For each historical sample image, the real category corresponding to the historical sample image is pre-marked, and the real probabilities corresponding to the historical sample image in at least one new category and multiple categories are correspondingly marked.
[0174] Step S603, based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value and the third old category prediction value, a target loss value is determined, and the target loss value is used to adjust the parameters of the original classification model.
[0175] Specifically, the target loss value is determined based on the predicted probability and the true probability of each newly added sample image in at least one new category and multiple old categories, as well as the predicted probability and the true probability of each historical sample image in at least one new category and multiple old categories.
[0176] It can be determined whether the original classification model has converged based on the target loss value. When it is determined that the original classification model has not converged, the model parameters of the original classification model are adjusted based on the target loss value. The next round of training is performed using the original classification model after parameter adjustment. When it is determined that the original classification model has converged, the training is terminated and the candidate classification model is output. Of course, in the embodiment of the present application, the training can also be terminated and the trained candidate classification model can be output when the number of iterative training reaches a preset number of times. This application does not make any specific limitations on this.
[0177] In an embodiment of the present application, a newly added sample image set and a historical sample image set are used to perform incremental iterative training on the original classification model. During the training process, based on the prediction values of multiple newly added sample images for the new and old categories, and the prediction values of multiple historical sample images for the new and old categories, the network parameters are adjusted so that the original classification model gradually learns the ability to recognize new categories without forgetting the classification ability of old categories, thereby improving the accuracy of the classification model.
[0178] Optionally, in the process of iteratively training the classification model, the embodiment of the present application adopts at least the following implementation methods to determine the target loss value:
[0179] Based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value, and the third old category prediction value, a cross entropy loss value is determined. Then, based on the obtained second old category prediction value and the third old category prediction value, a knowledge distillation loss value is determined. Based on the cross entropy loss value and the knowledge distillation loss value, a target loss value is determined.
[0180] Specifically, based on the second new category prediction value and the second old category prediction value corresponding to each new sample image, the prediction probability corresponding to each of the at least one new category and the multiple old categories is determined. Based on the third new category prediction value and the third old category prediction value corresponding to each of the historical sample images, the prediction probability corresponding to each of the at least one new category and the multiple old categories is determined.
[0181] Based on the predicted probability and true probability of each newly added sample image in at least one new category and multiple old categories, and the predicted probability and true probability of each historical sample image in at least one new category and multiple old categories, the cross entropy loss value is determined, as shown in the following formula (13):
[0182]
[0183] in, represents the cross entropy loss value, y * represents the true category label, Indicates a new set of sample images. represents a set of historical sample images, represents a 0-1 function, p θ (i|x) represents the predicted probability of input x corresponding to category i, C (b) represents the number of new categories, Represents the number of old categories.
[0184] When determining the knowledge distillation loss value, the present application embodiment provides at least the following implementation methods:
[0185] In the first implementation mode, the original classification model is obtained by correcting the output of the corresponding historical classification model using the historical offset. At this time, the knowledge distillation loss value is determined by combining the predicted value output by the historical classification model and the original classification model, including the following steps:
[0186] Each newly added sample image in the set of newly added sample images is input into the historical classification model to obtain the first candidate prediction value corresponding to the corresponding newly added sample image in multiple old categories. Then each historical sample image in the set of historical sample images is input into the historical classification model to obtain the second candidate prediction value corresponding to the corresponding historical sample image in multiple old categories. Based on the obtained first candidate prediction value, second candidate prediction value, second old category prediction value, and third old category prediction value, the knowledge distillation loss value is determined.
[0187] Specifically, a newly added sample image corresponds to a first candidate prediction value in each old category, and the first candidate prediction value indicates: the predicted logits output of the newly added sample image in an old category output by the historical classification model. A historical sample image corresponds to a second candidate prediction value in each old category, and the second candidate prediction value indicates: the predicted logits output of the historical sample image in an old category output by the historical classification model.
[0188] Based on the second old category prediction value corresponding to each new sample image output by the original classification model, the corrected prediction probability corresponding to each new sample image in the multiple old categories is determined. Based on the third old category prediction value corresponding to each historical sample image output by the original classification model, the corrected prediction probability corresponding to each historical sample image in the multiple old categories is determined.
[0189] Based on the first candidate prediction value corresponding to each new sample image output by the historical classification model, the uncorrected prediction probability corresponding to each new sample image in multiple old categories is determined. Based on the second candidate prediction value corresponding to each historical sample image output by the historical classification model, the uncorrected prediction probability corresponding to each historical sample image in multiple old categories is determined.
[0190] Based on the corrected prediction probabilities and uncorrected prediction probabilities corresponding to each newly added sample image in multiple old categories, and the corrected prediction probabilities and uncorrected prediction probabilities corresponding to each historical sample image in multiple old categories, the knowledge distillation loss value is determined, as shown in the following formula (14):
[0191]
[0192] in, represents the knowledge distillation loss value, represents the number of old categories, represents the corrected predicted probability of input x corresponding to category i, represents the corrected classification model (original classification model), α represents the historical offset;
[0193] q θ(i|x) represents the uncorrected predicted probability of input x corresponding to category i, f θ (.) represents the classification model before correction (historical classification model), and T represents a temperature parameter.
[0194] In an embodiment of the present application, in the process of training the classification model, the knowledge distillation loss value is determined based on the category prediction values output by the classification model before and after correction. Therefore, the target loss value is determined based on the knowledge distillation loss value and the cross entropy loss value. When the target loss value is used to adjust the model parameters, the classification model can retain the classification ability of the old categories as much as possible, thereby improving the accuracy of the classification model.
[0195] In the second implementation mode, the original classification model is a historical classification model without output correction. In this case, the knowledge distillation loss value is determined directly based on the second old category prediction values and the third old category prediction values output by the original classification model.
[0196] After determining the cross entropy loss value and knowledge distillation loss value in the manner described above, the target loss value is determined using the following formula (15):
[0197]
[0198] in, Represents the target loss value, and λ is used to balance the cross entropy loss value and the knowledge distillation loss value. λ base is a hyperparameter.
[0199] In an embodiment of the present application, during the process of training the original classification model, a target loss value is determined based on the knowledge distillation loss value and the cross entropy loss value, and the target loss value is used to adjust the model parameters, so that the original classification model gradually learns the ability to recognize new categories while not forgetting the classification ability of old categories, thereby improving the accuracy of the classification model.
[0200] Optionally, in the above step S502, multiple corrected sample images can be selected from the newly added sample image set and the historical sample image set respectively; multiple corrected sample images can also be selected from other sample sets other than the newly added sample image set and the historical sample image set; multiple corrected sample images can also be selected from the newly added sample image set, the historical sample image set and other sample sets.
[0201] In a specific implementation, each corrected sample image corresponds to a new category or an old category, and multiple corrected sample images can be randomly extracted from the newly added sample image set and the historical sample image set. The sample sets other than the newly added sample image set and the historical sample image set can be in the verification set and the test set. Among the multiple corrected sample images, the total number of corrected sample images corresponding to at least one new category is the same as the total number of corrected sample images corresponding to the multiple old categories.
[0202] The incremental learning method in the embodiment of the present application supports directly selecting and obtaining multiple corrected sample images from a newly added sample image set and a historical sample image set, and / or selecting and obtaining multiple corrected sample images from other sample sets, thereby improving the diversity of the corrected sample images obtained, and further improving the accuracy of the output correction of the model.
[0203] Optionally, it has been deduced above that the target offset can be determined by using the above formula (12) to obtain the target offset. However, the above formula (12) involves two integral terms, which are and In order to avoid complicated integral calculations, the embodiment of the present application uses Monte Carlo integration to perform approximate calculations on the above two integral terms respectively.
[0204] Specifically, for category i, Monte Carlo integration is used to transform the integral term The approximate calculation is as follows:
[0205]
[0206] in, represents the subspace containing the i-th category, Data Points is collected from the uniform distribution q(x).
[0207] Monte Carlo integration is used to transform the integral term The method of approximate calculation is similar to that of using Monte Carlo integration to convert the integral term The method for performing the approximate calculation is the same and will not be repeated here.
[0208] After approximating the above two integral terms, the calculation formula of the offset can be adjusted to the following formula (17):
[0209]
[0210] in, Z s =f θ (x s )[i].
[0211] In addition, in order to reduce the estimation error, all categories can be divided into multiple clusters. Since the number of samples corresponding to the categories in the same cluster is close, they can correspond to the same offset. In view of this, in an embodiment of the present application, multiple old categories are used as a base cluster and at least one new category is used as a cluster to be corrected. The above formula (17) is adaptively modified and used to calculate a target offset based on the principle of energy alignment between the base cluster and the cluster to be corrected.
[0212] Specifically, after inputting a plurality of corrected sample images into the candidate classification model, a first average energy value corresponding to at least one new category is determined based on the obtained first new category prediction value. Then, a second average energy value corresponding to a plurality of old categories is determined based on the obtained first old category prediction value. The difference between the first average energy value and the second average energy value is used as the target offset.
[0213] For each new category, an energy value corresponding to the new category is determined based on the first new category prediction values corresponding to each of the multiple corrected sample images in the new category. Then, the first new category prediction values corresponding to at least one new category are averaged to determine a first average energy value.
[0214] For each old category, based on the first old category prediction values corresponding to each of the multiple corrected sample images in the old category, the energy value corresponding to the old category is determined. Then, the first old category prediction values corresponding to the multiple old categories are averaged to determine a second average energy value.
[0215] Then, the difference between the second average energy value and the first average energy value is used as the target offset, as shown in the following formula (18):
[0216]
[0217] in, Represents a base class cluster, which includes multiple old categories. represents the cluster to be corrected, which includes at least one new category, C i represents the number of old categories, C j represents the number of new categories, and S represents the number of corrected sample images.
[0218] It should be noted that in the embodiments of the present application, at least one new category may be used as a base category cluster, and multiple old categories may be used as category clusters to be corrected. The present application does not make any specific limitation on this.
[0219] In an embodiment of the present application, based on the principle of energy alignment between the base class cluster and the class cluster to be corrected, the target offset is calculated, and then the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, thereby ensuring that the prediction values output by the target classification model on the new and old categories are balanced, thereby improving the classification accuracy of the classification model.
[0220] Optionally, in the above step S503, the embodiment of the present application uses at least the following method to perform output correction on the candidate classification model to obtain the target classification model:
[0221] A target offset is added to an output layer of the candidate classification model to obtain a target classification model, wherein the target offset is used to correct a new category prediction value corresponding to at least one new category.
[0222] Specifically, after training to obtain a candidate classification model, the target offset is added as a correction parameter in the output layer of the candidate classification model to obtain a target classification model. Then, when the target classification model is subsequently used to classify, the correction parameter can be used to correct the new category prediction value corresponding to at least one new category to output a new category prediction value with higher accuracy.
[0223] It should be noted that after adding the target offset as a correction parameter in the output layer of the candidate classification model, the correction parameter can also be used to correct the old category prediction values corresponding to multiple old categories. This application does not make any specific limitations on this.
[0224] For example, if Figure 7 As shown, the newly added sample set L and the historical sample set M are used to iteratively train the original classification model to obtain a candidate classification model 701, wherein the historical sample set M corresponds to two old categories, namely old category 1 and old category 2, and the newly added sample set L corresponds to two new categories, namely new category 1 and new category 2, and the samples in the newly added sample set L and the historical sample set M are all images.
[0225] Randomly select sample images from the newly added sample set L and the historical sample set M as the corrected sample images to obtain the corrected sample set N, input each corrected sample image in the corrected sample set N into the candidate classification model, and obtain the first new category prediction value corresponding to the two new categories and the first old category prediction value corresponding to the two old categories of the corresponding corrected sample image, and each corrected sample corresponds to a new category or an old category.
[0226] The target offset 702 is determined based on the obtained first new category prediction value and the first old category prediction value, and then the target offset 702 is added to the output layer of the candidate classification model 701 to obtain the target classification model 703, wherein the target classification model 703 uses the target offset 702 to correct the new category prediction values corresponding to the new category 1 and the new category 2 respectively.
[0227] In the implementation of this application, the target offset is used to correct the output of the candidate classification model to obtain the target classification model, so that the prediction value output by the target classification model on the new and old categories is balanced, thereby improving the classification accuracy of the classification model. In addition, the target offset is directly used to correct the output of the candidate separation model, and there is no need to achieve output correction of the candidate separation model through repeated training, thereby improving the efficiency of incremental learning and reducing the consumption of training resources.
[0228] Optionally, the above historical sample image set is selected from the historical training data used to train the original classification model. After the target offset is used to correct the output of the candidate classification model and the target classification model is obtained, the newly added sample image set is added as a historical sample to the historical training data, and at least one new category is added as an old category to multiple old categories, so as to update the historical training data and the old categories.
[0229] In the specific implementation, steps S501 to S503 described in the foregoing correspond to one incremental learning. In the actual application process, multiple incremental learnings can be performed according to actual needs. The new categories corresponding to the multiple incremental learnings do not overlap, and accordingly, the newly added sample image sets of each incremental learning do not overlap. The number of new categories corresponding to each enhanced learning can be the same or different. After each incremental learning, at least one new category of this incremental learning is added as an old category to the multiple old categories, and the updated multiple old categories are used for the next incremental learning. In addition, the newly added sample image set is added to the historical training data as a historical sample. In the next incremental learning, a historical sample image set is selected from the updated historical training data. At this time, the historical sample image set corresponds to the updated multiple old categories. The number of historical samples in the historical sample image set selected for each incremental learning is the same, and the method of each incremental learning is steps S501 to S503 described in the foregoing, which will not be repeated here.
[0230] For example, the training data set for incremental learning is set to Among them, B represents the number of incremental learning, is a new sample image collection, Corresponding to C (b) New categories, is a historical sample image set, a historical sample image set From historical training data A collection of historical sample images selected from correspond Old categories, among which, and b>1. It is assumed that in each incremental learning, the number of historical sample images in the historical sample image set is the same, that is, Therefore, as the number of categories increases, the number of historical samples corresponding to each old category in the historical sample image collection becomes smaller, and the imbalance between the new and old categories becomes more serious.
[0231] Based on this, the embodiment of the present application provides a method for multiple incremental learning based on energy alignment, such as Figure 8 As shown, the following steps are included:
[0232] Step S801: Obtaining a training data set for incremental learning
[0233] Step S802, initializing the classification model f θ .
[0234] Step S803: Use the first newly added sample image set Based on formula (13), the initialized classification model is iteratively trained to obtain the target classification model 1.
[0235] Step S804, using the target classification model 1 as the teacher model for the next incremental learning.
[0236] Specifically, the first newly added sample image set Add it to the historical training data as a historical sample, and add the first newly added sample image set The corresponding C (1) The new categories are added as old categories to the old categories collection.
[0237] Step S805: Obtain the bth newly added sample image set from the training data set Randomly extract the bth historical sample image set from the historical training data
[0238] Among them, the historical sample image collection Corresponding to all old categories in the old category set, where b is greater than 1.
[0239] Step S806: Use the bth newly added sample image set and the bth historical sample image set Based on the above formula (15), the teacher model obtained by the previous incremental learning is iteratively trained to obtain the candidate classification model b.
[0240] Step S807: From the bth newly added sample image set and the bth historical sample image set A plurality of corrected sample images are selected, and the above formula (18) is used to determine the target offset based on the plurality of corrected sample images.
[0241] Among them, the historical sample image collection The corresponding multiple old categories are base class clusters, and the new sample image set is added The corresponding at least one new category is the cluster to be corrected.
[0242] Step S808, use the target offset to correct the output of the candidate classification model b, obtain the target classification model b, and the target classification model b will be used as the teacher model for the next incremental learning.
[0243] The bth newly added sample image set Add the bth newly added sample image set to the historical training data as a historical sample The corresponding C (b) The new categories are added as old categories to the old categories collection.
[0244] Step S809, set b=b+1.
[0245] Step S810, determine whether b is greater than B, if so, execute step S811, otherwise execute step S805.
[0246] Step S811, output target classification model b-1.
[0247] In order to verify the effect of the incremental learning method based on energy alignment, this application uses the training dataset ImageNet to conduct relevant experiments, where ImageNet contains 1.2 million training images and 50,000 verification images. ImageNet 100 contains 100 randomly selected categories, and ImageNet1000 contains 1000 categories. All experiments use ResNet18 as the backbone network and are implemented using Pytorch.
[0248] In the experimental setting, ImageNet 100 and ImageNet 1000 both underwent 10 incremental learnings, with 10 new categories and 100 new categories added in each incremental learning. In addition, 2,000 and 20,000 images of old categories were retained in each incremental learning. Finally, the classification accuracy of each incremental learning and the average classification accuracy of the 10 incremental learnings were calculated.
[0249] Specifically, see Table 1 below, which is a comparison of the effects of the incremental learning method in this application and other incremental learning algorithms on the ImageNet 100 dataset.
[0250] Table 1.
[0251]
[0252]
[0253] See Table 1 below, which shows the comparison results of the incremental learning method in this application and other incremental learning algorithms for the ImageNet 1000 dataset.
[0254] Table 2.
[0255] Classification accuracy 1 2 3 4 5 6 7 8 9 10 average LqCy 90.1 77.7 63.9 51.8 43.0 35.5 31.6 28.4 26.4 24.3 42.5 iCaRL 90.0 83.0 77.5 70.5 63.0 57.5 53.5 50.0 48.0 44.0 60.8 EEIL 94.9 94.9 84.7 77.8 71.7 66.8 62.5 59.0 55.2 52.3 69.4 BiC 94.1 92.5 89.6 89.1 85.7 83.2 80.2 77.5 75.0 73.2 82.9 RPS - - - - - - - - - - 78.3 WA 93.9 91.5 89.4 87.7 86.5 85.6 84.5 83.2 82.1 81.1 85.7 This application 94.4 92.5 90.4 89.0 87.7 86.8 85.7 84.5 83.4 82.6 87.0
[0256] As can be seen from Tables 1 and 2 above, the incremental learning method based on energy alignment in this application has achieved the best average accuracy on both the ImageNet 100 and ImageNet 1000 datasets, surpassing other incremental learning algorithms. In addition, the incremental learning method based on energy alignment proposed in this application does not require changing the training method or the trained model parameters, but only requires adjusting the output logit of the trained model. It is a simple and effective incremental learning method that handles the data imbalance problem in incremental learning very well.
[0257] Secondly, for the dataset ImageNet 100, Figure 9a The comparison results of the incremental learning method in this application and other incremental learning algorithms in each incremental learning are shown as an example. For the dataset ImageNet 1000, Figure 9b The following example shows the comparison results of the incremental learning method in this application and other incremental learning algorithms in each incremental learning. Figure 9a and Figure 9b It can be seen that the incremental learning method based on energy correspondence in this application can greatly improve the classification accuracy of each incremental learning.
[0258] In addition, by comparing the classification confusion matrices of the incremental learning method in this application with those of other incremental learning algorithms, it can be seen that other incremental learning algorithms do not use energy alignment, so many training data that should have corresponded to old categories are classified into new categories, showing a strong classification bias. Under the incremental learning method using energy alignment in this application, the classification distribution is relatively uniform, indicating that the energy alignment algorithm plays a significant role in correcting the model bias.
[0259] based on Figure 1c The system architecture diagram shown in the figure, the embodiment of the present application provides a process of an image classification method, such as Fig.10 As shown, the process of this method can be Figure 1c The terminal device 101 or the server 102 shown is executed, or the terminal device 101 and the server 102 can be interactively executed, including the following steps:
[0260] Step S1001, input the image to be processed into the target classification model to obtain the initial prediction values corresponding to each of the multiple old categories of the image to be processed, and the corrected target prediction value corresponding to at least one new category.
[0261] Specifically, the target classification model is obtained by using the image processing method described above, which will not be repeated here. The image to be processed corresponds to an initial prediction value on each old category, and the initial prediction value represents the predicted logits output of the image to be processed in an old category. The image to be processed corresponds to a target prediction value on each new category, and the target prediction value represents the corrected predicted logits output of the image to be processed in a new category.
[0262] Step S1002, based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category, determine the target category corresponding to the image to be processed from the multiple old categories and the at least one new category.
[0263] Specifically, the above formula (2) is used to determine the target prediction probability of the image to be processed in at least one new category and each of the multiple old categories based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category. Then, the obtained target prediction probabilities are sorted in descending order of the prediction probability to obtain a sorting result. The category corresponding to the target prediction probability ranked first in the sorting result is used as the target category corresponding to the image to be processed.
[0264] In the embodiment of the present application, since the target classification model is trained by energy alignment, the prediction values output by the target classification model on the new and old categories are balanced. Therefore, when the target classification model is used to predict the target category corresponding to the image to be processed, the classification accuracy can be effectively improved.
[0265] Optionally, in the above step 1001, the embodiment of the present application adopts at least the following method to obtain the initial prediction values corresponding to each of the multiple old categories of the image to be processed, and the corrected target prediction value corresponding to at least one new category:
[0266] Through the target classification model, feature extraction is performed on the image to be processed to obtain a target feature image, and based on the target feature image, the initial prediction value corresponding to the image to be processed in at least one new category and multiple old categories is determined. Through the target classification model, the initial prediction value corresponding to each of the at least one new category is corrected by using the target offset to obtain a target prediction value corresponding to each of the at least one new category.
[0267] Specifically, the image to be processed corresponds to an initial prediction value in each new category, and the initial prediction value represents the predicted logits output of the image to be processed in a new category. After the target classification model determines the predicted logits output corresponding to the image to be processed in at least one new category based on the target feature image, the target classification model uses the above formula (11) to correct the initial prediction value corresponding to at least one new category based on the target offset to obtain the target prediction value corresponding to at least one new category.
[0268] Specifically, taking a new category as an example, the predicted logits output corresponding to the new category obtained by the target classification model is added to the target offset to obtain the target prediction value corresponding to the new category (i.e., the corrected predicted logits output).
[0269] In the embodiment of the present application, based on the principle of energy alignment between the base class cluster and the class cluster to be corrected, the target offset is calculated, and then the target offset is used to correct the output of the candidate classification model to obtain the target classification model. Therefore, when the target classification model is used to classify the image to be processed, after obtaining the initial prediction value corresponding to at least one new category of the image to be processed, the target classification model corrects the initial prediction value corresponding to at least one new category through the target offset to obtain a target prediction value with higher accuracy, thereby improving the classification accuracy when the target prediction value is used to determine the category corresponding to the image to be processed.
[0270] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of an image processing device, such as Fig.11 As shown, the device 1100 includes:
[0271] The model training module 1101 is used to iteratively train the original classification model using the newly added sample image set and the historical sample image set until the iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category and the historical sample image set corresponds to multiple old categories;
[0272] A first prediction module 1102 is used to obtain a plurality of corrected sample images, and input each of the corrected sample images into the candidate classification model to obtain a first new category prediction value corresponding to the at least one new category and a first old category prediction value corresponding to the plurality of old categories of the corresponding corrected sample image, wherein each of the corrected sample images corresponds to one of the new categories or one of the old categories;
[0273] The correction module 1103 is used to determine a target offset based on the obtained first new category prediction value and the first old category prediction value, and use the target offset to perform output correction on the candidate classification model to obtain a target classification model.
[0274] Optionally, the correction module 1103 is specifically used for:
[0275] Determine a first average energy value corresponding to the at least one new category based on the obtained first new category prediction value;
[0276] Determining second average energy values corresponding to the plurality of old categories based on the obtained first old category prediction value;
[0277] The difference between the first average energy value and the second average energy value is used as the target offset.
[0278] Optionally, the correction module 1103 is specifically used for:
[0279] The target offset is added to the output layer of the candidate classification model to obtain the target classification model, wherein the target offset is used to correct the new category prediction value corresponding to the at least one new category.
[0280] Optionally, the historical sample image set is selected from historical training data used to train the original classification model;
[0281] The correction module 1103 is also used for:
[0282] The target offset is used to perform output correction on the candidate classification model. After obtaining the target classification model, the newly added sample image set is added as a historical sample to the historical training data, and the at least one new category is added as an old category to the multiple old categories.
[0283] Optionally, the model training module 1101 is further used for:
[0284] Before iteratively training the original classification model using the newly added sample image set and the historical sample image set, the original classification model is determined by any of the following methods:
[0285] Using the initialized classification model as the original classification model;
[0286] Using the classification model obtained by training with historical training data as the original classification model;
[0287] The classification model obtained by training with historical training data and output correction with historical offsets is used as the original classification model.
[0288] Optionally, the first prediction module 1102 is specifically configured to:
[0289] The plurality of corrected sample images are respectively selected from the newly added sample image set and the historical sample image set; or,
[0290] The plurality of corrected sample images are obtained by selecting from other sample sets other than the newly added sample image set and the historical sample image set.
[0291] Optionally, the model training module 1101 is specifically used for:
[0292] Input each newly added sample image in the set of newly added sample images into the original classification model to obtain a second new category prediction value corresponding to the at least one new category and a second old category prediction value corresponding to the plurality of old categories of the corresponding newly added sample image;
[0293] Input each historical sample image in the historical sample image set into the original classification model to obtain a third new category prediction value corresponding to the at least one new category and a third old category prediction value corresponding to the plurality of old categories of the corresponding historical sample;
[0294] Based on the obtained second new category prediction value, second old category prediction value, third new category prediction value and third old category prediction value, a target loss value is determined, and the target loss value is used to adjust the parameters of the original classification model.
[0295] Optionally, the model training module 1101 is specifically used for:
[0296] Determine a cross entropy loss value based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value, and the third old category prediction value;
[0297] Determine a knowledge distillation loss value based on the obtained second oldest category prediction value and the third oldest category prediction value;
[0298] The target loss value is determined based on the cross entropy loss value and the knowledge distillation loss value.
[0299] Optionally, the original classification model is obtained by correcting the output of the corresponding historical classification model using the historical offset;
[0300] The model training module 1101 is specifically used for:
[0301] Inputting each newly added sample image in the set of newly added sample images into the historical classification model to obtain a first candidate prediction value corresponding to the plurality of old categories of the corresponding newly added sample image;
[0302] Inputting each historical sample image in the historical sample image set into the historical classification model to obtain a second candidate prediction value corresponding to the plurality of old categories of the corresponding historical sample image;
[0303] A knowledge distillation loss value is determined based on the obtained first candidate prediction value, the second candidate prediction value, the second old category prediction value, and the third old category prediction value.
[0304] In the embodiment of the present application, the original classification model is iteratively trained using the new and old sample image sets to obtain a candidate classification model, so the candidate classification model has learned the classification ability of the new category while not forgetting the classification ability of the old category. The candidate classification model is used to perform classification prediction on multiple corrected sample images, and based on the obtained prediction values of the multiple corrected sample images on the new and old categories, the target offset is determined, and then the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, so that the prediction values output by the target classification model on the new and old categories are balanced, thereby improving the classification accuracy of the classification model. In addition, the target offset is directly used to perform output correction on the candidate separation model, and there is no need to achieve output correction of the candidate separation model through repeated training, thereby improving the efficiency of incremental learning and reducing training resource consumption.
[0305] Based on the same technical concept, the present application embodiment provides a structural diagram of an image classification device, such as Fig.12 As shown, the device 1200 includes:
[0306] The second prediction module 1201 is used to input the image to be processed into a target classification model to obtain initial prediction values of the image to be processed corresponding to each of the multiple old categories and a corrected target prediction value corresponding to at least one new category, wherein the target classification model is obtained by using the above-mentioned image processing device;
[0307] The classification module 1202 is used to determine the target category corresponding to the image to be processed from the multiple old categories and the at least one new category based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category.
[0308] Optionally, the second prediction module 1201 is specifically used for:
[0309] By using the target classification model, feature extraction is performed on the image to be processed to obtain a target feature image, and based on the target feature image, initial prediction values of the image to be processed corresponding to each of the at least one new category and the plurality of old categories are determined;
[0310] The target classification model is used to correct the initial prediction value corresponding to each of the at least one new category using the target offset to obtain the target prediction value corresponding to each of the at least one new category.
[0311] In the embodiment of the present application, based on the principle of energy alignment between the base class cluster and the class cluster to be corrected, the target offset is calculated, and then the target offset is used to correct the output of the candidate classification model to obtain the target classification model. Therefore, when the target classification model is used to classify the image to be processed, after obtaining the initial prediction value corresponding to at least one new category of the image to be processed, the target classification model corrects the initial prediction value corresponding to at least one new category through the target offset to obtain a target prediction value with higher accuracy, thereby improving the classification accuracy when the target prediction value is used to determine the category corresponding to the image to be processed.
[0312] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1c The terminal device and / or server shown, such as Fig.13 As shown, it includes at least one processor 1301 and a memory 1302 connected to the at least one processor. The specific connection medium between the processor 1301 and the memory 1302 is not limited in the embodiment of the present application. Fig.13 For example, the processor 1301 and the memory 1302 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0313] In the embodiment of the present application, the memory 1302 stores instructions that can be executed by at least one processor 1301, and the at least one processor 1301 can perform the steps of the above-mentioned image processing and image classification methods by executing the instructions stored in the memory 1302.
[0314] Among them, the processor 1301 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and incremental learning and image classification can be achieved by running or executing instructions stored in the memory 1302 and calling data stored in the memory 1302. Optionally, the processor 1301 may include one or more processing units, and the processor 1301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1301. In some embodiments, the processor 1301 and the memory 1302 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0315] Processor 1301 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or can be executed by a combination of hardware and software modules in the processor.
[0316] Memory 1302 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1302 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1302 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1302 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0317] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned image processing and image classification methods.
[0318] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the above-mentioned image processing and image classification methods.
[0319] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0320] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0321] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0322] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0323] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0324] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An image processing method, characterized in that: include: The original classification model is iteratively trained using a newly added sample image set and a historical sample image set until an iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category, and the historical sample image set corresponds to multiple old categories; the original classification model is obtained by iteratively training a trained classification model using historical training data, and output correction is performed using a historical offset after the training; the newly added sample image set and the historical sample image set are non-overlapping sample sets; the at least one new category does not have overlapping categories with the multiple old categories; Acquire multiple corrected sample images from other sample sets other than the newly added sample image set and the historical sample image set, and input each of the corrected sample images into the candidate classification model to obtain a first new category prediction value corresponding to the at least one new category and a first old category prediction value corresponding to the multiple old categories of the corresponding corrected sample image, wherein each of the corrected sample images corresponds to one of the new categories or one of the old categories; and the total number of corrected sample images corresponding to the at least one new category is the same as the total number of corrected sample images corresponding to the multiple old categories; A target offset is determined based on the obtained first new category prediction value and the first old category prediction value, and the target offset is used to perform output correction on the candidate classification model to obtain a target classification model.
2. The method according to claim 1, characterized in that The step of determining the target offset based on the obtained first new category prediction value and the first old category prediction value comprises: Determine a first average energy value corresponding to the at least one new category based on the obtained first new category prediction value; Determining second average energy values corresponding to the plurality of old categories based on the obtained first old category prediction value; The difference between the first average energy value and the second average energy value is used as the target offset.
3. The method according to claim 1, characterized in that The step of using the target offset to perform output correction on the candidate classification model to obtain a target classification model includes: The target offset is added to the output layer of the candidate classification model to obtain the target classification model, wherein the target offset is used to correct the new category prediction value corresponding to the at least one new category.
4. The method according to claim 1, characterized in that The historical sample image set is selected from the historical training data used to train the original classification model; After the target offset is used to perform output correction on the candidate classification model to obtain the target classification model, the method further includes: The newly added sample image set is added as a historical sample to the historical training data, and the at least one new category is added as an old category to the multiple old categories.
5. The method according to any one of claims 1 to 4, characterized in that: When the original classification model is iteratively trained, each iteration process includes the following steps: Input each newly added sample image in the set of newly added sample images into the original classification model to obtain a second new category prediction value corresponding to the at least one new category and a second old category prediction value corresponding to the plurality of old categories of the corresponding newly added sample image; Input each historical sample image in the historical sample image set into the original classification model to obtain a third new category prediction value corresponding to the at least one new category and a third old category prediction value corresponding to the plurality of old categories of the corresponding historical sample image; Based on the obtained second new category prediction value, second old category prediction value, third new category prediction value and third old category prediction value, a target loss value is determined, and the target loss value is used to adjust the parameters of the original classification model.
6. The method according to claim 5, characterized in that The step of determining the target loss value based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value, and the third old category prediction value comprises: Determine a cross entropy loss value based on the obtained second new category prediction value, the second old category prediction value, the third new category prediction value, and the third old category prediction value; Determine a knowledge distillation loss value based on the obtained second oldest category prediction value and the third oldest category prediction value; The target loss value is determined based on the cross entropy loss value and the knowledge distillation loss value.
7. The method according to claim 6, characterized in that The original classification model is obtained by correcting the output of the corresponding historical classification model using the historical offset; The step of determining the knowledge distillation loss value based on the obtained second old category prediction value and the third old category prediction value includes: Inputting each newly added sample image in the set of newly added sample images into the historical classification model to obtain a first candidate prediction value corresponding to the plurality of old categories of the corresponding newly added sample image; Inputting each historical sample image in the historical sample image set into the historical classification model to obtain a second candidate prediction value corresponding to the plurality of old categories of the corresponding historical sample image; A knowledge distillation loss value is determined based on the obtained first candidate prediction value, the second candidate prediction value, the second old category prediction value, and the third old category prediction value.
8. An image classification method, characterized in that: include: Inputting the image to be processed into a target classification model to obtain initial prediction values corresponding to each of the multiple old categories of the image to be processed and a corrected target prediction value corresponding to at least one new category, wherein the target classification model is obtained by using the method described in any one of claims 1 to 7; Based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category, a target category corresponding to the image to be processed is determined from the multiple old categories and the at least one new category.
9. The method according to claim 8, characterized in that The step of inputting the image to be processed into the target classification model to obtain initial prediction values of the image to be processed corresponding to each of the multiple old categories and a corrected target prediction value corresponding to at least one new category includes: By using the target classification model, feature extraction is performed on the image to be processed to obtain a target feature image, and based on the target feature image, initial prediction values of the image to be processed corresponding to each of the at least one new category and the plurality of old categories are determined; The target classification model is used to correct the initial prediction value corresponding to each of the at least one new category using the target offset to obtain the target prediction value corresponding to each of the at least one new category.
10. An image processing device, characterized in that: include: A model training module, used to iteratively train the original classification model using a newly added sample image set and a historical sample image set until an iteration stop condition is met to obtain a candidate classification model, wherein the newly added sample image set corresponds to at least one new category, the historical sample image set corresponds to multiple old categories, the original classification model is obtained by iteratively training the trained classification model using historical training data, and after the training is completed, the output is corrected using a historical offset; the newly added sample image set and the historical sample image set are non-overlapping sample sets; the at least one new category does not have overlapping categories with the multiple old categories; a first prediction module, used for obtaining a plurality of corrected sample images from other sample sets other than the newly added sample image set and the historical sample image set, and inputting each of the corrected sample images into the candidate classification model to obtain a first new category prediction value corresponding to the at least one new category and a first old category prediction value corresponding to the multiple old categories of the corresponding corrected sample image, wherein each of the corrected sample images corresponds to one of the new categories or one of the old categories; and the total number of corrected sample images corresponding to the at least one new category is the same as the total number of corrected sample images corresponding to the multiple old categories; A correction module is used to determine a target offset based on the obtained first new category prediction value and the first old category prediction value, and use the target offset to perform output correction on the candidate classification model to obtain a target classification model.
11. An image classification device, characterized in that: include: a second prediction module, configured to input the image to be processed into a target classification model, and obtain initial prediction values of the image to be processed corresponding to each of the plurality of old categories, and a corrected target prediction value corresponding to at least one new category, wherein the target classification model is obtained by using the apparatus according to claim 10; A classification module is used to determine the target category corresponding to the image to be processed from the multiple old categories and the at least one new category based on the initial prediction values corresponding to each of the multiple old categories and the target prediction value corresponding to each of the at least one new category.
12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any method described in claims 1 to 9 are implemented.
13. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 9.
14. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 9.
