Image classification method, device, electronic device and storage medium

Abstract

This application provides an image classification method, apparatus, electronic device, and storage medium. It acquires an image to be detected and a set of candidate feature vectors. The candidate feature vectors in the set are statistical information of features extracted during the training of an image classification model for each task's positive sample image set. The target feature extractor in the image classification model extracts features from the image to be detected, obtaining the target feature vector of the image. Based on the distance between the target feature vector and each candidate feature vector in the set, a classification index characterizing whether the image to be detected is an abnormal image is determined. This application uses statistical information of features extracted during the training of an image classification model for each task's positive sample image set to determine whether an image is defective, achieving incremental learning while reducing memory usage.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an image classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] In incremental learning scenarios, when tasks change over time and data from old tasks cannot be used to train anomaly detection models, the anomaly detection models may overfit to data from new tasks, resulting in catastrophic forgetting.

[0003] Existing incremental learning methods based on playback mostly mitigate catastrophic forgetting by storing input images and labels from old tasks. However, since the images themselves contain a lot of redundant information, they consume a lot of unnecessary memory. Summary of the Invention

[0004] To address the computationally intensive nature of existing playback-based incremental learning methods, this application provides an image classification method, apparatus, electronic device, and storage medium:

[0005] According to a first aspect of this application, an image classification method is provided, comprising:

[0006] Obtain the image to be detected and the candidate feature vector set; the candidate feature vectors in the candidate feature vector set are statistical information of the features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model;

[0007] The target feature extractor in the image classification model is used to extract features from the image to be detected, thereby obtaining the target feature vector of the image to be detected.

[0008] Based on the distance between the target feature vector and each candidate feature vector in the candidate feature vector set, a classification index for the image to be detected is determined; the classification index characterizes whether the image to be detected is an abnormal image.

[0009] According to a second aspect of this application, an image classification apparatus is provided, comprising:

[0010] The acquisition module is used to acquire the image to be detected and the candidate feature vector set. The candidate feature vectors in the candidate feature vector set are statistical information of the features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model.

[0011] The feature extraction module is used to extract features from the image to be detected using the target feature extractor in the image classification model, so as to obtain the target feature vector of the image to be detected.

[0012] The determination module is used to determine the classification index of the image to be detected based on the distance between the target feature vector and each candidate feature vector in the candidate feature vector set; the classification index characterizes whether the image to be detected is an abnormal image.

[0013] On the other hand, the image classification device also includes: a training module;

[0014] This training module includes:

[0015] The acquisition submodule is used to acquire the training image set for each task in the task set; the training image set includes a positive sample image set and a deformed image set, and the deformed images in the deformed image set are obtained by deforming the corresponding positive sample images;

[0016] The first training submodule is used to determine the target task in the task set, and to train the initial image classification model using the positive sample image set of the target task to obtain the candidate image classification model and the first feature vector set of the target task during the training process.

[0017] The first determining submodule is used to determine the statistical information of the first feature vector set as candidate feature vectors in the candidate feature vector set.

[0018] The second training submodule is used to fix the classifier in the candidate image classification model and train the candidate feature extractor in the candidate image classification model using the training image set of each remaining task in the remaining task set to obtain the target feature extractor and the second feature vector set of each remaining task during the training process. The second feature vector set is the feature vector set obtained by extracting features from the positive sample image set of the remaining tasks. The remaining task set consists of multiple tasks in the task set other than the target task.

[0019] The second training submodule is used to determine the statistical information of each second feature vector set, which is used as the candidate feature vector in the candidate feature vector set, and to update the candidate feature vector set.

[0020] The image classification model consists of a classifier and a target feature extractor.

[0021] On the other hand, the first training submodule is used to perform feature extraction processing on the positive sample image set of the target task using the feature extractor to be trained in the initial image classification model to obtain the first feature vector set.

[0022] The classifier to be trained in the initial image classification model is used to classify and predict the positive sample image set of the target task based on the first feature vector set, so as to obtain the first prediction classification result of the positive sample image set of the target task.

[0023] Based on the first predicted classification result, adjust the parameters of the feature extractor and the classifier to be trained until the first preset training termination condition is met, and obtain the candidate feature extractor and classifier.

[0024] On the other hand, the second training submodule is used to perform feature extraction processing on the positive sample image set and deformed image set of each remaining task using the candidate feature extractor, to obtain the second feature vector set of the positive sample image set and the third feature vector set of the deformed image set.

[0025] The classifier is used to perform classification prediction on the sample image set based on the second feature vector set and on the deformed image set based on the third feature vector set, respectively, to obtain the second prediction classification result of the positive sample image set and the third prediction classification result of the deformed image set for the remaining task.

[0026] The parameters of the candidate feature extractor are adjusted based on the second and third predicted classification results until the second preset training termination condition is met, thus obtaining the target feature extractor.

[0027] On the other hand, candidate eigenvectors include the average eigenvector and the covariance;

[0028] The module to be determined includes:

[0029] The third determination submodule is used to determine the feature vector difference between the target feature vector and the average feature vector among the candidate feature vectors for each candidate feature vector in the candidate feature vector set;

[0030] The fourth determination submodule is used to determine the classification index of the image to be detected based on the transpose of the feature vector difference, the inverse of the covariance, and the product of the feature vector difference.

[0031] On the other hand, the fourth determination submodule is used to take the transpose of the feature vector difference, the inverse of the covariance, and the minimum product of the feature vector difference as the classification index of the image to be detected.

[0032] If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the least product, the image to be detected is determined to be an abnormal image.

[0033] According to a third aspect of this application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image classification method of the first aspect of this application.

[0034] According to a fourth aspect of this application, a computer storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the image classification method of the first aspect of this application.

[0035] According to a fifth aspect of this application, a computer program product is provided, comprising at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is loaded and executed by a processor to implement the image classification method of the first aspect of this application.

[0036] The image classification method, apparatus, electronic device, and storage medium provided in this application have the following technical effects:

[0037] This method involves acquiring the image to be detected and a set of candidate feature vectors. The candidate feature vectors in the set are statistical information of features extracted during the training of the image classification model for each task's positive sample image set. The target feature extractor in the image classification model is used to extract features from the image to be detected, resulting in the target feature vector. Based on the distance between the target feature vector and each candidate feature vector in the set, a classification index for the image to be detected is determined. This classification index characterizes whether the image to be detected is an abnormal image. Based on this embodiment, the method determines whether an image is defective by statistically extracting features from the positive sample image set for each task during the training of the image classification model. This achieves incremental learning while reducing memory usage and information loss from using images from older tasks. Furthermore, by using the positive sample image set to train the initial image classification model, no category information needs to be distinguished during training, and no labels from older tasks need to be stored, further reducing memory usage. Attached Figure Description

[0038] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0040] Figure 2 This is a flowchart illustrating a traditional defect detection method.

[0041] Figure 3This is a flowchart illustrating a defect detection method in an incremental learning scenario provided in an embodiment of this application;

[0042] Figure 4 This is a flowchart illustrating an image classification method provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of an image classification method provided in an embodiment of this application;

[0044] Figure 6 This is a flowchart illustrating a training method for an image classification model provided in an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of a training method for an image classification model provided in an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the hardware structure of an electronic device for implementing the image classification method provided in this application embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely one embodiment of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0049] The term "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this application. In the description of the embodiments of this application, it should be understood that the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," and "third," etc., may explicitly or implicitly include one or more of that feature. Furthermore, the terms "first," "second," and "third," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprising," "having," and "being," and any variations thereof, are intended to cover non-exclusive inclusion.

[0050] It is understood that in the specific embodiments of this application, image data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0051] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, big data, etc.

[0052] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0053] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0054] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0055] Anomaly Detection: The main task is to determine whether an image is abnormal. During the training process, anomaly detection is trained using only normal images, while during prediction, it is necessary to determine whether an image is abnormal from a test set that includes both normal and abnormal images.

[0056] Catastrophic forgetting: In scenarios with non-independent and identically distributed data, when a new task arrives, if the model can only acquire data from the new task, the model tends to overfit the data from the new task, resulting in a sharp drop in the model's performance on the old task, i.e., catastrophic forgetting occurs.

[0057] Incremental Learning, also known as Continual Learning or Life Long Learning, aims to address catastrophic forgetting and achieve a balance between model plasticity and stability.

[0058] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. The application environment may include a terminal 10 and a server 20. The terminal 10 and the server 20 can be directly or indirectly connected via wired or wireless communication.

[0059] In some possible implementations, terminal 10 can send the image to be detected to server 20. The server can provide image classification services. By using the target feature extractor in the image classification model to perform feature extraction processing on the image to be detected, the target feature vector of the image to be detected is obtained. This vector is then used to calculate the statistical information of the features obtained by feature extraction for the positive sample image set of each task in the task set during the training process of the image classification model, thereby determining the classification index that characterizes whether the image to be detected is an abnormal image.

[0060] Terminal 10 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal 10 may have client software, such as an application (App), installed to provide human-computer interaction functions. This application can be a standalone application or a subroutine within an application. Users of Terminal 10 can log in to the application using pre-registered user account information, which may include a username and password.

[0061] Server 20 can be a standalone physical server, a service cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc.

[0062] In some possible implementations, both terminal 10 and server 20 can be node devices in the blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thus enabling information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block. This prevents data tampering in the blockchain and guarantees the security and reliability of the data in the blockchain.

[0063] Anomaly detection can include defect detection, and the task characteristics of defect detection itself have received widespread attention in recent years. The task setting for defect detection is quite unique: during training, only normal images are used, while during prediction, it needs to determine whether an image is defective from a test set containing both normal and defective images. Due to the specific nature of the task, the model is required to learn the patterns of normal images from them alone, thereby identifying all defective images. This is similar to outlier detection, which identifies exceptional instances that do not conform to the patterns.

[0064] Existing defect detection scenarios, such as industrial defect detection and medical image analysis, require detection models to have incremental learning capabilities. This means that when a new task arrives, the model should be able to perform image judgments for the new task while maintaining performance on previous tasks. However, all current defect detection methods—whether traditional methods like Support Vector Data Description (SVDD) and its variant Passive Nondestructive Assay (PANDA), or reconstruction-based defect learning methods like DRAEM, data-augmented image anomaly detection localization methods like CutPaste, normalizing flow-based anomaly detection methods, or knowledge-distillation-based reverse distillation—are trained on independent and identically distributed data, focusing on performance on new tasks. When multiple tasks are input sequentially, the accuracy for previous tasks cannot be guaranteed.

[0065] Figure 2This is a flowchart illustrating a traditional defect detection method. Traditional defect detection models can either store models from old tasks and retrain a new model for the new task's data, or retrain models from both tasks. This is not only computationally intensive but also degrades system performance. Furthermore, memory constraints or data access limitations make this method difficult to apply in real-world scenarios. Existing playback-based incremental learning methods mostly mitigate catastrophic forgetting by storing input images and labels from old tasks. However, images themselves contain a large amount of redundant information, consuming unnecessary memory. Storing input images from old tasks is equivalent to applying data from old tasks, violating the incremental learning principle of "using only data from the new task." To address this, this application proposes a method that utilizes the experience learned from playing back old tasks—specifically, the Distribution of Normal Embeddings (DNE) of normal images—to achieve defect detection in incremental learning scenarios. Figure 3 This is a flowchart illustrating a defect detection method in an incremental learning scenario provided in this application embodiment. This defect detection method can be applied to industrial defect detection scenarios. For example, on an actual production line, the deployed defect detection model initially only performs defect detection on a single product (e.g., a screw). When a new product (e.g., a nut) is added to the production line, based on… Figure 3 The method shown can maintain the performance of the defect detection model on both new and old tasks in this non-stationary data distribution. That is, after adding a new product (nut), it can not only perform defect detection on the nut well, but also maintain the detection effect on the screw.

[0066] The following describes a specific embodiment of an image classification method according to this application. Figure 4 This is a flowchart illustrating an image classification method provided in an embodiment of this application. Figure 5 This is a schematic diagram of an image classification method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many execution orders and does not represent the only execution order. In actual execution, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0067] Specifically, such as Figure 4 As shown, the image classification method may include:

[0068] S401: Obtain the image to be detected and the candidate feature vector set; the candidate feature vectors in the candidate feature vector set are statistical information of the features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model.

[0069] Figure 6 This is a flowchart illustrating a training method for an image classification model provided in an embodiment of this application. In practical applications, methods such as... Figure 6 The method shown trains an initial image classification model to obtain an image classification model. The initial image classification model may include a feature extractor to be trained and a classifier to be trained. Specific training steps for the image classification model may include:

[0070] S601: Obtain the training image set for each task in the task set; the training image set includes a positive sample image set and a deformed image set, and the deformed images in the deformed image set are obtained by deforming the corresponding positive sample images.

[0071] In this embodiment of the application, during the training phase, an initial image classification model f = f can be provided. e °h and a mini-batch of training images X∈R for each task B×C×H×W Among them, f e Let represent the pre-trained backbone network of the initial image classification model, i.e., the feature extractor to be trained, and h represent the classification head of the initial image classification model, i.e., the classifier to be trained. Furthermore, the pre-trained backbone network follows R... B×C×H×W →R B×D That is, mapping the input image to features Z∈R B×D Where B can represent the number of training image frames in a single training process, C can represent the number of channels in each training image frame, H can represent the height of each training image frame, W can represent the width of each training image frame, and D can represent the feature dimension. Optionally, the pre-trained backbone network can be a neural network for feature extraction, such as an Alex Network (AlaxNet), a Very Deep Convolutional Network (VGGNet), or a Residual Network (ResNet). The classification head can be a fully connected layer, a 1×1 convolutional layer, or multiple stacked fully connected layers.

[0072] This application trains the initial image classification model using a set of positive sample images, so that no category information is needed to distinguish during the training process. This means that there is no need to save the labels of the old tasks, which can reduce memory usage.

[0073] S603: Determine the target task in the task set, and train the initial image classification model using the positive sample image set of the target task to obtain the candidate image classification model and the first feature vector set of the target task during the training process.

[0074] In this embodiment, the feature extractor to be trained in the initial image classification model can be used to extract features from the positive sample image set of the target task, resulting in a first feature vector set. Then, the classifier to be trained in the initial image classification model can be used to perform classification prediction on the positive sample image set of the target task based on the first feature vector set, obtaining a first predicted classification result for the positive sample image set of the target task. The parameters of the feature extractor and the classifier to be trained can then be adjusted based on the first predicted classification result until a first preset training termination condition is met, resulting in a candidate feature extractor and a classifier.

[0075] In some possible implementations, the target task may be a task randomly selected from a task set. For example, the target task may be the first of several tasks that are sequentially input during the training of the initial image classification model.

[0076] In practical applications, for the positive sample image set n of the target task τ τ To obtain all feature vectors of the positive sample image set for the target task, i.e., the first feature vector set, a cache Z can be constructed. epoch This is used to temporarily store the feature vector Z during a single training iteration until a round of training for the target task is completed. Specifically, Z... epoch It can be defined as:

[0077] Z epoch ={Z1,Z2,...,Z M}

[0078]

[0079] Among them, Z m (m=1,2...M) can represent the feature vector subset of the positive sample image subset of the target task in a single training process, B can represent the number of frames of positive sample images in the positive sample image subset in a single training process, M can represent the number of training iterations, and M is the integer obtained by rounding up the ratio of the total number of frames of the positive sample image subset of the target task τ to the number of frames of the positive sample image subset in a single training process.

[0080] For example, the sample image set n for the target task τ τ =80, and in a single training iteration, the initial image classification model is trained using B = 8 frames of positive sample images. For the first training iteration, the feature vector z of each positive sample image b can be extracted using the feature extractor to be trained in the initial image classification model. b The feature vector subset Z1 = {z1, z2, ..., z8} obtained during the initial training process is stored in the cache Z. epochThis yields the feature vector set {Z1} for the target task during the initial training process. The classifier to be trained in the initial image classification model can be used to classify and predict the positive sample image subset during the initial training process based on the feature vector subset Z1, obtaining the probability that each positive sample image in the positive sample image subset belongs to a normal image. Then, the parameters of the feature extractor and classifier to be trained can be adjusted based on the probability that each positive sample image belongs to a normal image until the probability that each sample image belongs to a normal image is greater than a threshold, resulting in the feature extractor and classifier after the initial training. Then, a second training can be performed. For the second training, the feature extractor after the initial training can be used to extract the feature vector of each positive sample image, obtaining the feature vector subset Z2 = {z9, z...} during the second training process. 10 ,...,z 16} and store it in cache Z. epoch This yields the feature vector set {Z1, Z2} for the target task during the second training process. That is, during training, Z... epoch The process is continuously updated. The feature vector subset obtained in the current training iteration is added to the feature vector set obtained in the previous training iteration, resulting in the feature vector set obtained in the current training iteration. Then, the classifier trained in the first iteration can be used to classify and predict the positive sample image subset from the second training iteration based on the feature vector subset Z2. This yields the probability that each positive sample image in the positive sample image subset belongs to a normal image. The parameters of the feature extractor and classifier trained in the first iteration can be adjusted based on the probability that each positive sample image belongs to a normal image until the probability that each sample image belongs to a normal image is greater than a threshold, resulting in the feature extractor and classifier trained in the second iteration. This process is repeated to obtain the candidate image classification model and the first feature vector set {Z1, Z2, ..., Z...} obtained during the first round of training for the target task. M}

[0081] S605: Determine the statistical information of the first feature vector set as candidate feature vectors in the candidate feature vector set.

[0082] Unlike traditional playback-based incremental learning methods that save memory by sampling images from the training set, this embodiment directly calculates the statistical information of the first feature vector set to represent the data features of the target task. Assuming the features follow a Gaussian distribution, the average eigenvector and covariance of the first feature vector set can be calculated. For the feature vectors z1, z2, ..., z in the first feature vector set... n ∈R D Covariance can be defined as:

[0083]

[0084] in, It can represent the average eigenvector of the first eigenvector set. It can represent the covariance of the first set of eigenvectors.

[0085] After training on the positive sample image set for the target task is complete, the cache Z can be released. epoch The statistics of the first feature vector set are stored in memory.

[0086] As is well known, when the total number of feature vectors *n* is sufficiently large compared to the feature dimension *D*, the covariance can be well approximated by the maximum likelihood estimator; this covariance is called the empirical covariance. However, in real-world scenarios, the total number of feature vectors *n* collected is generally small, and the dimension *D* of the output features obtained from deep feature representation is large. Therefore, it is difficult to guarantee that *n* is much larger than *D*, which makes it difficult to invert the empirical covariance. To compute the precision matrix, shrinkage covariance can be introduced as an estimate of the covariance matrix. The definition of shrinkage covariance is as follows:

[0087]

[0088] Among them, I d It can represent the identity matrix, α can represent the contraction coefficient, when α is 0, the contraction covariance is equal to the empirical covariance, and tr can represent a matrix.

[0089] S607: The classifier in the fixed candidate image classification model is trained using the training image set of each remaining task in the remaining task set to obtain the target feature extractor and the second feature vector set of each remaining task during the training process.

[0090] In this embodiment of the application, the second feature vector set may be a feature vector set obtained by extracting features from the positive sample image set of the remaining tasks, and the remaining task set may be multiple tasks in the task set other than the target task.

[0091] In this embodiment, a candidate feature extractor can be used to extract features from the positive sample image set and the deformed image set for each remaining task, resulting in a second feature vector set for the positive sample image set and a third feature vector set for the deformed image set. Then, a classifier can be used to perform classification prediction based on the second feature vector set and the third feature vector set, respectively, to obtain a second predicted classification result for the positive sample image set and a third predicted classification result for the deformed image set for the remaining task. The parameters of the candidate feature extractor can then be adjusted based on the second and third predicted classification results until a second preset training termination condition is met, resulting in the target feature extractor.

[0092] In some possible implementations, for each remaining task, a candidate feature extractor in the candidate image classification model can be used to extract feature vectors from the positive sample image set and the deformed image set for that remaining task. Then, the classifier in the candidate image classification model can be used to classify and predict the positive sample image set and the deformed image set for that remaining task based on the feature vectors, obtaining the probability that each frame of the positive sample image set belongs to a normal image and the probability that each frame of the deformed image set belongs to an abnormal image. A loss function can then be constructed to adjust the parameters of the candidate feature extractor until the loss value of the loss function is less than a preset loss threshold, or until the required number of iterations is met, thus obtaining the target feature extractor. Specifically, the loss function can be constructed as follows:

[0093]

[0094] Here, CE can represent cross-entropy, which represents the number of learnable adoptions of the classification head in the target task (which will be frozen in the remaining tasks), and x can represent positive sample images. It can represent deformed images.

[0095] In some possible implementations, when the task changes, the model's weights tend to overfit the data of the new task, overwriting the weights learned from the old task. This results in the feature distribution stored in memory becoming inaccurate. Therefore, after training on the positive sample image set of the old task, the classifier head can be fixed. That is, during subsequent adjustments to the parameters of the candidate feature extractor using the positive sample image set of the new task, the classifier parameters are not adjusted. This makes the features output by the feature extractor in the model more similar to those of the old task, thereby suppressing changes in the feature distribution.

[0096] S609: Determine the statistical information of each second feature vector set as a candidate feature vector in the candidate feature vector set, and update the candidate feature vector set; wherein, the classifier and the target feature extractor constitute the image classification model.

[0097] In some possible implementations, the statistical information of the second feature vector set for each remaining task can be determined using the same method as determining the statistical information of the first feature vector set, which will not be elaborated here.

[0098] To facilitate understanding, a specific example will be given below to illustrate the training method of the above image classification model. Figure 7 This is a schematic diagram of a training method for an image classification model provided in an embodiment of this application. The feature extractor 703 to be trained can be used to extract features from a normal image 701 containing hazelnuts, obtaining a first feature vector set 705. Then, the statistical information of the first feature vector set 705 can be used as candidate feature vectors in a candidate feature vector set 707. Simultaneously, the normal image 701 containing hazelnuts can be classified and predicted based on the first feature vector set 705, obtaining a predicted classification result. Based on the predicted classification result, the parameters of the feature extractor 703 and the classifier to be trained are adjusted to obtain a candidate feature extractor 709 and a classifier 711. Next, the candidate feature extractor 709 can be used to extract features from the normal image 701 containing hazelnuts to obtain a second feature vector set, and the candidate feature extractor 709 can be used to extract features from a deformed image 713 containing hazelnuts to obtain a third feature vector set 715. Then, the statistical information of the second feature vector set can be used as candidate feature vectors in a candidate feature vector set 707. Simultaneously, a predicted classification result can be obtained by classifying and predicting the normal image 701 containing hazelnuts based on the second feature vector set, and a predicted classification result can be obtained by classifying and predicting the deformed image 713 containing hazelnuts based on the third feature vector set. The parameters of the candidate feature extractor 709 can be adjusted based on the predicted classification result to obtain the target feature extractor.

[0099] like Figure 7 As shown, the statistical information of the features of each task in the task set, such as the number of frames of positive sample images in the positive sample image set for each task, the average feature vector, and the covariance, are stored in memory. Because the data features of each task are not the same, a new memory space needs to be allocated for each task to store its statistical information. That is, for T tasks, T distributions need to be stored. Therefore, after obtaining the statistical information of each task in the task set, the feature vector set of each task can be released, and the statistical information of each task, i.e., the distribution of each task, can be stored directly.

[0100] Continue to refer to Figure 4 After completing step S401: obtaining the image to be detected and the candidate feature vector set, step S403 can be executed.

[0101] S403: Use the target feature extractor in the image classification model to perform feature extraction processing on the image to be detected, and obtain the target feature vector of the image to be detected.

[0102] In this embodiment, the target feature extractor in the trained image classification model can be used to perform feature extraction processing on the image to be detected, so as to obtain the target feature vector z of the image to be detected.

[0103] S405: Based on the distance between the target feature vector and each candidate feature vector in the candidate feature vector set, determine the classification index of the image to be detected; the classification index characterizes whether the image to be detected is an abnormal image.

[0104] In this embodiment, the candidate feature vector may include the average feature vector and the covariance. After obtaining the target feature vector, for each candidate feature vector in the candidate feature vector set, the feature vector difference between the target feature vector and the average feature vector in the candidate feature vector set can be determined. Then, the classification index of the image to be detected can be determined based on the transpose of the feature vector difference, the inverse of the covariance, and the product of the feature vector difference.

[0105] In some possible implementations, the transpose of the feature vector difference, the reciprocal of the covariance, and the least product of the feature vector differences can be used as the classification index of the image to be detected. If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the least product, the image to be detected can be determined to be an abnormal image.

[0106] In some possible implementations, the average product of the transpose of the feature vector difference, the inverse of the covariance, and the product of the feature vector differences can be used as the classification index of the image to be detected. If the classification index of the image to be detected is greater than a preset average threshold, the image to be detected can be determined to be an abnormal image.

[0107] In this embodiment, the candidate feature vector may also include the average feature vector and the contracted covariance. After obtaining the target feature vector, for each candidate feature vector in the candidate feature vector set, the feature vector difference between the target feature vector and the average feature vector in the candidate feature vector set can be determined. Then, the classification index of the image to be detected can be determined based on the transpose of the feature vector difference, the inverse of the contracted covariance, and the product of the feature vector difference.

[0108] In some possible implementations, the transpose of the feature vector difference, the inverse of the contracted covariance, and the minimum product of the feature vector difference can be used as the classification index of the image to be detected. If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the minimum product, the image to be detected can be determined to be an abnormal image.

[0109] In some possible implementations, such as Figure 5As shown, based on the average feature vector and covariance of the old tasks in memory, the Gaussian distribution corresponding to the features of each old task can be reconstructed. Then, n samples can be sequentially taken from the distribution of the τ-th task. τ The system uses 10 features and constructs a new Gaussian distribution based on these features. In other words, it constructs a Gaussian distribution based on features from all tasks. Therefore, this Gaussian distribution can effectively predict whether an image to be detected is a defective image, without requiring any task identities, i.e., it does not need to specify which task the image to be detected originates from. Finally, Mahalanobis distance can be used as a classification metric to determine whether an image to be detected is a defective image. Mahalanobis distance can be defined as:

[0110]

[0111] Where z can represent the target feature vector of the image to be detected, and M(z) can represent the classification index.

[0112] In addition, Euclidean distance or angular distance can be used to measure whether the image to be detected is a defective image.

[0113] The image classification method provided in this application determines whether an image is defective by extracting features from the positive sample image set for each task in the task set during the training process of the image classification model. This achieves incremental learning while reducing memory usage and information loss caused by using images from older tasks. Furthermore, by using the positive sample image set to train the initial image classification model, no category information needs to be distinguished during training, and no labels from older tasks need to be stored, further reducing memory usage.

[0114] This application also provides an image classification device in its embodiments. Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application, as shown below. Figure 8 As shown, the image classification device may include:

[0115] The acquisition module 801 is used to acquire the image to be detected and the candidate feature vector set; the candidate feature vectors in the candidate feature vector set are statistical information of the features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model.

[0116] The feature extraction module 803 is used to perform feature extraction processing on the image to be detected using the target feature extractor in the image classification model to obtain the target feature vector of the image to be detected.

[0117] The determination module 805 is used to determine the classification index of the image to be detected based on the distance between the target feature vector and each candidate feature vector in the candidate feature vector set; the classification index characterizes whether the image to be detected is an abnormal image.

[0118] In some possible implementations, the image classification device further includes: a training module;

[0119] This training module includes:

[0120] The acquisition submodule is used to acquire the training image set for each task in the task set; the training image set includes a positive sample image set and a deformed image set, and the deformed images in the deformed image set are obtained by deforming the corresponding positive sample images;

[0121] The first training submodule is used to determine the target task in the task set, and to train the initial image classification model using the positive sample image set of the target task to obtain the candidate image classification model and the first feature vector set of the target task during the training process.

[0122] The first determining submodule is used to determine the statistical information of the first feature vector set as candidate feature vectors in the candidate feature vector set.

[0123] The second training submodule is used to fix the classifier in the candidate image classification model and train the candidate feature extractor in the candidate image classification model using the training image set of each remaining task in the remaining task set to obtain the target feature extractor and the second feature vector set of each remaining task during the training process. The second feature vector set is the feature vector set obtained by extracting features from the positive sample image set of the remaining tasks. The remaining task set consists of multiple tasks in the task set other than the target task.

[0124] The second training submodule is used to determine the statistical information of each second feature vector set, which is used as the candidate feature vector in the candidate feature vector set, and to update the candidate feature vector set.

[0125] The image classification model consists of a classifier and a target feature extractor.

[0126] In some possible implementations, the first training submodule is used to perform feature extraction processing on the positive sample image set of the target task using the feature extractor to be trained in the initial image classification model to obtain a first feature vector set;

[0127] The classifier to be trained in the initial image classification model is used to classify and predict the positive sample image set of the target task based on the first feature vector set, so as to obtain the first prediction classification result of the positive sample image set of the target task.

[0128] Based on the first predicted classification result, adjust the parameters of the feature extractor and the classifier to be trained until the first preset training termination condition is met, and obtain the candidate feature extractor and classifier.

[0129] In some possible implementations, the second training submodule is used to perform feature extraction processing on the positive sample image set and deformed image set of each remaining task using a candidate feature extractor to obtain a second feature vector set of the positive sample image set and a third feature vector set of the deformed image set.

[0130] The classifier is used to perform classification prediction on the sample image set based on the second feature vector set and on the deformed image set based on the third feature vector set, respectively, to obtain the second prediction classification result of the positive sample image set and the third prediction classification result of the deformed image set for the remaining task.

[0131] The parameters of the candidate feature extractor are adjusted based on the second and third predicted classification results until the second preset training termination condition is met, thus obtaining the target feature extractor.

[0132] In some possible implementations, the candidate feature vector includes the average feature vector and the covariance;

[0133] The module to be determined includes:

[0134] The third determination submodule is used to determine the feature vector difference between the target feature vector and the average feature vector among the candidate feature vectors for each candidate feature vector in the candidate feature vector set;

[0135] The fourth determination submodule is used to determine the classification index of the image to be detected based on the transpose of the feature vector difference, the inverse of the covariance, and the product of the feature vector difference.

[0136] In some possible implementations, the fourth determining submodule is used to use the transpose of the feature vector difference, the inverse of the covariance, and the least product of the feature vector difference as the classification index of the image to be detected.

[0137] If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the least product, the image to be detected is determined to be an abnormal image.

[0138] The apparatus and method embodiments in this application are based on the same application concept.

[0139] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the image classification method provided in the above method embodiments.

[0140] Figure 9 This is a schematic diagram of the hardware structure of an electronic device for implementing the image classification method provided in this application embodiment. The electronic device can participate in or include the image classification apparatus provided in this application embodiment. Figure 9 As shown, the electronic device may include one or more processors 901 (shown as 901a and 901b in the figure) 901 (processor 901 may include, but is not limited to, a microprocessor 901 MCU or a programmable logic device FPGA, etc.), a memory 903 for storing data, and a transmission device 905 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and / or a power supply. Those skilled in the art will understand that... Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0141] It should be noted that the aforementioned one or more processors 901 and / or other data processing circuits are generally referred to as "data processing circuits" in this application. The data processing circuit can be embodied, in whole or in part, in software, hardware, firmware, or any other combination. Furthermore, the data processing circuit can be a single, independent processing module, or it can be integrated, in whole or in part, into any other element within an electronic device (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor 901 control (e.g., selection of a variable resistor termination path connected to an interface).

[0142] The memory 903 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image classification method in this embodiment. The processor 901 implements the image classification method described above by running the software programs and modules stored in the memory 903 and executing various functional applications and data processing. The memory 903 may include high-speed random access memory, and may also include non-volatile random access memory 903, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory 903. In some possible embodiments, the memory 903 may further include remotely configured memories 903 relative to the processing, which can be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0143] The transmission device 905 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device. In one example, the transmission device 905 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 905 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0144] The display can be, for example, a touchscreen liquid crystal display (LED), which allows users to interact with the user interface of an electronic device (or mobile device).

[0145] This application provides a computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an image classification method in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the image classification method provided in the above method embodiment.

[0146] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0147] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, while this specification describes specific embodiments, other embodiments are also within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in the order shown in different embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific order or sequence of connections to achieve the desired results; in some implementations, parallel processing of multiple tasks is possible or may be advantageous.

[0148] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus and electronic devices are described simply because they are based on similar method embodiments; relevant parts can be referred to the descriptions of the method embodiments.

[0149] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image classification method, characterized in that, include: Obtain the image to be detected and the candidate feature vector set; The candidate feature vectors in the candidate feature vector set are statistical information of features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model; the candidate feature vectors include the average feature vector and the covariance. The target feature vector of the image to be detected is obtained by using the target feature extractor in the image classification model to perform feature extraction processing on the image to be detected. For each candidate feature vector in the candidate feature vector set, determine the feature vector difference between the target feature vector and the average feature vector among the candidate feature vectors; The classification index of the image to be detected is determined based on the transpose of the feature vector difference, the reciprocal of the covariance, and the product of the feature vector difference; the classification index characterizes whether the image to be detected is an abnormal image.

2. The method according to claim 1, characterized in that, The training steps of the image classification model include: Obtain the training image set for each task in the task set; the training image set includes a positive sample image set and a deformed image set, and the deformed images in the deformed image set are obtained by deforming the corresponding positive sample images; The target task in the task set is determined, and the initial image classification model is trained using the positive sample image set of the target task to obtain the candidate image classification model and the first feature vector set of the target task during the training process; Statistical information of the first feature vector set is determined and used as candidate feature vectors in the candidate feature vector set; The classifier in the candidate image classification model is fixed, and the candidate feature extractor in the candidate image classification model is trained using the training image set of each remaining task in the remaining task set to obtain the target feature extractor and the second feature vector set of each remaining task during the training process; the second feature vector set is the feature vector set obtained by feature extraction of the positive sample image set of the remaining task, and the remaining task set is multiple tasks in the task set other than the target task; Determine the statistical information of each second feature vector set as a candidate feature vector in the candidate feature vector set, and update the candidate feature vector set; The classifier and the target feature extractor constitute the image classification model.

3. The method according to claim 2, characterized in that, The step of training the initial image classification model using the positive sample image set of the target task includes: The first feature vector set is obtained by using the feature extractor to be trained in the initial image classification model to perform feature extraction on the positive sample image set of the target task. The classifier to be trained in the initial image classification model is used to classify and predict the positive sample image set of the target task based on the first feature vector set, so as to obtain the first predicted classification result of the positive sample image set of the target task. Based on the first predicted classification result, adjust the parameters of the feature extractor and the classifier to be trained until the first preset training termination condition is met, and obtain the candidate feature extractor and the classifier.

4. The method according to claim 2, characterized in that, The step of training the candidate feature extractor in the candidate image classification model using the training image set of each remaining task in the remaining task set includes: The candidate feature extractor is used to perform feature extraction processing on the positive sample image set and deformed image set of each of the remaining tasks to obtain the second feature vector set of the positive sample image set and the third feature vector set of the deformed image set; The classifier is used to perform classification prediction on the sample image set based on the second feature vector set and on the deformed image set based on the third feature vector set, respectively, to obtain the second predicted classification result of the positive sample image set and the third predicted classification result of the deformed image set for the remaining task; The parameters of the candidate feature extractor are adjusted based on the second and third prediction classification results until the second preset training termination condition is met, thereby obtaining the target feature extractor.

5. The method according to claim 1, characterized in that, The step of determining the classification index of the image to be detected based on the product of the transpose of the feature vector difference, the reciprocal of the covariance, and the feature vector difference includes: The transpose of the feature vector difference, the reciprocal of the covariance, and the minimum product of the feature vector difference are used as the classification index of the image to be detected. If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the least product, the image to be detected is determined to be an abnormal image.

6. An image classification device, characterized in that, include: The acquisition module is used to acquire the image to be detected and the candidate feature vector set; The candidate feature vectors in the candidate feature vector set are statistical information of features obtained by feature extraction for the positive sample image set of each task in the task set during the training of the image classification model; the candidate feature vectors include the average feature vector and the covariance. The feature extraction module is used to perform feature extraction processing on the image to be detected using the target feature extractor in the image classification model to obtain the target feature vector of the image to be detected. The determination module is used to determine the classification index of the image to be detected based on the distance between the target feature vector and each candidate feature vector in the candidate feature vector set; the classification index characterizes whether the image to be detected is an abnormal image; The determining module includes: The third determining submodule is used to determine the feature vector difference between the target feature vector and the average feature vector among the candidate feature vectors for each candidate feature vector in the candidate feature vector set; The fourth determination submodule is used to determine the classification index of the image to be detected based on the transpose of the feature vector difference, the reciprocal of the covariance, and the product of the feature vector difference.

7. The apparatus according to claim 6, characterized in that, The image classification device further includes: a training module; The training module includes: The acquisition submodule is used to acquire the training image set for each task in the task set; the training image set includes a positive sample image set and a deformed image set, and the deformed images in the deformed image set are obtained by deforming the corresponding positive sample images; The first training submodule is used to determine the target task in the task set, train the initial image classification model using the positive sample image set of the target task, and obtain the candidate image classification model and the first feature vector set of the target task during the training process. The first determining submodule is used to determine the statistical information of the first feature vector set as candidate feature vectors in the candidate feature vector set. The second training submodule is used to fix the classifier in the candidate image classification model, and train the candidate feature extractor in the candidate image classification model using the training image set of each remaining task in the remaining task set, to obtain the target feature extractor and the second feature vector set of each remaining task during the training process; the second feature vector set is the feature vector set obtained by feature extraction of the positive sample image set of the remaining task, and the remaining task set is multiple tasks in the task set other than the target task; The second training submodule is used to determine the statistical information of each of the second feature vector sets as candidate feature vectors in the candidate feature vector set, and to update the candidate feature vector set. The classifier and the target feature extractor constitute the image classification model.

8. The apparatus according to claim 7, characterized in that, The first training submodule is used to perform feature extraction processing on the positive sample image set of the target task using the feature extractor to be trained in the initial image classification model to obtain the first feature vector set; The classifier to be trained in the initial image classification model is used to classify and predict the positive sample image set of the target task based on the first feature vector set, so as to obtain the first predicted classification result of the positive sample image set of the target task. Based on the first predicted classification result, adjust the parameters of the feature extractor and the classifier to be trained until the first preset training termination condition is met, and obtain the candidate feature extractor and the classifier.

9. The apparatus according to claim 7, characterized in that, The second training submodule is used to perform feature extraction processing on the positive sample image set and deformed image set of each remaining task using the candidate feature extractor, so as to obtain the second feature vector set of the positive sample image set and the third feature vector set of the deformed image set; The classifier is used to perform classification prediction on the sample image set based on the second feature vector set and on the deformed image set based on the third feature vector set, respectively, to obtain the second predicted classification result of the positive sample image set and the third predicted classification result of the deformed image set for the remaining task; The parameters of the candidate feature extractor are adjusted based on the second and third prediction classification results until the second preset training termination condition is met, thereby obtaining the target feature extractor.

10. The apparatus according to claim 6, characterized in that, The fourth determining submodule is used to take the transpose of the feature vector difference, the reciprocal of the covariance, and the minimum product of the feature vector difference as the classification index of the image to be detected; If the classification index of the image to be detected is greater than the preset reference index of the candidate feature vector corresponding to the least product, the image to be detected is determined to be an abnormal image.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image classification method as described in any one of claims 1-5.

12. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the image classification method as described in any one of claims 1-5.

13. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the image classification method as described in any one of claims 1-5.

Images