Image classification methods and apparatus, electronic devices and storage media

Abstract

This application provides an image classification method, apparatus, electronic device, and storage medium, belonging to the field of artificial intelligence technology. The method includes acquiring an image to be classified; inputting the image to be classified into a target image classification model to obtain an image classification result; wherein the target image classification model is obtained through the following steps: determining the prediction result of the local image training data based on local image training data and the local image classification model; determining the uncertainty of the local image training data based on the prediction result and the actual label of the local image training data; inputting the local image training data into the local image classification model for training based on the uncertainty, wherein local image training data with lower uncertainty is input into the local image classification model first; and sending the parameters of the trained local image classification model to a central server for weighted summation to obtain a trained target image classification model. This application can improve the accuracy of image classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image classification method and apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, various types of applications have emerged. Among them, image classification is a branch of artificial intelligence applications, which can perform corresponding tasks in multiple scenarios, such as face recognition, traffic scene recognition, image retrieval, automatic album categorization, and medical image recognition.

[0003] In related technologies, image classification is usually implemented using neural network models. However, due to factors such as the complexity of the image acquisition environment and the varying quality of acquisition equipment, some image data often contains noise interference, making it difficult for the model to effectively extract image features and make accurate classification judgments, resulting in low accuracy of image classification results. Summary of the Invention

[0004] The main objective of this application is to provide an image classification method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of image classification.

[0005] To achieve the above objectives, a first aspect of this application proposes an image classification method, the method comprising:

[0006] Obtain the image to be classified;

[0007] The image to be classified is input into the target image classification model, and the target image classification model is used to extract features from the image to be classified to obtain image feature data;

[0008] Based on the image feature data, the target image classification model is used to perform classification prediction on the image to be classified, and the image classification result is obtained.

[0009] The target image classification model is obtained through the following steps:

[0010] Acquire multiple local image training data and local image classification models;

[0011] The prediction result of the local image training data is determined based on the local image training data and the local image classification model, wherein the prediction result includes the prediction probability of multiple categories;

[0012] The uncertainty of the local image training data is determined based on the prediction results and the actual labels of the local image training data.

[0013] The local image training data is input into the local image classification model for training based on the uncertainty, wherein the local image training data with lower uncertainty is input into the local image classification model for training first;

[0014] The local model parameters of the trained local image classification model are sent to the central server for weighted summation of multiple local model parameters to obtain the trained target image classification model.

[0015] In some embodiments, the step of inputting the local image training data into the local image classification model for training based on the uncertainty includes:

[0016] The local image training data with uncertainty less than or equal to the classification threshold are divided into a low uncertainty dataset;

[0017] The local image training data with uncertainty greater than the classification threshold are classified into a high uncertainty dataset;

[0018] After inputting the local image training data from the low uncertainty dataset into the local image classification model for training, the local image training data from the high uncertainty dataset is then input into the local image classification model for training.

[0019] In some embodiments, the step of inputting the local image training data into the local image classification model for training based on the uncertainty includes:

[0020] The local image training data are arranged in ascending order of uncertainty.

[0021] The arranged local image training data is sequentially input into the local image classification model for training until the loss value of the local image classification model is less than the preset loss value.

[0022] In some embodiments, the step of inputting local image training data from the low-uncertainty dataset into the local image classification model for training, and then inputting local image training data from the high-uncertainty dataset into the local image classification model for training, includes:

[0023] All local image training data in the low uncertainty dataset are input into the local image classification model for training;

[0024] The local image training data in the high uncertainty dataset are input one by one into the local image classification model for training until the loss value of the local image classification model is less than the preset loss value.

[0025] In some embodiments, the loss value of the local image classification model is calculated using a cross-entropy loss function, which is expressed as follows:

[0026]

[0027] Where L is the loss value, N is the number of local image training data, and y ic Let y be a sign function, where y is the true class of the local image training data i equal to c. ic =1, otherwise y ic =0, P ic M represents the predicted probability that local image training data i belongs to category c, where M represents the total number of categories.

[0028] In some embodiments, determining the prediction result of the local image training data based on the local image training data and the local image classification model includes:

[0029] The local image training data is divided into pre-training data and test data;

[0030] The local image training data, which is used as pre-training data, is input into the local image classification model for training to obtain the pre-trained model;

[0031] The local image training data, which serves as test data, is input into the pre-trained model to obtain the prediction results of the local image training data.

[0032] In some embodiments, the uncertainty is calculated using the following formula:

[0033]

[0034] Wherein, P(y=t|x) represents the predicted probability of the actual label y of the local image training data x being category t, obtained by the pre-trained model predicting the local image training data x, and T represents the total number of predicted local image training data.

[0035] To achieve the above objectives, a second aspect of this application provides an image classification apparatus, the apparatus comprising:

[0036] The first module is used to acquire the image to be classified.

[0037] The second module is used to input the image to be classified into the target image classification model, extract features from the image to be classified through the target image classification model to obtain image feature data; and perform classification prediction on the image to be classified through the target image classification model based on the image feature data to obtain the image classification result.

[0038] The target image classification model is obtained through the following steps:

[0039] Acquire multiple local image training data and local image classification models;

[0040] The prediction result of the local image training data is determined based on the local image training data and the local image classification model, wherein the prediction result includes the prediction probability of multiple categories;

[0041] The uncertainty of the local image training data is determined based on the prediction results and the actual labels of the local image training data.

[0042] The local image training data is input into the local image classification model for training based on the uncertainty, wherein the local image training data with lower uncertainty is input into the local image classification model for training first;

[0043] The local model parameters of the trained local image classification model are sent to the central server for weighted summation of multiple local model parameters to obtain the trained target image classification model.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image classification method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image classification method described in the first aspect.

[0046] The image classification method, apparatus, electronic device, and storage medium proposed in this application determine the uncertainty of local image training data based on the prediction results of a local image classification model on local image training data. This uncertainty characterizes the ease or difficulty of data training; data with lower uncertainty is easier for the model to learn. First, local image training data with lower uncertainty is input into the local image classification model for training, which can find the globally optimal solution for the model parameters, reducing the waste of local device computing resources. Then, local image training data with higher uncertainty is input into the local image classification model for training, which can obtain a local optimal solution based on the globally optimal solution, reducing the influence of noisy samples and thus improving the accuracy of the local model parameters. The local model parameters obtained from local training are then sent to a central server for weighted summation of multiple parameters to obtain the target image classification model. Using this target image classification model to classify images can improve the accuracy of image classification. Attached Figure Description

[0047] Figure 1 This is a flowchart of the image classification method provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 Flowchart of the target image classification model construction process in step S102;

[0049] Figure 3 yes Figure 2 The flowchart of step S202 in the document;

[0050] Figure 4 yes Figure 2 The flowchart of step S204 in the process;

[0051] Figure 5 yes Figure 4 The flowchart of step S403 in the process;

[0052] Figure 6 yes Figure 2 The flowchart of step S204 in the process;

[0053] Figure 7 This is a schematic diagram illustrating the target image classification model construction process provided in another embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the structure of the image classification device provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Image classification is an image processing method that distinguishes different categories of objects based on the different features they reflect in image information. It uses computers to perform quantitative analysis of images, classifying each pixel or region in an image into one of several categories, thus replacing human visual interpretation.

[0062] Based on this, embodiments of this application provide an image classification method and apparatus, an electronic device and a storage medium, aiming to reduce the waste of computing resources and improve the accuracy of image classification.

[0063] The image classification method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the image classification method in this application is described.

[0064] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0065] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0066] The image classification method provided in this application relates to the field of artificial intelligence technology. The image classification method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image classification method, but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0069] Figure 1 This is an optional flowchart of the image classification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 and S103.

[0070] Step S101: Obtain the image to be classified;

[0071] Step S102: Input the image to be classified into the target image classification model, and extract features from the image to be classified through the target image classification model to obtain image feature data;

[0072] Step S103: Based on the image feature data, the image to be classified is predicted using the target image classification model to obtain the image classification result;

[0073] Please refer to Figure 2 The target image classification model in this application embodiment is obtained through steps S201 to S205:

[0074] Step S201: Obtain multiple local image training data and a local image classification model;

[0075] Step S202: Determine the prediction result of the local image training data based on the local image training data and the local image classification model, wherein the prediction result includes the prediction probability of multiple categories;

[0076] Step S203: Determine the uncertainty of the local image training data based on the prediction results and the actual labels of the local image training data;

[0077] Step S204: Input the local image training data into the local image classification model for training based on the uncertainty. The local image training data with lower uncertainty is first input into the local image classification model for training.

[0078] Step S205: Send the local model parameters of the trained local image classification model to the central server for weighted summation to obtain the trained target image classification model.

[0079] Steps S101 to S103 and S201 to S205, as illustrated in this embodiment, determine the uncertainty of the local image training data based on the prediction results of the local image classification model. This uncertainty characterizes the ease or difficulty of data training; data with lower uncertainty is easier for the model to learn. First, inputting local image training data with lower uncertainty into the local image classification model allows finding the global optimum of the model parameters, reducing the waste of local device computing resources. Then, inputting local image training data with higher uncertainty into the local image classification model allows obtaining a local optimum based on the global optimum, reducing the influence of noisy samples and improving the accuracy of the model parameters. Finally, the local model parameters obtained from local training are sent to a central server for weighted summation of multiple local model parameters to obtain the target image classification model. This not only reduces the waste of computing resources on both the local device and the central server but also improves the performance of the target image classification model by finding a local optimum based on the global optimum. Using this target image classification model to classify images can improve the accuracy of image classification.

[0080] In step S101 of some embodiments, the image to be classified may be a set of computer data. This set of computer data is used to record the brightness information of each point on the image. By restoring this set of computer data through a display device, the corresponding brightness information can be displayed at each point position on the display device, thereby displaying the image. The image to be classified may be an image input by the user through a terminal interface, an image stored locally on the terminal, or an image from the network. This application embodiment does not impose specific limitations.

[0081] In steps S102 and S103 of some embodiments, in federated learning, the target image classification model is obtained by the central server aggregating and optimizing the model parameters trained on local image classification models by multiple edge devices. The target image classification model is stored in the central server. When the terminal executing the image classification method is an edge device, the edge device only needs to download the parameters of the target image classification model from the central server and use the downloaded parameters as the parameters of its local image classification model to obtain the target image classification model, without needing to download the complete target image classification model from the central server, thus reducing the amount of data interaction between the edge device and the central server. When the terminal executing the image classification method is not an edge device, it needs to download the complete target image classification model from the central server. In the target image classification model, the image classification process includes feature extraction of the image to be classified and classifying the features using a classifier to predict the category. Specifically, the target image classification model first processes and extracts the features of the image to be classified, obtaining image feature data. Then, a classifier maps the extracted image feature data into a probability vector with a length equal to the number of categories. The value at each position in the probability vector represents the probability value of the corresponding category. Further, a discriminator outputs the category corresponding to the maximum value in the probability vector; this category is the image classification result. The classifier of the target image classification model in this embodiment can be built using algorithms such as support vector machines, neural networks, and random forests. Feature extraction can employ convolutional neural network algorithms, HOG (Histogram of Oriented Gradients) algorithms, scale-invariant feature transform algorithms, etc. This embodiment does not impose specific limitations.

[0082] In steps S201 to S205 of some embodiments, the construction process of the target image classification model can be implemented in a federated learning framework. The federated learning framework includes multiple edge devices and a central server. The edge devices train local image classification models using local image training data based on image classification learning tasks issued by the central server to obtain local model parameters. The central server receives the local model parameters from the multiple edge devices and performs a weighted summation of these parameters to obtain the target image classification model. Further, the central server compares the performance of the currently weighted summed target image classification model with that of the previous target image classification model. If the current target image classification model performs better, the edge devices download the latest target image classification model parameters from the central server to update their local image classification models. Taking medical image recognition as an example, the edge devices can be local servers within hospitals, and the central server interacts with multiple local servers. The central server issues an image classification task for liver tumor recognition to the local servers. The local server filters liver images from its local database based on the image classification task, and performs tumor labeling and other processing on the liver images to obtain local image training data. This local image training data is then used to train the specified algorithm model for the image classification task, yielding the local model parameters. The central server performs a weighted summation of the local model parameters obtained from multiple hospital local servers to obtain a target image classification model that can be used to identify liver tumors.

[0083] It is understood that the central server performs a weighted summation of the local model parameters of multiple local image classification models. This can be done by taking the average of the multiple local model parameters or by assigning weights to the multiple local model parameters and then taking the average. This application embodiment does not impose any specific limitations.

[0084] It is understood that steps S201 to S205 in the embodiments of this application are performed by the edge device.

[0085] In step S201 of some embodiments, the local image classification model includes feature extraction of the image to be classified and prediction of the features using a classifier to obtain a prediction result. The prediction result includes the predicted probabilities of multiple categories. The classifier algorithm used by the local image classification model can be support vector machine, Naive Bayes, neural network, random forest, etc. The algorithm used by the local image classification model can be specified by the image classification task issued by the central server, and the corresponding algorithm code can be downloaded from the central server or from other paths. Before the edge device trains the local image classification model, multiple local image training data that meet the task requirements are selected from the local database according to the image classification task, and then the training dataset is formed. The local image training data includes image samples and actual labels of the samples. The actual labels are the true classification results of the image samples.

[0086] In step S202 of some embodiments, the forward propagation process of local image training data in the local image classification model is the process of classifying and predicting image samples to obtain prediction results. Image samples from the local image training data are input into the pre-trained local image classification model to obtain the predicted probabilities of the image samples belonging to different categories, thereby obtaining the prediction results. For example, if the image classification task is to classify animals, after inputting animal image samples into the local image classification model, the intermediate features of the image samples are first processed and extracted. Then, the classifier maps the extracted features into a probability vector [0.5, 0.3, 02] with a length of 3. This probability vector represents the probability values ​​of cats, tigers, and lions as 50%, 30%, and 20%, respectively. In the embodiments of this application, the prediction result of the local image training data can be a probability vector composed of multiple category probability values ​​obtained after the image samples from the local image training data pass through the classifier of the local image classification model.

[0087] In step S203 of some embodiments, after obtaining the prediction result of the local image training data, the predicted probability of the sample image belonging to the actual category is determined based on the actual label of the local image training data. The uncertainty of the local image training data can be determined based on this predicted probability. For example, if the predicted probabilities for cat, tiger, and lion are 0.5, 0.3, and 0.2 respectively, and the actual label is tiger, then the predicted probability of the actual category is 0.3. The predicted probability of the sample image belonging to the actual category is negatively correlated with the uncertainty. The higher the predicted probability that a sample image belongs to the actual category, the more accurate the local image classification model's classification prediction for that image sample, and the easier it is to learn the features of that image sample. This results in lower uncertainty in the local image training data. Conversely, the lower the predicted probability that a sample image belongs to the actual category, the greater the prediction error of the local image classification model for that image sample, and the more difficult it is to learn the features of that image sample. This also results in higher uncertainty in the local image training data. Therefore, this embodiment of the application determines the uncertainty of the local image training data based on the prediction results of the local image classification model and the actual labels. This uncertainty characterizes the training difficulty of the local image training data. Local image training data with low uncertainty is easy to learn (simple samples), while local image training data with high uncertainty is difficult to learn (difficult samples). This allows for quick and automatic filtering of simple and difficult samples from multiple local image training data sets, reducing manual filtering costs and improving the efficiency of filtering simple and difficult samples.

[0088] In step S203 of some embodiments, the uncertainty of the local image training data can be calculated using formula (1):

[0089]

[0090] Where P(y=t|x) represents the predicted probability of the actual label y of the local image training data x being category t, obtained by the pre-trained model predicting the local image training data x, and T represents the total number of local image training data predicted.

[0091] In step S204 of some embodiments, after determining the prediction results and calculating the uncertainty of the local image training data through the forward propagation process of the local image classification model, training data with low uncertainty is preferentially input into the initialized local image classification model for training based on the uncertainty of the local image training data. During the training process of the local image classification model based on training data with low uncertainty, the model parameters are optimized by calculating the gradient of backpropagation based on the actual labels of the training data. Since data with lower uncertainty is easier for the model to learn, training the model first with local image training data with low uncertainty can find the global optimal solution of the model parameters, improve the model's generalization ability and convergence speed, and reduce the waste of local device computing resources. Then, inputting local image training data with higher uncertainty into the local image classification model for training can obtain a local optimal solution based on the global optimal solution, reduce the influence of noisy samples, and improve the accuracy of model parameters.

[0092] In step S205 of some embodiments, after the edge device completes the training of its local image classification model, it sends the trained local model parameters to the central server for integration with the local model parameters obtained by other edge devices. Since the local model parameters trained by the edge devices avoid local optima, the central server can quickly obtain a high-performance target image classification model during the aggregation of multiple local model parameters, reducing the number of times parameters need to be repeatedly updated and iterated using the edge devices, thereby reducing the computational resources of the central server. The target image classification model obtained by the central server by aggregating the parameters from the edge devices also has high accuracy. Using this target image classification model to classify images can improve the accuracy of image classification results.

[0093] Please see Figure 3 In some embodiments, step S202 may include, but is not limited to, steps S301 to S303;

[0094] Step S301: Divide the multiple local image training data into pre-training data and test data;

[0095] Step S302: Input the local image training data, which is used as pre-training data, into the local image classification model for training to obtain the pre-trained model;

[0096] Step S303: Input the local image training data, which is used as test data, into the pre-trained model to obtain the prediction results of the local image training data.

[0097] In step S301 of some embodiments, the pre-training data is local image training data used to initialize model parameters to obtain a pre-trained model. The test data is local image training data used to obtain the parameters of the local image classification model. A certain number of local image training data are randomly selected from the training dataset as pre-training data, and the remaining local image training data in the training dataset are used as test data. In the process of randomly selecting pre-training data, a fixed number can be selected, or a proportion of the number can be selected. For example, 200 local image training data can be randomly selected from the training dataset as pre-training data, or 10% of the local image training data in the training dataset can be selected as pre-training data.

[0098] In steps S302 and S303 of some embodiments, before using the local image classification model to perform classification prediction to determine the uncertainty of the local image training data, it is necessary to first set the model parameters to obtain a pre-trained model capable of performing preliminary image classification prediction. By inputting the local image training data, which serves as pre-training data, into the local image classification model for training, preliminary model parameters are obtained, thereby acquiring a pre-trained model for the current image classification task. Using this pre-trained model, the test data can be classified and predicted more accurately, thus improving the accuracy of the uncertainty of the local image training data.

[0099] Please see Figure 4 In some embodiments, step S204 may include, but is not limited to, steps S401 to S403;

[0100] Step S401: Divide the local image training data with uncertainty less than or equal to the classification threshold into the low uncertainty dataset;

[0101] Step S402: Local image training data with uncertainty greater than the classification threshold are divided into high uncertainty datasets;

[0102] Step S403: Input the local image training data from the low uncertainty dataset into the local image classification model for training, and then input the local image training data from the high uncertainty dataset into the local image classification model for training.

[0103] In steps S401 and S402 of some embodiments, multiple local image training data can be divided into easy and hard samples by setting a classification threshold. Specifically, low-uncertainty data and high-uncertainty datasets are created. The low-uncertainty dataset is used to store easy samples, and the high-uncertainty dataset is used to store hard samples. After determining the uncertainty of the local image training data, the uncertainty is compared with the classification threshold. If the uncertainty is less than or equal to the classification threshold, the local image training data is stored in the low-uncertainty dataset; if the uncertainty is greater than the classification threshold, the local image training data is stored in the high-uncertainty dataset. Through the above process, multiple local image training data are sequentially stored in their corresponding datasets, thereby dividing the multiple local image training data into easy and hard samples.

[0104] In step S403 of some embodiments, during the local image classification model training process, local image training data from the low uncertainty dataset is first input into the local image classification model to train and obtain the global optimal solution of the model parameters. Then, local image training data from the high uncertainty dataset is input into the local image classification model to optimize and obtain the local optimal solution of the model parameters based on the global optimal solution.

[0105] It is understood that the embodiments of this application can also divide multiple local image training data into samples with different levels of training difficulty by setting multiple classification thresholds. For example, the embodiments of this application can set a first classification threshold and a second classification threshold to create a first uncertainty dataset, a second uncertainty dataset, and a third uncertainty dataset. Local image training data with uncertainty less than the first classification threshold is stored in the first uncertainty dataset, local image training data with uncertainty between the first and second classification thresholds is stored in the second uncertainty dataset, and local image training data with uncertainty greater than the second classification threshold is stored in the third uncertainty dataset. Then, the first uncertainty dataset, the second uncertainty dataset, and the third uncertainty dataset are sequentially input into the local image classification model for training. The embodiments of this application do not impose specific limitations on the granularity of dividing the training difficulty of multiple local image training data.

[0106] Please see Figure 5 In some embodiments, step S403 may include, but is not limited to, steps S501 and S502.

[0107] Step S501: Input the local image training data from the low uncertainty dataset into the local image classification model for training;

[0108] Step S502: Input the local image training data from the high uncertainty dataset into the local image classification model one by one for training until the loss value of the local image classification model is less than the preset loss value.

[0109] In step S501 of some embodiments, in the early stage of training the local image classification model, all local image training data in the low uncertainty dataset is put into training, while local image training data in the high uncertainty dataset is not included in the initial training, in order to reduce the influence of sample noise, avoid getting trapped in local optima, and search for the global optimal solution of model parameters.

[0110] In step S502 of some embodiments, after the local image training data in the low uncertainty dataset is used for training, as the number of training iterations increases, the local image training data in the high uncertainty dataset is input one by one into the local image classification model for training, until the loss value of the local image classification model is less than a preset loss value or until all the local image training data in the high uncertainty dataset has been used for training. Further, the local image training data in the high uncertainty dataset can be input into the local image classification model one by one according to their uncertainty from smallest to largest.

[0111] In steps S501 and S502 of some embodiments, the training process of the local image classification model is specifically as follows: Local image training data from the dataset is input into the initialized local image classification model to obtain the model's output prediction results. The accuracy of the local image classification model's predictions can be evaluated based on the prediction results and the aforementioned actual labels, thereby updating the model's parameters. The accuracy of the model's classification prediction results can be measured by a loss function. The loss function is defined on a single training data point and is used to measure the prediction error of a single local image training data point. Specifically, the loss value of the training data point is determined by the actual label of the single local image training data point and the model's prediction result for that local image training data point. However, a large amount of local image training data is used in actual training. Therefore, a cost function is generally used to measure the overall error of multiple local image training data points that have been used for training. The cost function is used to calculate the average prediction error of all multiple local image training data points that have been used for training, which can better measure the model's classification prediction performance. There are many commonly used loss functions, such as 0-1 loss function, squared loss function, absolute loss function, log loss function, cross-entropy loss function, etc., which can all be used as loss functions for machine learning models and will not be elaborated on here. In this embodiment, any loss function can be selected to determine the training loss value. Based on the training loss value, the backpropagation algorithm is used to update the model parameters, and a well-trained local image classification model can be obtained after several iterations. Specifically, the number of iterations can be preset, or training can be considered complete when the accuracy requirement on the test set is met.

[0112] In this embodiment, the loss value of the local image classification model is calculated using the cross-entropy loss function. The cross-entropy loss function can accelerate the convergence speed of the network model and is applicable to multi-classification tasks, with high versatility. The cross-entropy loss function is expressed as formula (2):

[0113]

[0114] Where L is the loss value, N is the number of local image training data, and y ic Let y be a sign function, where y is the true class of the local image training data i equal to c. ic =1, otherwise y ic =0, P ic M represents the predicted probability that local image training data i belongs to category c, where M represents the total number of categories.

[0115] Please see Figure 6 In some embodiments, step S204 may include, but is not limited to, steps S601 and S602.

[0116] Step S601: Arrange multiple local image training data in ascending order of uncertainty;

[0117] Step S602: The arranged local image training data is sequentially input into the local image classification model for training until the loss value of the local image classification model is less than the preset loss value.

[0118] In some embodiments, steps S601 and S602, besides the embodiment where multiple local image training data used as test data are divided into low-uncertainty datasets and high-uncertainty datasets respectively, and all data from the low-uncertainty datasets are first used for training, followed by the data from the high-uncertainty datasets sequentially, it is also possible not to divide the multiple local image training data. Instead, the multiple local image training data are arranged in ascending order of uncertainty, and the arranged local image training data are sequentially input into the local image classification model for training until the loss value of the local image classification model is less than a preset loss value or until all data has been used for training. The training process of the local image classification model has been described in the above embodiments and will not be repeated here.

[0119] Please see Figure 7 In some embodiments, the process of an edge device acquiring a target image classification model is described, such as... Figure 7 As shown, the edge device includes an APL (Automatic Progressive Learning) module, which includes a sample difficulty estimator and a sample scheduler.

[0120] Step S701: The edge device trains a local image classification model to obtain local model parameters, as follows:

[0121] Step S7011: Randomly select a portion of local image pre-training data from the training dataset as pre-training data, and use the remaining local image training data in the training dataset as test data.

[0122] Step S7012: Input the pre-trained data into the local image classification model to obtain the pre-trained model;

[0123] Step S7013: Input the test data into the pre-trained model to obtain the prediction results of the test data;

[0124] Step S7014: Determine the uncertainty of the test data based on the prediction results of the test data and the actual labels of the test data;

[0125] Step S7015: Sort the multiple test data according to the uncertainty of the test data;

[0126] Step S7016: Divide the sorted test data into low uncertainty data and high uncertainty data;

[0127] Step S7017: After feeding all low uncertainty data into the local image classification model, feed multiple high uncertainty data into the local image classification model one by one for training until all test data are fed into the training.

[0128] Step S7018: Validate the trained local image classification model based on the validation data, and obtain the local model parameters of the local image classification model after passing the validation.

[0129] In step S702, the edge device uploads the local model parameters to the central server;

[0130] Step S703: The central server integrates and evaluates multiple local model parameters to obtain the target model parameters;

[0131] In step S704, the edge device downloads the target model parameters from the central server to obtain the target image classification model.

[0132] Please see Figure 8 This application also provides an image classification apparatus that can implement the above-described image classification method. The apparatus includes:

[0133] The first module is used to acquire the image to be classified.

[0134] The second module is used to input the image to be classified into the target image classification model, extract features from the image to be classified through the target image classification model to obtain image feature data; and perform classification prediction on the image to be classified through the target image classification model based on the image feature data to obtain the image classification result.

[0135] The target image classification model is obtained through the following steps:

[0136] Acquire multiple local image training data and local image classification models;

[0137] The prediction results of the local image training data are determined based on the local image training data and the local image classification model. The prediction results include the prediction probabilities of multiple categories.

[0138] The uncertainty of the local image training data is determined based on the prediction results and the actual labels of the local image training data.

[0139] Local image training data is input into the local image classification model for training based on uncertainty. Local image training data with lower uncertainty is input into the local image classification model for training first.

[0140] The local model parameters of the trained local image classification model are sent to the central server for weighted summation of multiple local model parameters to obtain the trained target image classification model.

[0141] The specific implementation of this image classification device is basically the same as the specific implementation of the image classification method described above, and will not be repeated here.

[0142] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0143] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0144] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0145] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the image classification method of the embodiments of this application.

[0146] The input / output interface 903 is used to implement information input and output;

[0147] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0148] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0149] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0150] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described image classification method.

[0151] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The image classification method, image classification device, electronic device, and storage medium provided in this application determine the uncertainty of local image training data based on the prediction results of a local image classification model on local image training data. This uncertainty characterizes the ease or difficulty of data training; data with lower uncertainty is easier for the model to learn. First, local image training data with lower uncertainty is input into the local image classification model for training, which can find the global optimal solution for the model parameters, reducing the waste of local device computing resources. Then, local image training data with higher uncertainty is input into the local image classification model for training, which can obtain a local optimal solution based on the global optimal solution, reducing the influence of noisy samples and thus improving the accuracy of the model parameters. The model parameters obtained from local training are then sent to a central server for weighted summation of multiple local model parameters to obtain the target image classification model. This not only reduces the waste of computing resources of local devices and the central server but also finds a local optimal solution based on the global optimal solution, improving the performance of the target image classification model. The target image classification model is then used to classify the images to be classified, improving the accuracy of image classification.

[0153] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0154] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0157] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0158] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0160] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image classification method, characterized in that, The method includes: Obtain the image to be classified; The image to be classified is input into the target image classification model, and the target image classification model is used to extract features from the image to be classified to obtain image feature data; Based on the image feature data, the target image classification model is used to perform classification prediction on the image to be classified, and the image classification result is obtained. The target image classification model is obtained through the following steps: Acquire multiple local image training data and local image classification models; The prediction result of the local image training data is determined based on the local image training data and the local image classification model, wherein the prediction result includes the prediction probability of multiple categories; The uncertainty of the local image training data is determined based on the prediction results and the actual labels of the local image training data; specifically, this includes: determining the predicted probability that a sample image belongs to the actual category based on the actual labels of the local image training data; and determining the uncertainty of the local image training data based on the predicted probability; wherein the uncertainty is negatively correlated with the predicted probability that a sample image belongs to the actual category. The local image training data is input into the local image classification model for training based on the uncertainty, wherein the local image training data with lower uncertainty is input into the local image classification model for training first; The local model parameters of the trained local image classification model are sent to the central server for weighted summation of multiple local model parameters to obtain the trained target image classification model.

2. The image classification method according to claim 1, characterized in that, The step of inputting the local image training data into the local image classification model for training based on the uncertainty includes: The local image training data with uncertainty less than or equal to the classification threshold are divided into a low uncertainty dataset; The local image training data with uncertainty greater than the classification threshold are classified into a high uncertainty dataset; After inputting the local image training data from the low uncertainty dataset into the local image classification model for training, the local image training data from the high uncertainty dataset is then input into the local image classification model for training.

3. The image classification method according to claim 1, characterized in that, The step of inputting the local image training data into the local image classification model for training based on the uncertainty includes: The local image training data are arranged in ascending order of uncertainty. The arranged local image training data is sequentially input into the local image classification model for training until the loss value of the local image classification model is less than the preset loss value.

4. The image classification method according to claim 2, characterized in that, The step of inputting local image training data from the low-uncertainty dataset into the local image classification model for training, and then inputting local image training data from the high-uncertainty dataset into the local image classification model for training, includes: All local image training data in the low uncertainty dataset are input into the local image classification model for training; The local image training data in the high uncertainty dataset are input one by one into the local image classification model for training until the loss value of the local image classification model is less than the preset loss value.

5. The image classification method according to claim 3, characterized in that, The loss value of the local image classification model is calculated using the cross-entropy loss function, which is expressed as follows: ； in, This is the loss value. The amount of local image training data. For the sign function, when the local image training data The true category equals , ,otherwise 0, Training data for local images Category c The predicted probability, This indicates the total number of categories.

6. The image classification method according to claim 1, characterized in that, Determining the prediction result of the local image training data based on the local image training data and the local image classification model includes: The local image training data is divided into pre-training data and test data; The local image training data, which is used as pre-training data, is input into the local image classification model for training to obtain the pre-trained model; The local image training data, which serves as test data, is input into the pre-trained model to obtain the prediction results of the local image training data.

7. The image classification method according to claim 6, characterized in that, The uncertainty is calculated using the following formula: = ； in, This indicates that the pre-trained model uses the local image training data. Local image training data obtained for prediction Actual label For category The predicted probability, This represents the total amount of local image training data predicted.

8. An image classification device, characterized in that, The device includes: The first module is used to acquire the image to be classified. The second module is used to input the image to be classified into the target image classification model, extract features from the image to be classified through the target image classification model to obtain image feature data; and perform classification prediction on the image to be classified through the target image classification model based on the image feature data to obtain the image classification result. The target image classification model is obtained through the following steps: Acquire multiple local image training data and local image classification models; The prediction result of the local image training data is determined based on the local image training data and the local image classification model, wherein the prediction result includes the prediction probability of multiple categories; The uncertainty of the local image training data is determined based on the prediction results and the actual labels of the local image training data; specifically, this includes: determining the predicted probability that a sample image belongs to the actual category based on the actual labels of the local image training data; and determining the uncertainty of the local image training data based on the predicted probability; wherein the uncertainty is negatively correlated with the predicted probability that a sample image belongs to the actual category. The local image training data is input into the local image classification model for training based on the uncertainty, wherein the local image training data with lower uncertainty is input into the local image classification model for training first; The local model parameters of the trained local image classification model are sent to the central server for weighted summation of multiple local model parameters to obtain the trained target image classification model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification method according to any one of claims 1 to 7.

Images