Image classification method, system and device and computer readable storage medium
By receiving client knowledge vectors and drift metrics, dynamically adjusting clustering, and introducing a negative migration suppression strategy, the problem of limited model optimization and generalization ability in image classification is solved, achieving accurate classification and improved stability while protecting privacy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-01-23
- Publication Date
- 2026-04-28
AI Technical Summary
In image classification tasks, due to the differences in the models of image acquisition devices in different regions and the need for data confidentiality, it is difficult to use the data used for training each client in a centralized manner, which limits the optimization degree and generalization ability of the model. Moreover, existing federated learning methods are difficult to classify accurately while protecting image privacy.
By receiving knowledge vectors from clients, drift magnitude and drift metrics are generated, clustering is dynamically adjusted, a negative migration suppression strategy is introduced, the model training process is optimized, and the clustering structure is ensured to be consistent with the data characteristics to avoid erroneous fusion.
It achieves accurate image classification while protecting image privacy, improves model stability and generalization performance, and reduces the risk of erroneous fusion.
Smart Images
Figure CN121937796A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology based on federated learning, and more specifically, to an image classification method, system, device, and computer-readable storage medium. Background Technology
[0002] In intelligent diagnostic tasks such as image classification, the development of neural network models enables image classification, such as categorizing images into specific targets like pedestrians and pets. However, this process is hampered by differences in image acquisition equipment models and acquisition procedures across different regions. Consequently, different clients use different data to train their image processing models. Furthermore, due to data confidentiality and security requirements, the data used for training by each client is difficult to centralize, resulting in a limited sample size for model training and severely restricting the model's optimization and generalization capabilities.
[0003] To address this issue, federated learning can be used to collaboratively train models without sharing raw data among clients. However, during training, the server only performs a one-time clustering of clients in the initial stage or completely ignores client differences, using the same hyperparameters for optimization across different clusters. This makes it difficult to adapt to the differences in knowledge between different clusters. As training progresses and client features evolve, the original clustering structure is still used, which can easily lead to incorrect model fusion and performance degradation. This results in insufficient optimization or getting stuck in local optima, impairing the generalization performance of the image processing model trained on the clients and making it unable to accurately classify images.
[0004] In conclusion, how to accurately classify images while protecting image privacy is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide an image classification method that can, to some extent, solve the technical problem of accurately classifying images while protecting image privacy. This application also provides an image classification system, an electronic device, and a computer-readable storage medium.
[0006] To achieve the above objectives, this application provides the following technical solution: An image classification method, applied to a server, includes: Receive knowledge vectors from each client participating in the training of the image processing model; Generate the drift of each client's knowledge vector between adjacent training rounds; The drift amount of each client is statistically processed to generate drift metrics for all clients; If the drift index is greater than the set value, the client is re-clustered based on the client's knowledge vector to obtain the target clustering result; The client's image processing model is trained based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model; The knowledge vector includes the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The uncertainty statistical features are used to reflect the model's prediction stability and the level of data noise.
[0007] Preferably, the step of statistically processing the drift amount of each client to generate a drift index for all clients includes: The drift amount of each client is averaged to generate a drift metric for all clients.
[0008] Preferably, training the client's image processing model based on the target clustering results includes: Receive model parameters from each client, including the model update amount and the number of samples in the local training images; For each cluster in the target clustering result, generate an intra-cluster model update vector based on the model parameters of the clients within the cluster; Based on the cluster-internal model update vector and the model parameters of the cluster-internal clients, a feedback index reflecting the consistency between the client and the overall cluster direction is generated. Based on the client's feedback metrics, generate the client's negative migration inhibition weight; Update cluster model parameters based on client-side negative migration suppression weights and model parameters; The cluster model parameters are sent to each client within the cluster so that the clients can use the cluster model parameters to train the image processing model.
[0009] Preferably, generating the intra-cluster model update vector based on the model parameters of the intra-cluster clients includes: The total number of client samples generated within the cluster; Based on the ratio of the number of samples of clients within the cluster to the total number, the model update amount of clients within the cluster is scaled to obtain the model update scaling amount of clients within the cluster. The model update scaling for all clients within the cluster is accumulated element by element to generate the cluster-wide model update scaling.
[0010] Preferably, the step of generating a feedback index reflecting the consistency between the client and the overall cluster direction based on the cluster-internal model update vector and the model parameters of the client within the cluster includes: The model parameters of the clients within the cluster and the update vector of the models within the cluster are processed by cosine similarity to generate similarity values for the clients within the cluster. The similarity value of clients within a cluster is used as a feedback indicator to reflect the consistency between the client and the overall cluster direction.
[0011] An image classification method, applied to a client, includes: The overall discriminative tendency index of the image processing model generated for the client; Uncertainty statistical features of the generated image processing model are used to reflect the model's predictive stability and data noise level. The overall discrimination tendency index and the uncertainty statistical characteristics are used as the client's knowledge vector; The client knowledge vectors are transmitted to the server so that the server receives the knowledge vectors of each client participating in the training of the image processing model, generates the drift amount of each client's knowledge vector between adjacent training rounds, performs statistical processing on the drift amount of each client, generates a drift index for all clients, and in response to the drift index being greater than the set value, the clients are re-clustered according to the client's knowledge vectors to obtain the target clustering result, and the client's image processing model is trained according to the target clustering result. Obtain the image to be processed; The classification results of the image to be processed are generated by the trained image processing model.
[0012] Preferably, the overall discriminative tendency index of the image processing model for generating the client includes: Generate the output values of the client's image processing model for the locally trained images; Generate the mean feature vector of the output value; The mean feature vector is used as an overall discriminative tendency index for the image processing model; The uncertainty statistical characteristics of the generated image processing model include: Perform a softmax operation on the output value to generate a predicted probability distribution value; Generate the prediction entropy value of the predicted probability distribution; The predicted entropy value is used as an uncertainty statistical feature of the image processing model.
[0013] An image classification system, applied to a server, includes: The knowledge vector receiving module is used to receive the knowledge vectors from various clients participating in the training of the image processing model. The drift generation module is used to generate the drift of the knowledge vector of each client between adjacent training rounds; The drift metric generation module is used to statistically process the drift amount of each client and generate drift metrics for all clients. The clustering module is used to re-cluster the client based on the client's knowledge vector in response to the drift index being greater than the set value, so as to obtain the target clustering result; The update module is used to train the client's image processing model based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model; The knowledge vector includes the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The uncertainty statistical features are used to reflect the model's prediction stability and the level of data noise.
[0014] An electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of any of the image classification methods described above.
[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image classification methods described above.
[0016] This application provides an image classification method applied to a server. The method receives knowledge vectors from various clients participating in the training of an image processing model; generates the drift amount of each client's knowledge vector between adjacent training rounds; statistically processes the drift amounts of each client to generate a drift index for all clients; in response to a drift index exceeding a set value, the clients are re-clustered based on their knowledge vectors to obtain a target clustering result; and the client's image processing model is trained based on the target clustering result, enabling the client to generate classification results for the image to be processed using the trained image processing model. The knowledge vectors include an overall discriminative tendency index and uncertainty statistical features of the image recognition model, whereby the uncertainty statistical features reflect the model's predictive stability and the level of data noise. In this application, the client does not transmit the images used for training, but instead transmits the overall discriminative tendency index and uncertainty statistical features of the image recognition model, protecting image privacy. These two data represent the client from the perspectives of "discriminative ability" and "uncertainty level," respectively, and are combined to form the client's knowledge vector. This effectively distinguishes clients with similar average prediction results but different data quality, noise levels, or acquisition device conditions, avoiding the information loss problem caused by relying solely on mean features. This dual-perspective knowledge vector provides a more robust and discriminative feature foundation for subsequent dynamic clustering and drift detection of the client. Subsequently, knowledge drift generated by the knowledge vector is introduced as the basis for clustering adjustment. When the change amplitude of all client knowledge vectors exceeds a set value, re-clustering is performed, realizing triggered re-clustering of the client. This not only avoids the computational overhead of forced clustering in each round, but also adjusts the clustering structure in a timely manner when client features evolve, ensuring that the client partitioning always remains consistent with the data characteristics of the current training stage. This effectively reduces the risk of erroneous fusion caused by static clustering, improves the stability of the federated training process and the final model performance, and enables the client to accurately classify images with the help of the image processing model. The image classification system, electronic device, and computer-readable storage medium provided in this application also solve the corresponding technical problems. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0018] Figure 1 A flowchart illustrating an image classification method provided in this application embodiment; Figure 2 A flowchart illustrating the process of tumor classification; Figure 3 This is a schematic diagram of the structure of an image classification system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 5 This is another structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] Please see Figure 1 , Figure 1 This is a flowchart of an image classification method provided in an embodiment of this application.
[0021] This application provides an image classification method applied to a server, which may include the following steps: Step S101: Receive the knowledge vectors of each client participating in the training of the image processing model. The knowledge vectors include the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The uncertainty statistical features are used to reflect the model's prediction stability and the level of data noise.
[0022] In practical applications, considering that federated learning requires multiple rounds of training, as the number of training rounds increases, the client's local image processing model's ability to represent image features changes. The client needs to accurately express this ability to the server so that the server can decide whether to re-cluster the clients. During this process, the client can generate an overall discrimination tendency index and uncertainty statistical features of the image recognition model based on the training results of the image processing model. These uncertainty statistical features reflect the model's prediction stability and data noise level. Correspondingly, the server needs to receive the knowledge vectors from each client participating in the image processing model training so that it can subsequently decide whether to re-cluster the clients in federated learning. It should be noted that the image type used for image processing model training can be set according to the images to be classified, such as ultrasound images, pedestrian images, industrial device images, vehicle images, etc. Accordingly, for example, for a tumor image in ultrasound, the image processing model output can be the tumor category; for a pedestrian image, the output can be the pedestrian's attribute category; for an industrial device image, the output can be the industrial device's fault type; and for a vehicle image, the output can be the vehicle category, etc.
[0023] In an exemplary embodiment, the types of the overall discriminant tendency index and uncertainty statistical features of the image processing model can be flexibly determined according to the application scenario. For example, the overall discriminant tendency index can be the average output vector of the image processing model on the local dataset, and the uncertainty statistical features can be the variance or prediction entropy of the output distribution of the image processing model on the local dataset. Based on this, in the process of generating the overall discriminant tendency index of its own image processing model, the client can generate the output value of the client's image processing model on the local training image, generate the mean feature vector of the output value, and use the mean feature vector as the overall discriminant tendency index of the image processing model. In the process of generating the uncertainty statistical features of the image processing model, the client can perform a softmax operation on the output value to generate the prediction probability distribution value, generate the prediction entropy value of the prediction probability distribution value, and use the prediction entropy value as the uncertainty statistical features of the image processing model.
[0024] For ease of understanding, let's assume the client... k In the t Image processing model after training rounds f for The model uses locally trained images x The output logits are: The corresponding prediction probability distribution is Then the mean eigenvector Represented as , This represents the local training image set of client k. Input features x k This includes pathological features such as tumor size, shape, growth rate, and cellular atypia, as well as clinical features such as patient age, gender, and family history. Output features y k Tumor classification labels are used to indicate benign, malignant, or different tumor subtypes; predictive entropy. Represented as , Then the client's knowledge vector It can be represented as .
[0025] Step S102: Generate the drift of the knowledge vector of each client between adjacent training rounds.
[0026] In practical applications, considering that excessive changes in the client's knowledge vectors between adjacent training epochs could lead to significant variations in the client's image processing model capabilities and consequently alter the client's clustering results, it is possible to generate the drift amount of each client's knowledge vector between adjacent training epochs. For example, if the drift is generated using the L2 norm, then .
[0027] Step S103: Perform statistical processing on the drift amount of each client to generate drift metrics for all clients.
[0028] In practical applications, if the drift amounts of all clients within a cluster change synchronously, the differences between clients within the cluster may not increase. In this case, there is no need to re-cluster. Based on this, the drift amounts of each client can be statistically processed to generate drift indices for all clients. These drift indices can be used to reflect the changes in the image processing model within the client from a global perspective.
[0029] In an exemplary embodiment, the type of statistical processing can be flexibly determined according to the application scenario. For example, to facilitate implementation and accelerate the efficiency of the solution, in the process of statistically processing the drift amount of each client and generating the drift index of all clients, the drift amount of each client can be averaged to generate the drift index of all clients. , , n This indicates the total number of clients.
[0030] Step S104: In response to the drift index being greater than the set value, the client is re-clustered based on the client's knowledge vector to obtain the target clustering result.
[0031] In practical applications, after generating the drift index, it can be compared with a set value. If the drift index is greater than the set value, the client can be re-clustered based on the client's knowledge vector to obtain the target clustering result. If the drift index is less than or equal to the set value, the overall silhouette coefficient of the current clustering result can be generated. , , , Indicates client k The average distance to other points within the same cluster. For the client k The average distance to the nearest other clusters, Indicates client k The setpoint and preset value are determined based on the client's knowledge vector. If the overall silhouette coefficient is less than the preset value, the client can be re-clustered based on the client's knowledge vector to obtain the target clustering result. If the overall silhouette coefficient is greater than or equal to the preset value, the current clustering result can be maintained, and the existing clustering result can be used as the target clustering result. The setpoint and preset value can be flexibly determined according to the application scenario. For example, the server can normalize the client's knowledge vector and determine the setpoint based on the statistics of the drift set of each client, such as taking the quantile or mean plus standard deviation of the drift. Of course, it can also be determined based on empirical values. For example, the setpoint can be taken in the empirical range of 0.2 to 0.4, or set as the initial clustering silhouette coefficient minus the allowable decrease, so as to trigger re-clustering when the clustering quality drops significantly, thereby balancing computational cost and clustering stability.
[0032] Step S105: Train the client's image processing model based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model.
[0033] In practical applications, after re-clustering the client to obtain the target clustering result, the client's image processing model can be trained based on the target clustering result. This allows the client to use the trained image processing model to generate classification results for the images to be processed.
[0034] In an exemplary embodiment, it is considered that within the same client cluster, although the overall data distribution of the clients is similar, the model updates of individual clients may still deviate from the overall optimization direction of the cluster due to differences in sample size, noise level, or data acquisition conditions. If a traditional weighted average method is directly used for model aggregation, the negative impact of abnormal client updates can be easily amplified, causing performance degradation of the cluster model. To avoid this situation, during the training of the client's image processing model based on the target clustering results, model parameters from each client can be received. These model parameters include the model update amount. and the number of samples of locally trained images , , This represents the model parameters of client k at time t. This represents the cluster model parameters of the j-th client cluster. For each cluster in the target clustering result, generate an intra-cluster model update vector based on the model parameters of the clients within the cluster. Based on the intra-cluster model update vector and the model parameters of the clients within the cluster, generate a feedback index reflecting the consistency between the client and the overall cluster direction. Based on the client's feedback index, generate the client's negative migration suppression weight. Update the cluster model parameters based on the client's negative migration suppression weight and model parameters. Send the cluster model parameters to each client within the cluster so that the clients can use the cluster model parameters to train the image processing model.
[0035] In this way, this application introduces a negative migration suppression strategy in the cluster model aggregation stage. It evaluates the model updates uploaded by each client within the cluster and dynamically adjusts the aggregation weights based on their consistency with the overall optimization direction of the cluster or their impact on the validation performance within the cluster. When a client's model update has a negative impact on the cluster model, the server will reduce its aggregation weight or exclude it from the current round of aggregation. Through this mechanism, this application can effectively suppress the negative migration effect caused by abnormal or noisy clients within the cluster, improve the stability and robustness of the cluster model update process, solve the problem that local differences between clients within the cluster may lead to a decrease in model performance, and further improve the generalization performance of image processing models in complex image classification scenarios.
[0036] In an exemplary embodiment, during the process of generating the intra-cluster model update vector based on the model parameters of intra-cluster clients, the total number of intra-cluster client samples can be generated. Based on the proportion of the number of samples from clients within the cluster to the total number of samples, the model update amount for clients within the cluster is scaled to obtain the model update scaling amount for clients within the cluster. The model update scaling amounts for all clients within the cluster are then summed element by element to generate the total model update amount for the cluster. , This allows for a comprehensive reflection of the overall optimization direction of all clients within the cluster by leveraging the amount of model updates within the cluster.
[0037] In an exemplary embodiment, during the process of generating a feedback index reflecting the consistency between the client and the overall cluster direction based on the intra-cluster model update vector and the intra-cluster client model parameters, cosine similarity processing can be performed on the intra-cluster client model parameters and intra-cluster model update vector to generate intra-cluster client similarity values. , , This represents a fixed value; the similarity value of clients within the cluster is used as a feedback indicator reflecting the consistency between the client and the overall cluster orientation. Accordingly, based on the client's feedback indicator, a negative migration inhibition weight is generated for each client. During the process, it can be done Generate, when (During reverse update) This indicates that the client update is considered to potentially cause negative migration to the cluster model, and will be removed or have its weight reset to zero in this round of aggregation. hour, This indicates that the higher the client consistency, the greater the weight; during the process of updating the cluster model parameters based on the client's negative migration suppression weights and model parameters, it can be achieved through... Generate the latest cluster model parameters , This represents the total weighted sample size corresponding to the valid clients retained within the cluster after the introduction of the negative migration suppression mechanism. This is used to ensure that the normalization benchmark for cluster model updates is determined solely by clients with the same orientation, thereby avoiding interference from negative migration clients on the magnitude of cluster model updates.
[0038] This application provides an image classification method applied to a server. The method receives knowledge vectors from various clients participating in the training of an image processing model; generates the drift amount of each client's knowledge vector between adjacent training rounds; statistically processes the drift amounts of each client to generate a drift index for all clients; in response to a drift index exceeding a set value, the clients are re-clustered based on their knowledge vectors to obtain a target clustering result; and the client's image processing model is trained based on the target clustering result, enabling the client to generate classification results for the image to be processed using the trained image processing model. The knowledge vectors include an overall discriminative tendency index and uncertainty statistical features of the image recognition model, whereby the uncertainty statistical features reflect the model's predictive stability and the level of data noise. In this application, the client does not transmit the images used for training, but instead transmits the overall discriminative tendency index and uncertainty statistical features of the image recognition model, protecting image privacy. These two data represent the client from the perspectives of "discriminative ability" and "uncertainty level," respectively, and are combined to form the client's knowledge vector. This effectively distinguishes clients with similar average prediction results but different data quality, noise levels, or acquisition device conditions, avoiding the information loss problem caused by relying solely on mean features. This dual-perspective knowledge vector provides a more robust and discriminative feature foundation for subsequent dynamic clustering and drift detection of the client. Subsequently, knowledge drift generated by the knowledge vector is introduced as the basis for clustering adjustment. When the change amplitude of all client knowledge vectors exceeds a set value, re-clustering is performed, realizing triggered re-clustering of the client. This not only avoids the computational overhead of forced clustering in each round, but also adjusts the clustering structure in a timely manner when client features evolve, ensuring that the client partitioning always remains consistent with the data characteristics of the current training stage. This effectively reduces the risk of erroneous fusion caused by static clustering, improves the stability of the federated training process and the final model performance, and enables the client to accurately classify images with the help of the image processing model.
[0039] To facilitate understanding of the image classification method provided in this application, let's assume that tumor images are being classified. The image processing model would then be a tumor classification model, and the number of clients participating in the training of this model would be three. The implementation process would be as follows: Figure 2 As shown, it can be as follows: Three clients collect and preprocess tumor-related data, including but not limited to medical imaging features, pathological indicators, and clinical features, and construct local datasets D1, D2, and D3. Each local dataset contains samples and their corresponding tumor classification labels. Server initialization starts with model f0( The initial model is then distributed to three clients as the initial model for local training on the three clients. The three clients train and distribute the model locally. When the accuracy of the training set reaches the set threshold α, the training stops, and the models f1, f2, and f3 are obtained. The server performs initial clustering based on the model parameters from the three clients to obtain the target clustering results. ; The server replicates and distributes the global initial model to each client cluster, serving as the initial cluster model for each cluster: ; In the t-th round of training, for each client cluster P j Client Using its local tumor dataset D k and the cluster model The local model is used as the initialization model for local training, and the model parameters are updated through gradient backpropagation to obtain the local model. The corresponding model parameters are denoted as The client sends the model update amount and the number of samples of tumor images trained locally as model parameters to the server; and after completing local training, the three clients extract dual-view knowledge vectors based on their local models. The knowledge vectors include the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The server generates the drift amount of each client's knowledge vector between adjacent training rounds; it statistically processes the drift amount of each client to generate a drift index for all clients; if the drift index is greater than a set value, the clients are re-clustered based on their knowledge vectors to obtain the target clustering result; if the drift index is less than or equal to the set value, the overall silhouette coefficient of the current clustering result can be generated; if the overall silhouette coefficient is less than a preset value, the clients are re-clustered based on their knowledge vectors to obtain the target clustering result; if the overall silhouette coefficient is greater than or equal to the preset value, the current clustering result can be maintained, that is, the existing clustering result is used as the target clustering result. For each cluster in the target clustering result, the server generates an intra-cluster model update vector based on the model parameters of the clients within the cluster; generates a feedback metric reflecting the consistency between the client and the overall cluster direction based on the intra-cluster model update vector and the model parameters of the clients within the cluster; generates negative migration suppression weights for the clients based on the client feedback metrics; generates the latest cluster model parameters based on the client's negative migration suppression weights and model parameters; and sends the cluster model parameters to each client within the cluster as the initial model for the next round of local training. Training terminates when the preset convergence conditions are met, such as reaching the maximum number of iterations T or the cluster model performance stabilizing on the validation set, and the client uses the final model as the tumor classification model; otherwise, it returns to the training in the t-th iteration, targeting each client cluster. P jClient Using its local dataset D k and the cluster model This serves as the initialization model for local training and subsequent steps; The client receives the tumor image to be processed, applies a trained tumor classification model to identify the tumor image, and obtains the tumor classification result.
[0040] Based on the above embodiments, this application provides an image classification method, applied to a client, which may include the following steps: Step 1: Generate the overall discrimination tendency index of the client's image processing model.
[0041] Step 2: Generate the uncertainty statistics of the image processing model. The uncertainty statistics are used to reflect the model's predictive stability and the level of data noise.
[0042] Step 3: Use the overall discrimination tendency index and uncertainty statistical characteristics as the client's knowledge vector.
[0043] Step 4: Transmit the client knowledge vectors to the server so that the server can receive the knowledge vectors of each client participating in the training of the image processing model, generate the drift amount of each client's knowledge vector between adjacent training rounds, perform statistical processing on the drift amount of each client, generate the drift index of all clients, and if the drift index is greater than a set value, then re-cluster the clients according to the client's knowledge vectors to obtain the target clustering result, and train the client's image processing model according to the target clustering result.
[0044] Step 5: Obtain the image to be processed.
[0045] Step 6: Generate classification results for the image to be processed using the trained image processing model.
[0046] In an exemplary embodiment, during the process of generating the overall discriminative tendency index of the client's image processing model, the client can generate the output value of the client's image processing model for the local training image; generate the mean feature vector of the output value, and use the mean feature vector as the overall discriminative tendency index of the image processing model; during the process of generating the uncertainty statistical features of the image processing model, the output value can be subjected to a softmax operation to generate a predicted probability distribution value, generate a predicted entropy value of the predicted probability distribution value, and use the predicted entropy value as the uncertainty statistical features of the image processing model.
[0047] The description of the image classification method applied to the client provided in this embodiment can be found in the above embodiments, and will not be repeated here.
[0048] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an image classification system provided in an embodiment of this application.
[0049] This application provides an image classification system applied to a server, which may include: The knowledge vector receiving module 101 is used to receive the knowledge vectors of each client participating in the training of the image processing model; The drift generation module 102 is used to generate the drift of the knowledge vector of each client between adjacent training rounds; The drift metric generation module 103 is used to perform statistical processing on the drift amount of each client and generate drift metrics for all clients. Clustering module 104 is used to re-cluster the client based on the client's knowledge vector in response to the drift index being greater than a set value, so as to obtain the target clustering result; The update module 105 is used to train the client's image processing model based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model; The knowledge vector includes the overall discrimination tendency index of the image recognition model and the uncertainty statistical features, which are used to reflect the model's prediction stability and the level of data noise.
[0050] This application provides an image classification system applied to a server, wherein the drift index generation module may include: The drift metric generation unit is used to average the drift amount of each client and generate the drift metric for all clients.
[0051] This application provides an image classification system applied to a server, wherein the update module may include: The model parameter receiving unit is used to receive model parameters from each client. The model parameters include the model update amount and the number of samples in the local training images. The update vector generation unit is used to generate an intra-cluster model update vector for each cluster in the target clustering result, based on the model parameters of the client within the cluster. The feedback index generation unit is used to generate feedback indices that reflect the consistency between the client and the overall cluster direction based on the cluster model update vector and the model parameters of the client within the cluster. The weight generation unit is used to generate negative migration suppression weights for the client based on the client's feedback metrics. The parameter update unit is used to update the cluster model parameters based on the client's negative migration suppression weights and model parameters; The parameter sending unit is used to send cluster model parameters to each client within the cluster, so that the clients can use the cluster model parameters to train the image processing model.
[0052] This application provides an image classification system applied to a server. The update vector generation unit is specifically used to: generate the total number of client samples within the cluster; scale the model update amount of the client within the cluster according to the ratio of the number of client samples within the cluster to the total number, to obtain the model update scaling amount of the client within the cluster; and accumulate the model update scaling amounts of all clients within the cluster element by element to generate the model update amount within the cluster.
[0053] This application provides an image classification system applied to a server. The feedback index generation unit is specifically used to: perform cosine similarity processing on the model parameters of the clients within the cluster and the update vector of the models within the cluster to generate similarity values of the clients within the cluster; and use the similarity values of the clients within the cluster as a feedback index reflecting the consistency between the client and the overall cluster direction.
[0054] This application provides an image classification method, which, when applied to a client, may include: The tendency index generation module is used to generate the overall discrimination tendency index of the client's image processing model; The statistical feature generation module is used to generate uncertainty statistical features of the image processing model. These uncertainty statistical features are used to reflect the model's predictive stability and the level of data noise. The knowledge vector generation module is used to take the overall judgment tendency index and uncertainty statistical characteristics as the client's knowledge vector; The knowledge vector transmission module is used to transmit client knowledge vectors to the server, so that the server can receive the knowledge vectors of each client participating in the training of the image processing model, generate the drift amount of each client's knowledge vector between adjacent training rounds, perform statistical processing on the drift amount of each client, generate the drift index of all clients, and if the drift index is greater than a set value, the clients are re-clustered according to the client's knowledge vector to obtain the target clustering result, and the client's image processing model is trained according to the target clustering result. The image acquisition module is used to acquire the image to be processed. The image recognition module is used to generate classification results for the images to be processed using a trained image processing model.
[0055] This application provides an image classification method applied to a client, wherein the tendency index generation module may include: The output value generation unit is used to generate the output values of the client's image processing model for the locally trained images; The mean generation unit is used to generate the mean feature vector of the output value; The tendency index generation unit is used to use the mean feature vector as the overall discriminative tendency index of the image processing model. The statistical feature generation module may include: The computation unit is used to perform softmax operation on the output value to generate the predicted probability distribution value; The prediction entropy generation unit is used to generate the prediction entropy value of the predicted probability distribution. The statistical feature generation unit is used to use the predicted entropy value as an uncertainty statistical feature of the image processing model.
[0056] This application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the image processing method provided in the embodiments of this application. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0057] An electronic device provided in this application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to implement the steps of the image classification method described in any of the above embodiments.
[0058] Please see Figure 5 Another electronic device provided in this application embodiment may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside. The display unit 204 may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes, but is not limited to, Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth Low Energy communication technology, and communication technology based on IEEE 802.11s.
[0059] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the image classification method described in any of the above embodiments.
[0060] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (compact disc read-only memory), or any other form of storage media known in the art.
[0061] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the image classification method described in any of the above embodiments.
[0062] For descriptions of relevant parts in the image classification system, electronic device, and computer-readable storage medium provided in this application's embodiments, please refer to the detailed description of the corresponding parts in the image classification method provided in this application's embodiments; they will not be repeated here. Furthermore, parts of the technical solutions provided in this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0063] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image classification method, characterized in that, Applied to servers, including: Receive knowledge vectors from each client participating in the training of the image processing model; Generate the drift of each client's knowledge vector between adjacent training rounds; The drift amount of each client is statistically processed to generate drift metrics for all clients; If the drift index is greater than the set value, the client is re-clustered based on the client's knowledge vector to obtain the target clustering result; The client's image processing model is trained based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model; The knowledge vector includes the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The uncertainty statistical features are used to reflect the model's prediction stability and the level of data noise.
2. The method according to claim 1, characterized in that, The process of statistically processing the drift amount of each client to generate drift metrics for all clients includes: The drift amount of each client is averaged to generate a drift metric for all clients.
3. The method according to claim 1, characterized in that, The step of training the client's image processing model based on the target clustering results includes: Receive model parameters from each client, including the model update amount and the number of samples in the local training images; For each cluster in the target clustering result, generate an intra-cluster model update vector based on the model parameters of the clients within the cluster; Based on the cluster-internal model update vector and the model parameters of the cluster-internal clients, a feedback index reflecting the consistency between the client and the overall cluster direction is generated. Based on the client's feedback metrics, generate the client's negative migration inhibition weight; Update cluster model parameters based on client-side negative migration suppression weights and model parameters; The cluster model parameters are sent to each client within the cluster so that the clients can use the cluster model parameters to train the image processing model.
4. The method according to claim 3, characterized in that, The step of generating an intra-cluster model update vector based on the model parameters of the intra-cluster clients includes: The total number of client samples generated within the cluster; Based on the ratio of the number of samples of clients within the cluster to the total number, the model update amount of clients within the cluster is scaled to obtain the model update scaling amount of clients within the cluster. The model update scaling for all clients within the cluster is accumulated element by element to generate the cluster-wide model update scaling.
5. The method according to claim 3, characterized in that, The step of generating a feedback metric reflecting the consistency between the client and the overall cluster direction based on the cluster-internal model update vector and the model parameters of the intra-cluster clients includes: The model parameters of the clients within the cluster and the update vector of the models within the cluster are processed by cosine similarity to generate similarity values for the clients within the cluster. The similarity value of clients within a cluster is used as a feedback indicator to reflect the consistency between the client and the overall cluster direction.
6. An image classification method, characterized in that, Applied to the client side, including: The overall discriminative tendency index of the image processing model generated for the client; Uncertainty statistical features of the generated image processing model are used to reflect the model's predictive stability and data noise level. The overall discrimination tendency index and the uncertainty statistical characteristics are used as the client's knowledge vector; The client knowledge vectors are transmitted to the server so that the server receives the knowledge vectors of each client participating in the training of the image processing model, generates the drift amount of each client's knowledge vector between adjacent training rounds, performs statistical processing on the drift amount of each client, generates a drift index for all clients, and in response to the drift index being greater than the set value, the clients are re-clustered according to the client's knowledge vectors to obtain the target clustering result, and the client's image processing model is trained according to the target clustering result. Obtain the image to be processed; The classification results of the image to be processed are generated by the trained image processing model.
7. The method according to claim 6, characterized in that, The overall discriminative tendency index of the image processing model for generating the client includes: Generate the output values of the client's image processing model for the locally trained images; Generate the mean feature vector of the output value; The mean feature vector is used as an overall discriminative tendency index for the image processing model; The uncertainty statistical characteristics of the generated image processing model include: Perform a softmax operation on the output value to generate a predicted probability distribution value; Generate the prediction entropy value of the predicted probability distribution; The predicted entropy value is used as an uncertainty statistical feature of the image processing model.
8. An image classification system, characterized in that, Applied to servers, including: The knowledge vector receiving module is used to receive the knowledge vectors from various clients participating in the training of the image processing model. The drift generation module is used to generate the drift of the knowledge vector of each client between adjacent training rounds; The drift metric generation module is used to statistically process the drift amount of each client and generate drift metrics for all clients. The clustering module is used to re-cluster the client based on the client's knowledge vector in response to the drift index being greater than the set value, so as to obtain the target clustering result; The update module is used to train the client's image processing model based on the target clustering results, so that the client can generate the classification results of the image to be processed through the trained image processing model; The knowledge vector includes the overall discrimination tendency index and uncertainty statistical features of the image recognition model. The uncertainty statistical features are used to reflect the model's prediction stability and the level of data noise.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image classification method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image classification method as described in any one of claims 1 to 7.