Image processing method and device, computer equipment and storage medium

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: performing feature extraction on a local training sample to obtain a first embedded feature; mixing the first embedded feature with the second embedded feature to obtain a mixed embedded feature; wherein the second embedded feature is an embedded feature which is acquired from the server and uploaded by other clients in the process of locally training the initial classification model; training the initial classification model based on the hybrid embedded features to obtain a trained classification model; uploading model parameters of the classification model to a server, so that the server performs model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used for classifying the to-be-processed image. By adopting the method, the accuracy of image processing can be effectively improved under the condition of ensuring the privacy security of the original data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer and internet technologies, image processing has been widely used in civilian and commercial fields. It plays an increasingly important role in fields such as medical imaging and is also facing higher requirements.

[0003] However, current mainstream image processing methods often perform poorly in medical image classification tasks, especially in the medical field. Image data acquired by different hospitals or equipment exhibits significant differences in image quality, contrast, resolution, and shooting angle. Furthermore, different regions have varying patient structures and clinical practices. Therefore, even when using federated learning training, the multi-client datasets participating in federated learning exhibit significant non-independent and identically distributed (Non-IID) characteristics in their sample features. This severely impacts the performance of federated learning algorithms, leading to difficulties in model convergence and poor classification accuracy. Therefore, effectively improving the accuracy of image processing while ensuring the privacy and security of the original data has become a pressing issue. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems. This method can reduce the impact of non-independent and identically distributed data on the classification results while ensuring the privacy and security of the participants' original data, thereby effectively improving the accuracy of image classification processing.

[0005] In a first aspect, this application provides an image processing method. The method includes: extracting features from local training samples to obtain a first embedded feature; mixing the first embedded feature with a second embedded feature to obtain a hybrid embedded feature; wherein the second embedded feature is an embedded feature uploaded by another client during the local training of an initial classification model, obtained from a server; training the initial classification model based on the hybrid embedded feature to obtain a trained classification model; uploading the model parameters of the classification model to the server, so that the server performs model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0006] Secondly, this application also provides an image processing apparatus. The apparatus includes: an extraction module for extracting features from local training samples to obtain a first embedded feature; a mixing module for mixing the first embedded feature with a second embedded feature to obtain a mixed embedded feature; wherein the second embedded feature is an embedded feature uploaded by another client during the local training of an initial classification model, obtained from a server; a training module for training the initial classification model based on the mixed embedded feature to obtain a trained classification model; and an upload module for uploading the model parameters of the classification model to the server, so that the server performs model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0007] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, performs the following steps: extracting features from local training samples to obtain a first embedded feature; mixing the first embedded feature with a second embedded feature to obtain a hybrid embedded feature; wherein the second embedded feature is an embedded feature uploaded by another client during the local training of an initial classification model, obtained from a server; training the initial classification model based on the hybrid embedded feature to obtain a trained classification model; uploading the model parameters of the classification model to the server, so that the server performs model aggregation based on each model parameter to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0008] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps: extracting features from local training samples to obtain a first embedded feature; mixing the first embedded feature with a second embedded feature to obtain a hybrid embedded feature; wherein the second embedded feature is an embedded feature uploaded by another client during the local training of an initial classification model, obtained from a server; training the initial classification model based on the hybrid embedded feature to obtain a trained classification model; uploading the model parameters of the classification model to the server, so that the server performs model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0009] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps: extracting features from local training samples to obtain a first embedded feature; mixing the first embedded feature with a second embedded feature to obtain a hybrid embedded feature; wherein the second embedded feature is an embedded feature uploaded by another client during the local training of an initial classification model, obtained from a server; training the initial classification model based on the hybrid embedded feature to obtain a trained classification model; uploading the model parameters of the classification model to the server, so that the server performs model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0010] The aforementioned image processing method, apparatus, computer equipment, storage medium, and computer program product extract features from local training samples to obtain a first embedded feature, and then mix the first embedded feature with a second embedded feature to obtain a hybrid embedded feature. The second embedded feature is an embedded feature uploaded by other clients during the local training of an initial classification model, obtained from the server. Further, based on the hybrid embedded feature, the initial classification model is trained to obtain a trained classification model, and the model parameters of the classification model are uploaded to the server so that the server can aggregate the models based on the parameters to obtain an aggregated global classification model. This global classification model is used to classify the image to be processed. Since the second embedded feature is an embedded feature uploaded by other clients during the local training of the initial classification model, obtained from the server, the mixed embedded feature obtained by mixing the first and second embedded features can complete the multi-class sample feature space. This allows for training of a single-client classifier on a mixed distribution of multiple different feature spaces. The single-client classifier can learn label discrimination under the feature distribution of other client datasets during the training phase, avoiding client drift caused by being misled by local incomplete feature distributions. Therefore, the classification boundaries learned by the classifiers of each client are close to consistent, resulting in better generalization ability of the global classification model obtained by the server based on the aggregation of model parameters. In other words, the aggregated global classification model has more accurate classification ability, which can reduce the impact of non-independent and identically distributed data on the classification results while ensuring the privacy and security of the original data of the participants, thereby effectively improving the accuracy of image classification processing. Attached Figure Description

[0011] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0012] Figure 2This is a flowchart illustrating an image processing method in one embodiment;

[0013] Figure 3 This is a schematic diagram of the overall process of a medical image federated learning optimization algorithm based on cross-client feature space completion and alignment, provided in one embodiment.

[0014] Figure 4 This is a schematic diagram of a few-shot federated learning framework based on cross-client feature space completion and alignment provided in one embodiment;

[0015] Figure 5 This is a structural block diagram of an image processing device in one embodiment;

[0016] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that in the following description, the terms "first, second, and third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] The image processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Terminal 102, i.e., the client, extracts features from local training samples to obtain a first embedded feature, and then mixes the first embedded feature with a second embedded feature to obtain a hybrid embedded feature. The second embedded feature is obtained by terminal 102, i.e., the client, from the server 104, from embedded features uploaded by other clients during the local training of the initial classification model. Further, terminal 102, i.e., the client, trains the initial classification model based on the hybrid embedded feature to obtain a trained classification model, and uploads the model parameters of the classification model to server 104. Server 104 then aggregates the models based on the parameters to obtain an aggregated global classification model, which is used to classify the image to be processed. Server 104 can distribute the aggregated global classification model to various clients. For example, server 104 can distribute the aggregated global classification model to terminal 102 so that terminal 102 can classify the image to be processed using the global classification model to obtain the classification result corresponding to the image to be processed.

[0020] The terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, IoT device, or portable wearable device. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices, etc.

[0021] Server 104 can be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP).

[0022] In addition, server 104 can also be a server cluster consisting of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0023] Terminal 102 and server 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network, etc., and this application does not impose any restrictions.

[0024] In one embodiment, such as Figure 2 As shown, an image processing method is provided. This method can be executed by a server or a terminal alone, or by both a server and a terminal. This method can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0025] Step 202: Extract features from the local training samples to obtain the first embedded features.

[0026] In this context, local training samples refer to the training samples used by each node (different clients) when using federated learning (i.e., distributed model training). In other words, local training samples in this application refer to the (local) training sample data constructed by each client using local privacy data. In some cases, in order to ensure the privacy and security of the original data of the participants (i.e., different clients), local training samples between different clients cannot be shared.

[0027] In the context of distributed model training, the first embedded feature refers to the feature extracted by each node (different client) from its own local training samples. For example, node 1 (client 1) extracts feature A1 from its own local training sample S1, and node 2 (client 2) extracts feature A2 from its own local training sample S2. Then, feature A1 is the first embedded feature corresponding to local training sample S1 of node 1 (client 1), and similarly, feature A2 is the first embedded feature corresponding to local training sample S2 of node 2 (client 2).

[0028] Step 204: Mix the first embedded feature with the second embedded feature to obtain a mixed embedded feature; wherein, the second embedded feature is an embedded feature uploaded by other clients during the local training of the initial classification model obtained from the server.

[0029] In this application, the first and second embedding features are only used to distinguish the embedding features extracted by different nodes (different clients). For example, the first embedding feature in this application can be the embedding feature extracted by the current node (current client), and the second embedding feature can be the embedding feature extracted by other nodes (other clients) other than the current node (current client). That is, the second embedding feature in this application is the embedding feature uploaded to the server by other clients during the local training of the initial classification model. The current node (current client) can download the embedding features uploaded by other clients during the local training of the initial classification model from the server.

[0030] Hybrid embedding features refer to the distribution of complete embedding features constructed by mixing the first embedding feature extracted by the current client with the second embedding feature extracted by other clients. That is, the hybrid embedding features in this application include not only the embedding features of local training samples but also the embedding features of local training samples from other clients. It can be understood that the hybrid embedding features in this application are used to train the classifier in the initial classification model to optimize the classification performance of the classifier.

[0031] Other clients refer to clients other than the current client. For example, if the current client is terminal 1, other clients refer to other terminals besides terminal 1, such as terminal 2, terminal 3, etc.

[0032] Step 206: Based on the hybrid embedding features, train the initial classification model to obtain the trained classification model.

[0033] In this context, the initial classification model refers to a model that is not trained or has not completed training, while the classification model refers to a trained model. The classification models trained locally by each client in this application are used to aggregate and obtain a global classification model. The global classification model is used to classify medical images in the medical field to obtain the classification results corresponding to the medical images.

[0034] Step 208: Upload the model parameters of the classification model to the server so that the server can aggregate the models based on the model parameters to obtain the aggregated global classification model; the global classification model is used to classify the image to be processed.

[0035] In this context, the global classification model refers to a model obtained by aggregating the model parameters of pre-trained (local) classification models uploaded by each client (node). For example, ... Figure 3 The diagram shown illustrates the overall process of the medical image federated learning optimization algorithm based on cross-client feature space completion and alignment provided in this application. The global classification model in this application can be as follows: Figure 3The global classification model in the server shown.

[0036] Specifically, the image processing method provided in this application can be widely applied to various image classification tasks, such as social media, games, ticketing, and medical images. For example, given a medical image of a patient, the image processing method provided in this application can quickly and accurately classify and identify the medical image, obtaining the corresponding classification result. This is of great significance in the practical application of medical image classification tasks, such as lung nodule identification, breast cancer screening, and brain tumor detection. That is, terminal devices in different regions (different geographical locations) can interact with the information platform (or application). When users (operating objects) in different regions want to classify locally acquired images, they can open the application (APP) on their terminal by triggering an operation and enter the application's main page by selecting an operation. In other words, users can log in to the application (such as a medical image application) by triggering an operation. Furthermore, users can initiate an image classification and recognition request on the application's main page by triggering an operation. For example, on the main page of a medical imaging application displayed on the terminal, each user of the medical imaging application (such as a doctor) can view the specific content and related functional information on the main page. Each user of the medical imaging application can trigger an image classification and recognition request by clicking the "Classification and Recognition" control. The terminal then responds to the image classification and recognition request triggered by the user on the main page of the medical imaging application, calls a pre-trained global classification model for classifying the image to be processed, and classifies and recognizes the image to be processed through the global classification model. This allows for a quick and accurate determination of the classification result corresponding to the image to be processed, and the classification result can be visualized.

[0037] In training the global classification model for classifying images, a federated learning approach can be used. This involves distributed model training across multiple nodes (clients) with local data, exchanging model parameters or intermediate results without needing to collect local sample data. This effectively decouples the need for a centralized dataset. Each node (client) can extract features from its own locally constructed training samples to obtain a first embedding feature, which is then mixed with a second embedding feature to obtain a hybrid embedding feature. The second embedding feature is obtained by each node (client) from the server, based on embedding features uploaded by other clients during their local training of the initial classification model. Furthermore, each node (client) can train the initial classification model based on the hybrid embedding feature to obtain a trained classification model. Finally, each node (client) can upload the model parameters of the trained classification model to the server, allowing the server to aggregate the uploaded parameters to obtain an aggregated global classification model. The server then distributes the parameters of the aggregated global classification model back to each node (client) so that each node (client) can use the global classification model to classify its local images.

[0038] Let's take a medical image classification and processing scenario as an example to illustrate this. Figure 3 The diagram shown illustrates the overall workflow of the medical image federated learning optimization algorithm based on cross-client feature space completion and alignment provided in this application. Figure 3 As shown, the model structure of the classification model trained locally on each client and the global classification model in this application can include two parts: a feature extractor and a classifier, for example, such as Figure 3 The client-side locally trained classification model shown includes a feature extractor for extracting embedded features and a classifier for mapping the feature space to the label distribution space. That is, when training the global classification model for classifying the image to be processed, a federated learning approach can be used, which requires distributed model training across multiple nodes (clients) that possess local data, such as... Figure 3Taking client N as an example, client N can extract features from local training samples Sn (local privacy data) using the feature extractor in the initial classification model (local model), obtaining first embedded features Sn. Then, the SFMC module (cross-client feature space completion module) in the initial classification model (local model) mixes the locally extracted first embedded features with second embedded features extracted by other clients to obtain hybrid embedded features. The second embedded features can be embedded features uploaded by other clients from the server simultaneously with client N uploading the locally extracted first embedded features to the server. Furthermore, client N can use the hybrid embedded features to perform operations such as... Figure 3 The local initial classification model shown is used for training to obtain the trained classification model. For example, client N can process the mixed embedding features using the classifier in the local model to obtain the class labels of the mixed embedding features. Then, based on the class labels, global class features, and first embedding features, the target loss value is determined. Training stops when the target loss value meets a preset loss condition, thus obtaining the trained classification model. Alternatively, client N can determine the target loss value based on the global class features, the first embedding features, and the class labels output by the classifier in the local model, until the training iterations reach a preset number, thus obtaining the trained classification model. The global class features are distributed to each client by the server. These global class features guide each client during local training to place samples of the same class closer to their global geometric center, implicitly aligning semantically similar substructures in the feature distribution of each client. This mitigates the impact of non-independent and identically distributed data between different clients on classification accuracy.

[0039] In this embodiment, a first embedding feature is obtained by extracting features from local training samples, and the first embedding feature is mixed with a second embedding feature to obtain a hybrid embedding feature. The second embedding feature is an embedding feature uploaded by other clients during the local training of the initial classification model, obtained from the server. Further, the initial classification model is trained based on the hybrid embedding feature to obtain a trained classification model, and the model parameters of the classification model are uploaded to the server so that the server can aggregate the models based on the model parameters to obtain an aggregated global classification model. The global classification model is used to classify the image to be processed. Since the second embedded feature is an embedded feature uploaded by other clients during the local training of the initial classification model, obtained from the server, the mixed embedded feature obtained by mixing the first and second embedded features can complete the multi-class sample feature space. This allows for training of a single-client classifier on a mixed distribution of multiple different feature spaces. The single-client classifier can learn label discrimination under the feature distribution of other client datasets during the training phase, avoiding client drift caused by being misled by local incomplete feature distributions. Therefore, the classification boundaries learned by the classifiers of each client are close to consistent, resulting in better generalization ability of the global classification model obtained by the server based on the aggregation of model parameters. In other words, the aggregated global classification model has more accurate classification ability, which can reduce the impact of non-independent and identically distributed data on the classification results while ensuring the privacy and security of the original data of the participants, thereby effectively improving the accuracy of image classification processing.

[0040] In one embodiment, after extracting features from local training samples to obtain the first embedded features, the method further includes:

[0041] The first embedded feature is uploaded to the server so that the server can classify the first embedded feature and the second embedded feature to obtain a category feature set; based on the embedded features in each category feature set, the client class feature is determined; based on each client class feature, the global class feature is determined; the global class feature contains the embedded features of each category.

[0042] Receive global class characteristics sent by the server;

[0043] The step of training the initial classification model based on the hybrid embedding features to obtain the trained classification model includes:

[0044] The initial classification model is trained based on global class features and hybrid embedding features to obtain the trained classification model.

[0045] In this context, a category feature set refers to a collection of features that share the same category (label). That is, the embedded features uploaded to the server by each client carry labels. For example, node 1 (client 1) extracts features from its local training sample S1 to obtain embedded feature A1, and node 2 (client 2) extracts features from its local training sample S2 to obtain embedded feature A2. Node 1 (client 1) can then upload embedded feature A1 and its corresponding sample label S1 to the server. Similarly, node 2 (client 2) can upload embedded feature A2 and its corresponding sample label S2 to the server. Assuming that sample label S1 and sample label S2 belong to the same category (e.g., both S1 and S2 are 1), the server can classify the embedded features uploaded by each client according to the category of the sample labels, obtaining one or more category feature sets. For example, one category feature set might include embedded feature A2 and embedded feature A1.

[0046] Client-side class features refer to the updated class features (also called class prototypes) for each client, updated according to the client dimension and based on the cached feature sets of each category. For example, the server caches several batches of embedding vectors from multiple clients collected through communication, categorized by label. After all client data in a round has been uploaded, the server can perform a smooth aggregation of multiple batches of embedding vectors for the same sample category (i.e., category feature set) in the cache using an exponential moving average method. This yields the updated class prototype for each client, avoiding drastic changes in the geometric center of features when the model is unstable. In this application, the class features, also called class prototypes, refer to the geometric center (or mean) of features of the same category.

[0047] Global class features refer to the average aggregation of the updated class features (also known as class prototypes) for each client to obtain the aggregated global class features (i.e., global class prototypes). For example, after obtaining the updated class prototypes for each client, the server can further aggregate the geometric centers of features from different clients using a federated average method. In other words, based on the class prototypes for each client, the global class prototype (also known as global category prototypes) is maintained and updated using an exponential moving average method. It can be understood that the global class features (global class prototypes) in this application contain the embedding vectors for each category. For example, the global class features (global class prototypes) contain the embedding vector h1 for the first category with sample label 1, the embedding vector h2 for the second category with sample label 2, the embedding vector h3 for the third category with sample label 3, and so on, representing multiple categories. That is, the number of categories in this application is the same as the number of categories of sample labels carried by the embedding vectors uploaded by each client. For example, if the total number of categories of sample labels carried by the embedding vectors uploaded by each client is 10, then the number of categories contained in the global class features (global class prototypes) is also 10.

[0048] Specifically, let's take the scenario of medical image classification processing as an example for illustration. For example... Figure 3 As shown, the model structure of the classification model trained locally on each client and the global classification model in this application can include two parts: a feature extractor and a classifier, so as to... Figure 3 Taking client N as an example, client N can extract features from local training samples Sn (local privacy data) using the feature extractor in the initial classification model (localmodel), obtaining first embedded features Sn (local features). Then, the SFMC module (i.e., cross-client feature space completion module) in the initial classification model (localmodel) mixes the locally extracted first embedded features with second embedded features extracted by other clients to obtain mixed embedded features. The second embedded features can be embedded features uploaded by other clients from the server while client N uploads the locally extracted first embedded features to the server. Furthermore, client N can use the mixed embedded features to perform operations such as... Figure 3 The local initial classification model shown is trained until the preset number of training iterations is reached, at which point training stops to obtain the trained classification model.

[0049] For example, during the final round of gradient descent in local training, the original local privacy data undergoes processes such as... Figure 3The local initial classification model (local model) shown has multiple convolutional layers in its feature extractor. Client N can select a layer's features according to a preset selection strategy, unfold them into embedding vectors, record their sample labels, and package them with the locally trained feature extractor and classifier model parameters, then upload them to the server. Alternatively, client N can select the last layer's features, unfold them into embedding vectors, record their sample labels, and package them with the locally trained feature extractor and classifier model parameters, then upload them to the server. This allows the server to classify the first and second embedding features based on the sample label categories, obtaining category feature sets for different sample categories. Further, the server performs smooth aggregation of the embedding vectors in multiple batches using an exponential moving average method for each category feature set, thus obtaining the smoothed aggregation of the class features corresponding to each client (i.e., client class features). Further still, the server performs average aggregation of the geometric centers of different client features using a federated average method, i.e., an exponential moving average method, again averaging and aggregating each client class feature, thus obtaining the averaged aggregation of the global category features (i.e., global class features). The global class features contain the embedding features of each category (sample label category).

[0050] Furthermore, the server can distribute the updated global class features to each client, enabling each client to train its initial classification model locally based on the global class features and hybrid embedding features, thus obtaining the trained classification model. It is understood that the reason the server needs to distribute the updated global class features to each client in this embodiment is to guide the updating of the local feature distribution on each client through the global class features. In this approach, each client constructs a feature subspace for each category using the output of the feature extractor during local gradient descent training. Guided by the corresponding category prototypes, the feature extractor is trained by encouraging the embedding of sample features of the same category to be close to their global geometric center through the loss function. This implicitly aligns semantically similar substructures in the feature distribution of each client, effectively mitigating the impact of non-independent and identically distributed local features from multiple clients on the classifier's learning of the classification boundary. This, in turn, improves the local classifier's learning of the classification boundary in the global sample feature space, significantly optimizing the performance of the classification model in the federated learning system. Consequently, the aggregated global classification model also possesses more accurate classification capabilities. This approach ensures the privacy and security of the original data of the participants while reducing the impact of non-independent and identically distributed data on the classification results, thereby effectively improving the accuracy of image classification processing.

[0051] In one embodiment, the step of extracting features from local training samples to obtain first embedded features includes:

[0052] The first embedded feature is obtained by extracting features from the local training samples using the feature extractor in the initial classification model.

[0053] The step of training the initial classification model based on the global class features and the hybrid embedding features to obtain the trained classification model includes:

[0054] The classifier in the initial classification model is used to process the hybrid embedding features to obtain the category labels of the hybrid embedding features;

[0055] The target loss value is determined based on the category label, global class features, and first embedding features. Training stops when the target loss value meets a preset loss condition, thus obtaining the trained classification model; or,

[0056] The target loss value is determined based on the category label, global class features, and first embedding features. Training is stopped when the preset number of training iterations is reached, in order to obtain the trained classification model.

[0057] Specifically, let's take the scenario of medical image classification processing as an example for illustration. For example... Figure 3 As shown, the classification model trained locally on each client in this application includes: a feature extractor and a classifier, to... Figure 3 Taking client N as an example, client N can use the feature extractor in the initial classification model (local model) to extract features from the local training sample Sn (local privacy data), obtaining the first embedded features Sn (local features). Then, the SFMC module (i.e., cross-client feature space completion module) in the initial classification model (local model) mixes the locally extracted first embedded features with second embedded features extracted by other clients to obtain a mixed embedded feature. The second embedded feature can be embedded features uploaded by other clients from the server at the same time client N uploads the locally extracted first embedded feature to the server. Furthermore, client N can use the mixed embedded feature as input parameters to the classifier in the initial classification model (local model) for processing. That is, client N uses... Figure 3 The classifier in the initial classification model shown processes the hybrid embedding features and outputs the category label of the hybrid embedding features. Furthermore, the client N can determine the target loss value based on the category label output by the classifier, the global class features sent by the server, and the first embedding features extracted by the feature extractor. Training stops when the target loss value meets the preset loss condition (e.g., the target loss value is less than the preset loss value threshold) to obtain the trained local classification model.

[0058] Alternatively, client N can determine the target loss value using the class label output by the classifier, the global class features sent by the server, and the first embedded features extracted by the feature extractor, until training stops after reaching a preset number of training iterations (e.g., 100 iterations), thus obtaining a trained local classification model. This enhances the classifier's ability to distinguish between different categories of samples through random feature space completion by multiple clients, overcoming the limitations of low data volume and feature offset in local training. Furthermore, feature space alignment guided by category prototypes makes the data features of multiple clients tend to be identically distributed, significantly optimizing the performance of the classification model in the federated learning system. This results in a more accurate classification model after aggregation, ensuring the privacy and security of the original data of the participants while reducing the impact of non-independent and identically distributed data on the classification results, thereby effectively improving the accuracy of image classification processing.

[0059] In one embodiment, the step of determining the target loss value based on the category label, global class features, and first embedding features includes:

[0060] The first loss value is determined based on the category label;

[0061] The second loss value is determined based on the global class features and the first embedding features;

[0062] The target loss value is determined based on the first loss value and the second loss value.

[0063] Specifically, with Figure 3 Taking client N as an example, client N uses, for instance... Figure 3 The initial classification model shown in the diagram processes the hybrid embedding features and outputs the class labels for these features. Client N can then calculate a first loss value based on the class labels and a preset function, and a second loss value based on the global class features and the first embedding features. Finally, client N can calculate a target loss value based on the first and second loss values. Training stops when the target loss value meets a preset loss condition, resulting in the trained classification model. For example, client N can calculate the sum of the first and second loss values ​​and use this sum as the target loss value. This allows the global class features, uniformly distributed by the server, to propagate back and align the data features of each client, making the data features of each client (node) in the distributed model training more uniformly distributed. This alleviates the problem of inconsistent feature distribution during multi-client training, significantly optimizing the performance of the classification model in the federated learning system. Consequently, the aggregated global classification model also possesses more accurate classification capabilities and sufficient privacy and security.

[0064] In one embodiment, a target loss value is determined based on the category label, global class features, and first embedding features, and training is stopped when the preset number of training iterations is reached, to obtain the trained classification model, including:

[0065] The target loss value is determined based on the first loss value and the second loss value. Training continues until the preset number of training iterations is reached, at which point the training stops, in order to obtain the trained classification model.

[0066] Specifically, with Figure 3 Taking client N as an example, client N uses, for instance... Figure 3 The initial classification model shown in the diagram processes the hybrid embedding features and outputs the class labels for these features. Client N can then calculate a first loss value based on the class labels output by the classifier and a preset function, and a second loss value based on the global class features and the first embedding features. Finally, client N can calculate a target loss value based on the first and second loss values. Training continues until a preset number of training iterations (e.g., 100) is reached, resulting in the trained classification model. Client N can also calculate the sum of the first and second loss values ​​and use this sum as the target loss value. This allows the global class features, uniformly distributed by the server, to propagate back and align the data features of each client, making the data features of each client (node) in the distributed model training more uniformly distributed. This alleviates the problem of inconsistent feature distribution during multi-client training, significantly optimizing the performance of the classification model in the federated learning system. Consequently, the aggregated global classification model also possesses more accurate classification capabilities and sufficient privacy and security.

[0067] In one embodiment, the step of determining the second loss value based on the global class features and the first embedding features includes:

[0068] Determine the target category corresponding to the first embedded feature;

[0069] Determine the similarity between the first embedded feature and the embedded features of the target category in the global class features;

[0070] The second loss value is determined based on similarity.

[0071] Here, the target category refers to the category corresponding to the sample label carried by the first embedded feature. For example, if the sample label carried by the first embedded feature is S1=1, then the target category corresponding to the first embedded feature is S1=1 (for example, the first category when S1=1).

[0072] Specifically, with Figure 3 Taking client N as an example, client N uses, for instance... Figure 3The classifier in the initial classification model shown processes the hybrid embedding features and outputs the category labels of the hybrid embedding features. Client N can then calculate a first loss value based on the category labels output by the classifier and a preset function, and a second loss value based on the global class features and the first embedding features. For example, client N can determine the target category (i.e., the category corresponding to the sample label) corresponding to the first embedding feature based on the sample label carried by the first embedding feature. Furthermore, client N can search for the embedding features of the target category from the global class features sent by the server, calculate the similarity between the first embedding feature and the target category embedding features in the global class features, and determine the second loss value based on the similarity. For example, client N can calculate the cosine similarity between the first embedding feature and the target category embedding features in the global class features, and use the cosine similarity as the second loss value; that is, the higher the similarity between the first embedding feature and the target category embedding features in the global class features, the smaller the second loss value. Therefore, to avoid client-side feature drift caused by multiple rounds of training on the local training set in scenarios where features are not independent and identically distributed, thus affecting the classification performance and generalization of the aggregated global model, this application introduces a module for client-server collaboration to align feature distributions. This module, guided by the prototype of each category, aligns semantically identical feature substructures across multiple clients. This is more conducive to the local classifier learning the classification boundary in the global sample feature space, thereby implicitly aligning semantically similar substructures in the feature distributions of each client, making the overall distribution closer to an independent and identically distributed state. This effectively mitigates the negative impact of non-independent and identically distributed data on the training of the classification model, thus significantly optimizing the performance of the classification model in the federated learning system. Consequently, the aggregated global classification model also possesses more accurate classification capabilities and sufficient privacy and security.

[0073] In one embodiment, after extracting features from local training samples to obtain the first embedded features, the method further includes:

[0074] Upload the first embedded feature to the server and retrieve the second embedded feature from the server;

[0075] The system receives the model parameters of the global classification model from the server, performs classification processing on the image to be processed using the global classification model, and obtains the classification result corresponding to the image to be processed.

[0076] Specifically, such as Figure 4 The diagram shown is a schematic representation of the few-shot federated learning framework based on cross-client feature space completion and alignment provided in this application. Figure 3Taking client N as an example, if client N communicates and exchanges model parameters and feature data with the server after each round of global training, it will incur certain network communication overhead and server computational overhead. Therefore, this application embodiment attempts to reduce the number of communication attempts and proposes a few-shot federated learning training framework based on the above-mentioned optimization algorithm, such as... Figure 4 As shown in the diagram, each client's local model (i.e., the local classification model) first trains multiple rounds on local data. After the local features output by the feature extractor in the local model stabilize, an interaction between the client and server occurs: each client uploads relevant parameters of its local model for federated averaging model aggregation; each client uploads features from multiple batches of local data output in the last round of local training to the server, which exchanges and distributes them, and aggregates the geometric centers (i.e., global class features) of each category's feature substructure for each client to align with. Each client then performs a second phase of local model training, performing multiple rounds of gradient descent training on the complete feature space composed of local sample features and other client-supplemented features. Simultaneously, category-guided feature geometric centers are used to assist the feature extractor training. After multiple rounds of local training in the second phase, the above client-server communication process can be repeated for further local training, helping the client model learn more global knowledge. Finally, after all local training on each client is complete, a final communication occurs. The client uploads the model parameters of its classification model to the server. The server then integrates the predicted outputs of multiple local models to obtain a global classification model. The server then distributes the parameters of this global classification model to each client, enabling them to classify the image to be processed using the global classification model provided by the server, thus obtaining the corresponding classification result. This significantly reduces the communication overhead in federated learning by increasing the number of local training rounds and decreasing the frequency of model aggregation and prototype updates. It achieves this by ensuring the privacy and security of the original data from each client while improving the efficiency of model training and the classification accuracy of the global model, all within limited computational resources.

[0077] In one embodiment, after receiving model parameters of a global classification model from a server, classifying the image to be processed using the global classification model, and obtaining the classification result corresponding to the image to be processed, the method further includes:

[0078] Feature extraction is performed on the local training samples to obtain the first embedded features;

[0079] The first embedding feature and the second embedding feature are blended to obtain the hybrid embedding feature; wherein, the second embedding feature is the embedding feature uploaded by other clients during the local training of the initial classification model obtained from the server;

[0080] Based on hybrid embedding features, the global classification model is iteratively trained to obtain the trained global classification model;

[0081] The model parameters of the trained global classification model are uploaded to the server so that the server can aggregate the models based on the parameters to obtain an updated global classification model.

[0082] Updating the global classification model refers to obtaining a new global classification model through iterative training based on the previous global classification model.

[0083] Specifically, with Figure 3 Taking client N as an example, client N receives the model parameters of the global classification model from the server. After classifying the image to be processed using the global classification model and obtaining the classification result, in the next round of distributed model training, client N can use the global classification model as its local classification model. Specifically, client N extracts features from the local training samples using the feature extractor in the global classification model to obtain a first embedded feature. This first embedded feature is then mixed with a second embedded feature to obtain a hybrid embedded feature. The second embedded feature is the embedded feature uploaded by other clients during the initial local training of their classification models, obtained from the server. Furthermore, client N can iteratively train the global classification model based on the hybrid embedded feature to obtain a trained global classification model. The model parameters of the trained global classification model are then uploaded to the server, allowing the server to aggregate the models based on these parameters, resulting in an updated global classification model. This significantly reduces the communication overhead in federated learning by increasing the number of local training rounds and reducing the frequency of model aggregation and prototype updates. This achieves both the protection of the privacy and security of the original data from each client and improved model training efficiency and global model classification accuracy within limited computational overhead.

[0084] This application also provides an application scenario in which the above-described image processing method is applied. The method provided in this application embodiment can be applied to various personalized video or image classification scenarios. The following uses a medical image classification task as an example to illustrate the image processing method provided in this application embodiment.

[0085] Leveraging advanced deep learning and computer vision technologies, deep image classification models can extract valuable semantic information from image datasets. Deep learning models rely on large-scale, diverse, and high-quality samples to train high-performance models, but collecting large-scale labeled datasets is both challenging and costly. In certain fields, such as finance and healthcare, establishing a centralized, multi-center dataset may face various constraints related to law, privacy, technology, and data ownership, especially the need to comply with international laws and regulations. Furthermore, training models using data from only one specific population may result in models that are not applicable to other populations.

[0086] One approach to overcome these obstacles is to employ federated learning, a method that decouples the need for a centralized dataset by training the model in a distributed manner across multiple nodes that possess local data. This is achieved by exchanging model parameters or intermediate results, without requiring the collection of local sample data. However, the heterogeneity of data between different clients often presents a significant challenge to federated learning. When dealing with real-world, non-independent, identically distributed datasets, federated learning aggregation models frequently experience performance degradation and slower convergence.

[0087] Traditional federated learning methods have focused on exploring the problem of non-independent and identically distributed labels in natural images, especially on small and simple datasets such as MNIST and CIFAR10. However, they lack applicability in heterogeneous feature spaces and exhibit significant performance degradation in medical image classification tasks.

[0088] Medical image classification tasks are of great significance in practical applications, such as lung nodule identification, breast cancer screening, and brain tumor detection. Researching efficient and stable federated learning algorithms applicable to real-world multi-regional, multi-center medical data distributions with significantly inconsistent sample feature distributions has high practical value. Therefore, this application proposes a practical federated learning algorithm based on feature space completion and alignment, aiming to solve the problem of non-IID medical image classification in multi-center datasets.

[0089] Many works have attempted to explore federated learning optimization algorithms to mitigate the adverse effects of non-IID data, focusing on calibration strategies in model aggregation or client update steps. However, traditional methods mainly consider imbalanced client sample numbers and inconsistent label distributions. While they perform well in natural image classification tasks, they do not coordinate and calibrate multi-client training at the feature level. Consequently, they often lack robustness in medical images, where feature distribution inconsistencies are more pronounced, and lack optimization specifically for medical imaging tasks.

[0090] Problems with traditional methods include:

[0091] 1. The most important and crucial issue:

[0092] In the field of medical applications, image data collected by different hospitals or equipment exhibit significant differences in image quality, contrast, resolution, and shooting angle. Furthermore, different regions have varying patient structures and clinical practices. Therefore, multi-client datasets participating in federated learning exhibit significant non-independent identically distributed (Non-IID) characteristics in their sample features. This severely degrades the performance of existing federated learning algorithms, leading to difficulties in convergence of classification models and a decrease in classification accuracy.

[0093] 2. Secondary issues:

[0094] (1) In medical diagnostic tasks, the data available to a single client is often very limited, making it difficult for the local classification model to fully learn the classification features and resulting in insufficient stability.

[0095] (2) Traditional federated learning algorithms that optimize heterogeneous feature spaces often use generative techniques, i.e., the client trains a generative model locally to generate supplementary samples (i.e., generate pseudo samples), and transmits generative models with large parameters or a large number of generated pseudo samples between clients. This paradigm is limited by the client's computing power and network transmission bandwidth, making it difficult to deploy and apply in practice. Furthermore, although the existing best generative model performs well in natural images, its generative effect in medical images is not good enough.

[0096] Therefore, to address the aforementioned issues, this application proposes a federated learning algorithm for the widespread problem of non-independent and identically distributed features across multiple clients in federated learning for medical image classification. This algorithm optimizes the federated learning process through cross-client feature space sampling, completion, and alignment. The overall architecture of the technical solution is shown in the diagram below. Figure 3 As shown in the diagram, this technical solution proposes a highly robust and widely applicable federated learning optimization algorithm that directly addresses multi-client feature non-IID scenarios. This method enhances the classifier's ability to distinguish between different categories of samples through random completion of the multi-client feature space, overcoming the limitations of low data volume and feature offset in local training. Furthermore, by using category prototype-guided feature space alignment, it brings the multi-client data features closer to a uniform distribution, thereby significantly optimizing the performance of the classification model in the federated learning system. Experiments were conducted on multiple medical imaging task datasets, including real-world multi-regional non-IID datasets. The method achieved significant performance improvements compared to baseline methods while maintaining sufficient privacy and security. Within the framework of this technical solution, the number of communication attempts can also be reduced to effectively alleviate the communication pressure on the federated learning system.

[0097] like Figure 3The diagram illustrates a medical image federated learning optimization algorithm based on cross-client feature space completion and alignment. This application's technical solution addresses the insufficient classification performance of aggregated models by optimizing the distribution of multi-client data at the feature level. Specifically, it includes the following two modules:

[0098] (1) Feature Space Completion Module: The purpose is to enable a client's classification model to learn more fully the classification targets under different feature spaces of other clients when features are heterogeneously distributed. This module constructs an extended hybrid feature space by combining the embedding vectors of local privacy data with the embedding vectors randomly sampled from multiple other clients, so that the local classifier can be trained on the completed feature distribution, thereby enhancing the model's accurate discrimination ability under different domain features.

[0099] (2) Prototype-guided feature space alignment module: To avoid client-side feature drift caused by multiple rounds of training on the local training set in scenarios where features are not independent and identically distributed, thus affecting the classification performance and generalization of the aggregated global model, a module is introduced to align feature distributions in collaboration between the client and the server. This module aligns feature substructures with the same semantics across multiple clients under the guidance of the prototype of each category, which is more conducive to the local classifier learning the classification boundaries in the global sample feature space.

[0100] The technical solution proposed in this application aims to mitigate the performance degradation of federated learning models caused by non-independent and identically distributed data at the feature level. This solution splits the local classification neural network on the client side into two parts: a feature extractor with ResNet as its backbone and a multilayer perceptron classifier. The former maps the original sample input to the feature space, while the latter further maps the feature space to the label distribution space. Unlike traditional federated averaging algorithms that train a complete local classification model using only the local dataset of each client, the solution provided in this application obtains additional feature embeddings from multiple clients to fine-tune the classification head of the local model. That is, through communication within the federated learning system, a complete multi-class sample feature space is completed, allowing direct training of the single-client classifier on a mixed distribution of multiple different feature spaces. Simultaneously, it attempts to adjust the feature subspaces of multiple clients, narrowing the Hausdorff distance in the global space for feature subsets from different clients but with the same class label. In other words, through federated training of the feature extractor, the feature spaces of multiple clients are adjusted to make the overall model closer to an independent and identically distributed state.

[0101] In terms of specific implementation, this technical solution can be divided into two modules:

[0102] Cross-client feature space completion module:

[0103] During the final gradient descent round of local training, the original privacy data passes through multiple convolutional layers of the feature extractor. Features from one layer can be selected and expanded into embedding vectors, with their sample labels recorded. These vectors, along with the model parameters of the locally trained feature extractor and classifier, are packaged and uploaded to the server, where they are forwarded. Simultaneously, the client downloads several batches of labeled feature data from other clients from the server. This supplements the limited feature space of the local training samples, restoring the dimensions originally used in the model's computation. These features are then combined with the local data to form a complete feature space for the latter part of the classification neural network training, optimizing the classification objective. This approach overcomes the limited learning space of the local model within the federated averaging framework, enabling it to learn label discrimination under the feature distributions of other client datasets during training. This avoids client drift caused by misleading local incomplete feature distributions. Consequently, the classification boundaries learned by the classifiers on each client are nearly consistent, resulting in a globally effective model with better generalization capabilities and better performance in tests.

[0104] Prototype-guided feature alignment module:

[0105] A simple approach to address feature offset is to directly align the feature distributions of different clients to the same distribution. However, this can impair the classification performance of the feature extractor. Therefore, this algorithm addresses the problem more finely from the perspective of class prototypes (the geometric centers of features within the same class). The specific process is as follows: The server caches several batches of embedding vectors collected from multiple clients by label classification. After all client data uploads in this round are completed, for samples of the same class in the cache, a smooth aggregation of multiple batches of embedding vectors is performed using an exponential moving average (updating the class prototype for each client) to avoid drastic changes in the feature geometric centers when the model is unstable. Then, a federated average is used to aggregate the feature geometric centers of different clients, and the global class prototype, containing the embedding vectors for each class, is maintained and updated using an exponential moving average. Finally, the class prototype is distributed to each client to guide the update of the client's local feature distribution. During local gradient descent training on the client side, the output of the feature extractor is used to construct feature subspaces for each category. The feature extractor is trained under the guidance of its corresponding category prototype. The loss function encourages the embedding of sample features of the same category to be close to its global geometric center, thereby implicitly aligning semantically similar substructures in the feature distribution of each client and mitigating the impact of non-independent and identically distributed data on classification.

[0106] Few-Shot Federated Learning Solution:

[0107] After each round of global training, model parameters and feature data are exchanged, which incurs network communication overhead and server computational overhead. Therefore, this application's technical solution also attempts to reduce the number of communication operations, proposing a few-shot federated learning training framework based on the aforementioned optimized algorithm, such as... Figure 4 As shown in the diagram, each client's local model is first trained multiple times on local data. After the local features output by the feature extractor stabilize, an interaction between the client and server occurs: each client uploads its local model for federated averaging model aggregation; each client uploads features from multiple batches of local data output during the last round of local training to the server, which then exchanges and distributes these features, aggregating the geometric centers of each category's feature substructure for client alignment. Each client then performs a second phase of local model training, conducting multiple rounds of gradient descent training on the complete feature space composed of local sample features and supplementary features from other clients. Simultaneously, category-guided feature geometric centers assist the feature extractor's training. After multiple rounds of local training in the second phase, the client-server communication process can be repeated for further local training, helping the client model learn more global knowledge. Finally, after all local training by each client is complete, a final communication occurs: the client uploads its model to the server, which can then integrate the predicted outputs of multiple models. By increasing the number of local training rounds and reducing the frequency of model aggregation and prototype updates, the communication overhead in federated learning is significantly reduced.

[0108] The beneficial effects of the technical solution in this application include:

[0109] 1. The proposed method, which combines multi-client feature space random completion with a class prototype-guided feature space alignment module, effectively alleviates the heterogeneous data feature distribution problem widely existing in medical applications, thereby improving the performance of federated learning. Experiments on multiple medical image datasets have shown that this method outperforms state-of-the-art federated learning algorithms, and demonstrates significant performance improvements on two real-world multi-center distributed datasets.

[0110] 2. The few-shot federated learning training framework with the above-mentioned optimization method proposed in this case shows that the application of the optimization method can greatly reduce the communication overhead in the federated learning framework while ensuring the classification effect of the global model.

[0111] 3. The method proposed in this case has been verified by multiple indicators under image reconstruction attacks, and can ensure the privacy and security of the original data of the participants.

[0112] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0113] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0114] In one embodiment, such as Figure 5 As shown, an image processing apparatus is provided, including: an extraction module 502, a mixing module 504, a training module 506, and an uploading module 508, wherein:

[0115] The extraction module 502 is used to extract features from local training samples to obtain the first embedded features.

[0116] The mixing module 504 is used to mix the first embedded feature with the second embedded feature to obtain a mixed embedded feature; wherein the second embedded feature is an embedded feature uploaded by other clients during the local training of the initial classification model obtained from the server.

[0117] Training module 506 is used to train the initial classification model based on the hybrid embedding features to obtain the trained classification model.

[0118] The upload module 508 is used to upload the model parameters of the classification model to the server, so that the server can perform model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

[0119] In one embodiment, the uploading module is further configured to upload the first embedded feature to the server, so that the server classifies the first embedded feature and the second embedded feature to obtain a category feature set; determine client class features based on the embedded features in each category feature set; determine global class features based on each client class feature; the global class features include the embedded features of each category; the apparatus further includes: a receiving module, configured to receive the global class features sent by the server; the training module is further configured to train the initial classification model based on the global class features and the hybrid embedded features to obtain the trained classification model.

[0120] In one embodiment, the extraction module is further configured to extract features from local training samples using the feature extractor in the initial classification model to obtain a first embedded feature; the training module is further configured to process the hybrid embedded feature using the classifier in the initial classification model to obtain a category label for the hybrid embedded feature; determine a target loss value based on the category label, the global class feature, and the first embedded feature, and stop training when the target loss value meets a preset loss condition, so as to obtain the trained classification model; or, determine a target loss value based on the category label, the global class feature, and the first embedded feature, and stop training when the number of training iterations reaches a preset number of training iterations, so as to obtain the trained classification model.

[0121] In one embodiment, the apparatus further includes: a determining module, configured to determine a first loss value based on the category label; determine a second loss value based on the global class feature and the first embedded feature; and determine a target loss value based on the first loss value and the second loss value.

[0122] In one embodiment, the determining module is further configured to determine the target category corresponding to the first embedded feature; determine the similarity between the first embedded feature and the embedded feature of the target category in the global class features; and determine the second loss value based on the similarity.

[0123] In one embodiment, the uploading module is further configured to upload the first embedded feature to the server and obtain the second embedded feature from the server; the device further includes: a receiving module, configured to receive the model parameters of the global classification model sent by the server, and perform classification processing on the image to be processed through the global classification model to obtain the classification result corresponding to the image to be processed.

[0124] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0125] In one embodiment, a computer device is provided, which may be a terminal or a server. In this embodiment, the computer device is described as a terminal, and its internal structure diagram is as follows. Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0126] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0128] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0129] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0131] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Feature extraction is performed on the local training samples to obtain the first embedded features; The first embedding feature and the second embedding feature are mixed to obtain a hybrid embedding feature; wherein, the second embedding feature is an embedding feature uploaded by other clients during the local training of the initial classification model obtained from the server; Based on the hybrid embedding features, the initial classification model is trained to obtain the trained classification model; The model parameters of the classification model are uploaded to the server, so that the server can aggregate the models based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

2. The method according to claim 1, characterized in that, After extracting features from the local training samples to obtain the first embedded features, the method further includes: The first embedded feature is uploaded to the server, so that the server classifies the first embedded feature and the second embedded feature to obtain a category feature set; based on the embedded features in each category feature set, a client class feature is determined; based on each client class feature, a global class feature is determined; the global class feature contains the embedded features of each category. Receive the global class feature sent by the server; The step of training the initial classification model based on the hybrid embedding features to obtain the trained classification model includes: Based on the global class features and the hybrid embedding features, the initial classification model is trained to obtain the trained classification model.

3. The method according to claim 2, characterized in that, The step of extracting features from local training samples to obtain the first embedded features includes: The feature extractor in the initial classification model is used to extract features from the local training samples to obtain the first embedded features. The step of training the initial classification model based on the global class features and the hybrid embedding features to obtain the trained classification model includes: The classifier in the initial classification model processes the hybrid embedding features to obtain the category label of the hybrid embedding features; The target loss value is determined based on the category label, the global class feature, and the first embedded feature. Training stops when the target loss value meets a preset loss condition, thus obtaining the trained classification model; or, The target loss value is determined based on the category label, the global class feature, and the first embedded feature. Training is stopped when the preset number of training iterations is reached, so as to obtain the trained classification model.

4. The method according to claim 3, characterized in that, Determining the target loss value based on the category label, the global class feature, and the first embedded feature includes: A first loss value is determined based on the category label; A second loss value is determined based on the global class feature and the first embedded feature; The target loss value is determined based on the first loss value and the second loss value.

5. The method according to claim 4, characterized in that, Determining the second loss value based on the global class feature and the first embedded feature includes: Determine the target category corresponding to the first embedded feature; Determine the similarity between the first embedded feature and the embedded feature of the target category in the global class features; The second loss value is determined based on the similarity.

6. The method according to claim 1, characterized in that, After extracting features from the local training samples to obtain the first embedded features, the method further includes: The first embedded feature is uploaded to the server, and the second embedded feature is obtained from the server; The system receives the model parameters of the global classification model sent by the server, performs classification processing on the image to be processed using the global classification model, and obtains the classification result corresponding to the image to be processed.

7. An image processing apparatus, characterized in that, The device includes: The extraction module is used to extract features from local training samples to obtain the first embedded features; A mixing module is used to mix the first embedded feature with the second embedded feature to obtain a mixed embedded feature; wherein the second embedded feature is an embedded feature uploaded by other clients during the local training of the initial classification model obtained from the server. The training module is used to train the initial classification model based on the hybrid embedding features to obtain the trained classification model. The upload module is used to upload the model parameters of the classification model to the server, so that the server can perform model aggregation based on the model parameters to obtain an aggregated global classification model; the global classification model is used to classify the image to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images