Image recognition method based on model distillation and federated learning
By constructing a multimodal semantic embedding space within a federated learning framework and utilizing a semantic anchoring mechanism for model distillation, the technical challenges of cross-modal models in image recognition tasks under resource constraints are addressed. This enables knowledge transfer between teacher and student models in image recognition tasks within a federated learning environment, reducing computational and storage overhead and enhancing learning stability and privacy protection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEBEI UNIV OF TECH
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-15
AI Technical Summary
Existing cross-modal models struggle to effectively transfer cross-modal semantic relationship information in federated learning environments, resulting in high demands for computing and storage resources, increased communication burden, insufficient privacy protection, and difficulty in deployment on resource-constrained edge devices.
We employ a model distillation method based on cross-modal relationships. Under the federated learning framework, we construct a multimodal semantic embedding space through a semantic anchoring mechanism to achieve knowledge transfer between teacher and student models, reduce computational and storage overhead, and deploy student models through federated learning.
It enables efficient and low-power intelligent image perception and analysis on resource-constrained edge devices, reduces communication overhead, enhances privacy protection and deployment flexibility, and improves the learning stability and generalization performance of student models.
Smart Images
Figure CN122049484A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition method based on model distillation and federated learning. Background Technology
[0002] The rapid development of artificial intelligence technology has enabled cross-modal learning methods that integrate multimodal information such as images and text to effectively improve the model's ability to understand high-level semantic information. However, existing cross-modal models are usually complex in structure and have a large number of parameters, requiring high computing and storage resources, making them difficult to deploy directly on edge devices with limited computing power and energy consumption.
[0003] In real-world applications, large amounts of image data are scattered across different terminals or edge nodes. Due to limitations in privacy protection, data security, and communication bandwidth, it is difficult to centrally upload the raw data to the cloud for unified training. Furthermore, existing federated learning methods mostly focus on single-modal tasks, typically using parameter averaging for model aggregation, which makes it difficult to fully explore cross-modal semantic relationships and provides insufficient support for model compression and edge deployment.
[0004] Knowledge distillation (KD), as a model compression and performance transfer technique, transfers knowledge from a large-scale teacher model to a lightweight student model, enabling the student model to achieve near-teacher model performance while maintaining low computational complexity. Existing research largely focuses on feature distillation or output distillation, directly constraining the student model's fit to intermediate features or predictions from the teacher model. However, these methods typically require introducing the complete teacher model or additional modal data on the client side, increasing computational and storage overhead and compromising privacy and system scalability.
[0005] For cross-modal tasks, existing methods attempt to achieve semantic alignment between images and text through joint embedding spaces, but most rely on centralized training architectures, which are difficult to adapt to federated learning environments. Furthermore, directly distilling teacher model parameters or features can easily increase the communication burden in federated scenarios, and the model's convergence performance is unstable under heterogeneous client conditions. Therefore, how to effectively transfer cross-modal semantic knowledge to lightweight student models within a federated learning framework, without introducing additional modal inputs or directly fitting teacher model parameters, remains a pressing technical problem to be solved. Summary of the Invention
[0006] This invention primarily addresses the heterogeneity problem of multimodal teacher and student model distillation within the current federated learning framework and the challenges of cross-modal processing in image recognition tasks. It proposes an image recognition method based on model distillation and federated learning. By employing a cross-modal model distillation method within the federated learning framework, knowledge transfer between heterogeneous models is achieved, reducing the computational and storage overhead of edge devices. Simultaneously, by deploying the aggregated lightweight student model on edge devices, efficient and low-power intelligent image perception and analysis are realized, making it suitable for edge intelligence and edge-cloud collaborative applications in resource-constrained scenarios.
[0007] This invention provides an image recognition method based on model distillation and federated learning, comprising the following processes: Step 1: The edge device acquires the image classification dataset; Step 2: Obtain the teacher model and generate semantic anchors and image embedding features for each image in the image classification dataset using the teacher model; and use the distances between multiple image embedding features and their corresponding semantic anchors to form a semantic relationship distribution. Step 3: The end-side device establishes a student model; Step 4: Use the image classification dataset to train the student model under the federated framework. During training, enable the distillation mechanism based on semantic anchors to obtain the trained student model. Step 5: Obtain the image input to be recognized and perform image recognition using a converged student model.
[0008] Furthermore, the edge device includes an image acquisition module, an image preprocessing module, a sensing and computing integrated chip, and a data storage module.
[0009] Furthermore, the teacher model includes an image encoder and a text encoder; The image encoder is used to extract features from the input image and generate image embedding features with global semantic expression capabilities; The text encoder is used to encode the input text labels and generate text embedding features that are located in the same multimodal semantic embedding space as the image embedding features by modeling the words and their contextual relationships. The image encoder and text encoder construct a shared multimodal semantic embedding space through joint training.
[0010] Furthermore, in step 2, for the text label of each image data in the image classification dataset, a text description is generated using the text encoder in the teacher model; the text labels of multiple image data in the image classification dataset are used to form multiple text descriptions, resulting in a set of text descriptions. In the text description set, multiple semantic anchors are manually determined and encoded to obtain the corresponding semantic anchor set; For image input in the image classification dataset, the image encoder in the teacher model is used to extract image embedding features and calculate the distance between the image embedding features and the semantic anchors; the distances between multiple image embedding features and their corresponding semantic anchors form a semantic relationship distribution.
[0011] Furthermore, the student model employs a convolutional neural network, comprising: a backbone network, a first convolutional layer, a second convolutional layer, a third convolutional layer, a pooling layer, a fully connected layer, and an output layer; The backbone network adopts the MobileNetV3 structure in convolutional neural networks, and the MobileNetV3 structure contains at least one dynamic block. The first convolutional layer uses a 5×5 convolutional kernel; Both the second and third convolutional layers employ depthwise separable convolutional structures. The pooling layer employs a downsampling structure; The output layer is set as a linear mapping layer, which is used to map the image features extracted by the student model to the multimodal semantic embedding space shared with the teacher model.
[0012] Furthermore, the student model is trained using a model distillation method. During the distillation training process, the student model learns the structured knowledge of the teacher model in the multimodal semantic embedding space by aligning the semantic relation distribution in the multimodal semantic embedding space, thereby obtaining cross-modal semantic expression capabilities using only the image modality.
[0013] Furthermore, the parameters of the student model include the convolutional layer weights. Convolutional layer bias Feature projection layer weights Feature projection layer bias Student model output feature dimensions Teacher model embedding spatial dimension Semantic relation: distillation temperature Number of semantic anchors Input dimension is Output dimension is .
[0014] Furthermore, step 4 includes steps 401 to 406: Step 401: During training, the images in the image classification dataset are used as the image input of the teacher model. Based on the text labels corresponding to the image input, a text description consistent with the semantics of the text labels is constructed, and semantic anchors are determined according to the text description. Step 402: Based on semantic anchors, the model distillation method is used to calculate the semantic relation distribution of the student model in the multimodal semantic embedding space; Step 403: Calculate the semantic relation distillation loss of the student model and output the total loss of the student model; The formula for the semantic relation distillation loss of the student model output is: ; ; ; in, The loss represents the semantic relation distillation loss of the student model output, expressed as the probability distribution of the teacher model in the multimodal semantic embedding space. Probability distribution of the student model in the multimodal semantic embedding space The divergence was calculated to obtain the result. The entropy of the teacher distribution; optimization is equivalent to minimizing the cross entropy. , Indicates the number of semantic anchors; The total loss of the student model is: ; in, This represents the total loss of the student model. This indicates the loss caused by hard labels. This indicates the proportion of total loss caused by hard labels; Step 404: Calculate the uncertainty of the student model based on the semantic relation distribution of the student model in the multimodal semantic embedding space; When the uncertainty of the student model is greater than a preset threshold, hard labels provided by the image classification dataset are introduced to strengthen the training of the student model and update the parameters of the student model; when the uncertainty is lower than the threshold, the parameters of the student model are updated based on semantic relation distillation loss. The formula for calculating the uncertainty of the student model: ; in, Indicates sample The prediction uncertainty Indicates the number of semantic anchors. Represents the student model for the first Predicted probability of each semantic anchor point; Step 405: Perform federated learning on the parameters of the student model from multiple end devices to generate updated global student model parameters and distribute them to each end device. Step 406: Following the process from Steps 401 to 405, perform multiple rounds of federated learning training to obtain a converged student model.
[0015] Furthermore, in step 405, the federated learning aggregation adopts a weighted average strategy, and the aggregation process can be represented as follows: ; in, Indicates the first The global student model weights obtained after round-robin communication Indicates the first k The weights of the local student model trained by each end device in the t-th round of communication. Indicates the first k The number of data samples used by each edge device for local training. This represents the total number of edge devices participating in federated learning; The parameters of the federated learning framework include the number of federated learning communication rounds. Number of local training rounds on the end side Number of semantic anchors Semantic relation: distillation temperature Hard label supervision loss weight The percentage of clients participating in the calculation of sample entropy .
[0016] This invention provides an image recognition method based on model distillation and federated learning, which has the following advantages: This invention introduces a semantic anchoring mechanism to construct a semantic relationship network between images and text in a multimodal semantic embedding space. This enables the teacher model to express its high-level cross-modal semantic knowledge in the form of semantic relationship distribution, thereby realizing cross-modal and cross-structural knowledge distillation between the teacher model and the student model. This avoids direct dependence on the internal structure and parameters of the teacher model and improves the universality and scalability of the distillation process.
[0017] This invention employs a cross-modal distillation method, which, within the federated learning framework, only requires the transmission of semantic anchor information and student model parameters, without needing to transmit original image data and teacher model parameters. This effectively reduces the communication overhead between the end device and the server, alleviates the problem of limited communication bandwidth in federated learning, and simultaneously enhances the system's privacy protection capabilities and deployment flexibility.
[0018] To quantify the learning progress of the student model during the distillation process, this invention introduces an uncertainty evaluation mechanism based on semantic relation distribution. By quantifying the prediction uncertainty of the student model in the semantic anchor space, it adaptively determines whether to introduce hard labels for reinforcement training, thereby achieving dynamic switching between soft-label distillation and hard-label supervision. This mechanism effectively improves the learning stability, generalization performance, and robustness of the student model in complex scenarios, further enhancing the practical value of the overall image recognition method. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the implementation of the image recognition method based on model distillation and federated learning provided by this invention. Figure 2 This is a schematic diagram of the image recognition method based on model distillation and federated learning provided by the present invention; Figure 3 This is a deployment diagram of the image recognition method based on model distillation and federated learning provided by the present invention. Detailed Implementation
[0020] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.
[0021] like Figure 1-2 As shown in the figure, an image recognition method based on model distillation and federated learning provided by an embodiment of the present invention includes the following process: Step 1: The edge device acquires the image classification dataset.
[0022] The image data in the image classification dataset has obvious uniform features, and the images are labeled with text, making it suitable for image classification tasks. These uniform features reflect the image category, and the category is labeled using text tags.
[0023] Image data can be acquired through intelligent image sensors.
[0024] Step 2: Obtain the teacher model and generate semantic anchors and image embedding features for each image in the image classification dataset using the teacher model; and form a semantic relationship distribution by using the distances between multiple image embedding features and their corresponding semantic anchors.
[0025] The teacher model includes an image encoder and a text encoder, both based on a Transformer architecture. The image encoder extracts features from the input image, generating image embedding features with global semantic representation capabilities, which are represented as vectors. The text encoder encodes input text labels, generating text embedding features within the same multimodal semantic embedding space as the image embedding features by modeling words and their contextual relationships. These text embedding features are vector-based descriptions of the text. The image encoder and text encoder jointly construct a shared multimodal semantic embedding space, ensuring comparability and semantic consistency between the image and text embedding features within this space. This provides high-level semantic guidance for subsequent knowledge distillation based on semantic relationships.
[0026] For each image data text label in the image classification dataset, a text description is generated using the text encoder in the teacher model; the text labels of multiple image data in the image classification dataset are used to form multiple text descriptions, resulting in a set of text descriptions. In the text description set, multiple semantic anchors are manually determined and encoded to obtain a corresponding semantic anchor set, which is then fixed as the semantic anchor space coordinate system in the subsequent training process. The semantic anchors are text descriptions representing semantic categories, such as "an image about a certain category." These semantic anchors remain unchanged throughout the distillation training process and do not participate in updating the student model parameters. The text descriptions can be generated in natural language and are used to represent the high-level semantic meaning of each category. Different text labels correspond to different text descriptions, and these text descriptions are distinguishable in the multimodal semantic embedding space.
[0027] For image input in the image classification dataset, image embedding features are extracted using the image encoder in the teacher model, and the distance between the image embedding features and semantic anchors is calculated. The distances between multiple image embedding features and their corresponding semantic anchors form a semantic relationship distribution. The set of semantic anchors is constructed by the edge device and uploaded to the cloud server.
[0028] The edge device of this invention consists of multiple modules, which simultaneously perform data acquisition and training. For example... Figure 3 As shown, the edge device includes an image acquisition module, an image preprocessing module, a sensing and computing integrated chip, a data storage module, etc. This edge device is commonly an FPGA.
[0029] Step 3: The end-side device establishes a student model.
[0030] The student model employs a convolutional neural network, comprising: a backbone network, a first convolutional layer, a second convolutional layer, a third convolutional layer, a pooling layer, a fully connected layer, and an output layer.
[0031] The backbone network adopts the MobileNetV3 structure from convolutional neural networks as a lightweight feature extraction backbone network. The MobileNetV3 structure contains at least one dynamic block, which achieves a dynamic balance between network expressive power and computational complexity by introducing adjustable channel widths and non-linear activation functions.
[0032] By using MobileNetV3 as the backbone network, the parameter size and computational load of the student model are effectively reduced while ensuring feature extraction capabilities, enabling the student model to run stably on edge devices and meeting the application requirements of edge devices with limited computing power, storage and power consumption.
[0033] Building upon the backbone network output, the student model further incorporates a multi-layer convolutional structure to enhance its ability to model image features hierarchically. This multi-layer convolutional structure includes a first convolutional layer, a second convolutional layer, and a third convolutional layer.
[0034] The first convolutional layer uses a 5×5 convolutional kernel to expand the feature receptive field in the shallow stage, thereby enhancing the ability to capture local spatial structure and low-frequency semantic information of the image.
[0035] Both the second and third convolutional layers employ a depthwise separable convolutional structure. The depthwise separable convolutional structure includes two stages: channel-wise convolution and pointwise convolution. Channel-wise convolution is used to extract the spatial features of each channel, while pointwise convolution linearly combines the features of each channel through 1×1 convolution to achieve the fusion of information between channels.
[0036] The pooling layer employs a downsampling structure, primarily used for feature map size compression and feature aggregation. By performing operations such as max pooling or average pooling within local regions, it reduces the spatial resolution of the feature map, thereby reducing the number of model parameters and computational complexity, while simultaneously enhancing the robustness of features to minor changes such as translation and scaling.
[0037] The fully connected layer is a crucial structure in neural networks used for feature synthesis and decision output. It achieves global fusion of high-dimensional features by establishing connections between all neurons in the previous layer and those in the current layer. Located in the higher layers or output stage of the network, the fully connected layer maps the extracted abstract features to a specific task space.
[0038] The output layer is configured as a linear mapping layer, used to map the image features extracted by the student model to a multimodal semantic embedding space shared with the teacher model. Through this linear mapping, the image features output by the student model maintain consistency with the semantic relationship distribution of the teacher model in terms of dimension and distribution.
[0039] The student model is trained using a model distillation method. During the distillation training process, the student model learns the structured knowledge of the teacher model in the multimodal semantic embedding space by aligning the semantic relation distribution in the multimodal semantic embedding space, thereby acquiring cross-modal semantic expression capabilities using only the image modality.
[0040] The parameters of the student model are determined using a forward computation method, which includes sequentially executing convolutional layers, pooling layers, and fully connected layers to obtain the image feature representation corresponding to the student model.
[0041] The parameters of the student model include the convolutional layer weights. Convolutional layer bias Feature projection layer weights Feature projection layer bias Student model output feature dimensions Teacher model embedding spatial dimension Semantic relation: distillation temperature Number of semantic anchors Input dimension is Output dimension is As shown in Table 1.
[0042] Table 1: Student Model Parameters
[0043] During distillation training, the student model does not contain a text encoding structure and does not receive any text label input. Its training objective is to learn to map images to a semantic relation distribution consistent with the teacher model.
[0044] Step 4: Use the image classification dataset to train the student model under the federated framework. During training, enable the distillation mechanism based on semantic anchors to obtain the trained student model.
[0045] Step 4 includes the following steps 401 to 406: Step 401: During training, the images in the image classification dataset are used as the image input of the teacher model. Based on the text labels corresponding to the image input, a text description consistent with the semantics of the text labels is constructed, and semantic anchors are determined according to the text description.
[0046] The text description is encoded using a teacher model to construct semantic anchors that correspond one-to-one with each category. These semantic anchors represent the reference representations of different categories in the multimodal semantic embedding space and remain relatively stable during model training, serving to constrain and guide the semantic alignment of image features.
[0047] Step 402: Based on semantic anchors, the model distillation method is used to calculate the semantic relation distribution of the student model in the multimodal semantic embedding space; The image to be processed is fed into the image encoder in the teacher model to extract features and obtain corresponding image embedding features. Simultaneously, text labels are fed into the text encoder in the teacher model, which generates a set of text descriptions and determines semantic anchors from the set of text descriptions. After normalizing the image embedding features and semantic anchors, the similarity between them is calculated to characterize the semantic association strength of the image features relative to each semantic anchor. Based on the similarity results, the semantic relationship distribution of the student model in the multimodal semantic embedding space is constructed to characterize the high-level cognitive information of the student model regarding the semantic structure of the image.
[0048] The normalization process employs a Softmax function with temperature parameter adjustment, enabling the student model to achieve cross-modal semantic representation capabilities without directly fitting the teacher model parameters; the temperature parameter is initialized. =0.1. Softmax function formula: ; in, The student model is represented in the first... The probability of semantic relations at each semantic anchor point This indicates that the image features extracted by the student model are compared with the first... The similarity value between semantic anchor vectors Indicates the number of semantic anchors. The distillation temperature represents the semantic relationship and is used to adjust the smoothness of the probability distribution.
[0049] Step 403: Calculate the semantic relation distillation loss of the student model and output the total loss of the student model; The formula for the semantic relation distillation loss of the student model output is:
[0050] in, The loss represents the semantic relation distillation loss of the student model output, expressed as the probability distribution of the teacher model in the multimodal semantic embedding space. Probability distribution of the student model in the multimodal semantic embedding space The divergence was calculated to obtain the result. The entropy of the teacher distribution is a constant. Optimization is equivalent to minimizing the cross-entropy. , Indicates the number of semantic anchors.
[0051] The semantic relation distillation loss is used to measure the consistency between the student model's understanding of image semantic relations in the multimodal semantic embedding space and the teacher model, thereby guiding the student model to inherit the cross-modal semantic structure information of the teacher model while maintaining a lightweight structure.
[0052] The total loss of the student model is: ; in, This represents the total loss of the student model. This indicates the loss caused by hard labels. This indicates the proportion of total loss caused by hard labels.
[0053] Steps 402 to 403 are used to implement semantic relation distillation training based on semantic anchors. The semantic relation distribution of the student model in the multimodal semantic embedding space is used as the prediction result of the student model.
[0054] Step 404: Calculate the uncertainty of the student model based on the semantic relation distribution of the student model in the multimodal semantic embedding space.
[0055] When the uncertainty of the student model is greater than a preset threshold, hard labels provided by the image classification dataset are introduced to strengthen the training of the student model and update the parameters of the student model; when the uncertainty is lower than the threshold, the parameters of the student model are updated based on semantic relation distillation loss. The formula for calculating the uncertainty of the student model: ; in, Indicates sample The prediction uncertainty Indicates the number of semantic anchors. Represents the student model for the first The predicted probabilities of each semantic anchor point are then processed through steps 401 to 404, completing the efficient training of the student model based on semantic anchor point distillation.
[0056] Step 405: Perform federated learning on the parameters of the student model from multiple end devices to generate updated global student model parameters and distribute them to each end device. Specifically, in each round of federated learning communication, each end device uses the collected image data locally to train the student model based on the model distillation method of cross-modal relationships, and obtains the locally updated student model parameters. Without uploading the original image data, each end device only uploads the parameters or parameter increments of the local student model to the cloud server. The cloud server performs weighted aggregation of the student model parameters from multiple end devices, generates global student model parameters, and distributes them to each end device to achieve collaborative fusion of knowledge from multiple end devices.
[0057] The federated learning aggregation adopts a weighted average strategy, and its aggregation process can be represented as follows: ; in, Indicates the first The global student model weights obtained after round-robin communication Indicates the first k The weights of the local student model trained by each end device in the t-th round of communication. Indicates the first k The number of data samples used by each edge device for local training. This represents the total number of edge devices participating in federated learning; The parameters of the federated learning framework include the number of federated learning communication rounds. Number of local training rounds on the end side Number of semantic anchors Semantic relation: distillation temperature Hard label supervision loss weight The percentage of clients participating in the calculation of sample entropy As shown in Table 2.
[0058] Table 2: Parameter settings for the federated learning framework based on semantic anchor distillation
[0059] Step 406: Following the process from Steps 401 to 405, perform multiple rounds of federated learning training to obtain a converged student model.
[0060] The resulting converged student model can be deployed on edge devices and can perform image classification, recognition, or semantic understanding tasks based solely on image input, thereby enabling efficient transfer of cross-modal knowledge to the lightweight student model on the edge while ensuring data privacy.
[0061] The teacher model and the trained student model can be deployed on edge devices. During training, the image data collected by the edge devices is first written to the data cache module and temporarily stored in the in-memory computing chip or external memory; the edge devices read the image data from the data cache according to a preset scheduling strategy and send it to the neural network processing unit to perform forward computation of the student model.
[0062] The cloud server receives model parameters uploaded from multiple edge devices and uses a federated learning aggregation strategy to generate updated global model parameters. These global model parameters are then distributed to each edge device in subsequent training rounds to continue the local training or inference process.
[0063] Step 5: Obtain the image input to be recognized and perform image recognition using a converged student model.
[0064] The image to be identified is acquired through an intelligent image sensor. After necessary preprocessing (such as image enhancement), it is input into the trained student model. The student model extracts features and performs semantic mapping on the image, and outputs the corresponding recognition result based on its semantic representation in the joint embedding space, thereby completing the image recognition and classification process.
[0065] During the recognition process, the edge device only performs the forward computation process of the student model and does not involve parameter updates; the image data is sent to the integrated sensing and computing chip via the data caching module, and convolution, pooling and fully connected operations are performed in sequence, and the image classification, recognition or semantic understanding results are output; during the recognition process, the student model can complete the task based solely on image input, without the need for text input or teacher model participation.
[0066] It should be noted that the data caching module can be set in on-chip storage or external memory. The integrated sensing and computing chip can be used for inference computing as well as to support lightweight local training operations. The neural network parameters are allowed to be transmitted bidirectionally during the training phase and loaded unidirectionally during the inference phase, thereby realizing efficient edge inference and federated collaborative training.
[0067] The thresholds mentioned above can be set according to the needs of actual applications, and the embodiments of the present invention do not impose any restrictions on them.
[0068] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.
[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image recognition method based on model distillation and federated learning, characterized in that, Includes the following processes: Step 1: The edge device acquires the image classification dataset; Step 2: Obtain the teacher model and generate semantic anchors and image embedding features for each image in the image classification dataset using the teacher model; Furthermore, the distances between multiple image embedding features and their corresponding semantic anchors are used to form a semantic relationship distribution; Step 3: The end-side device establishes a student model; Step 4: Use the image classification dataset to train the student model under the federated framework. During training, enable the distillation mechanism based on semantic anchors to obtain the trained student model. Step 5: Obtain the image input to be recognized and perform image recognition using a converged student model.
2. The image recognition method based on model distillation and federated learning according to claim 1, characterized in that, The end-side device includes an image acquisition module, an image preprocessing module, an integrated sensing and computing chip, and a data storage module.
3. The image recognition method based on model distillation and federated learning according to claim 1, characterized in that, The teacher model includes an image encoder and a text encoder; The image encoder is used to extract features from the input image and generate image embedding features with global semantic expression capabilities; The text encoder is used to encode the input text labels and generate text embedding features that are located in the same multimodal semantic embedding space as the image embedding features by modeling the words and their contextual relationships. The image encoder and text encoder construct a shared multimodal semantic embedding space through joint training.
4. The image recognition method based on model distillation and federated learning according to claim 3, characterized in that, In step 2, for the text label of each image data in the image classification dataset, a text description is generated using the text encoder in the teacher model; the text labels of multiple image data in the image classification dataset are used to form multiple text descriptions, resulting in a set of text descriptions. In the text description set, multiple semantic anchors are manually determined and encoded to obtain the corresponding semantic anchor set; For image input in the image classification dataset, the image encoder in the teacher model is used to extract image embedding features and calculate the distance between the image embedding features and the semantic anchors; the distances between multiple image embedding features and their corresponding semantic anchors form a semantic relationship distribution.
5. The image recognition method based on model distillation and federated learning according to claim 1, characterized in that, The student model uses a convolutional neural network, including: a backbone network, a first convolutional layer, a second convolutional layer, a third convolutional layer, a pooling layer, a fully connected layer, and an output layer; The backbone network adopts the MobileNetV3 structure in convolutional neural networks, and the MobileNetV3 structure contains at least one dynamic block. The first convolutional layer uses a 5×5 convolutional kernel; Both the second and third convolutional layers employ depthwise separable convolutional structures. The pooling layer employs a downsampling structure; The output layer is set as a linear mapping layer, which is used to map the image features extracted by the student model to the multimodal semantic embedding space shared with the teacher model.
6. The image recognition method based on model distillation and federated learning according to claim 5, characterized in that, The student model is trained using a model distillation method. During the distillation training process, the student model learns the structured knowledge of the teacher model in the multimodal semantic embedding space by aligning the semantic relation distribution in the multimodal semantic embedding space, thereby obtaining cross-modal semantic expression capabilities using only the image modality.
7. The image recognition method based on model distillation and federated learning according to claim 5, characterized in that, The parameters of the student model include the convolutional layer weights. Convolutional layer bias Feature projection layer weights Feature projection layer bias Student model output feature dimensions Teacher model embedding spatial dimension Semantic relation: distillation temperature Number of semantic anchors Input dimension is Output dimension is .
8. The image recognition method based on model distillation and federated learning according to claim 1, characterized in that, Step 4 includes the following steps 401 to 406: Step 401: During training, the images in the image classification dataset are used as the image input of the teacher model. Based on the text labels corresponding to the image input, a text description consistent with the semantics of the text labels is constructed, and semantic anchors are determined according to the text description. Step 402: Based on semantic anchors, the model distillation method is used to calculate the semantic relation distribution of the student model in the multimodal semantic embedding space; Step 403: Calculate the semantic relation distillation loss of the student model and output the total loss of the student model; The formula for the semantic relation distillation loss of the student model output is: ； ； ； in, The loss represents the semantic relation distillation loss of the student model output, expressed as the probability distribution of the teacher model in the multimodal semantic embedding space. Probability distribution of the student model in the multimodal semantic embedding space The divergence was calculated to obtain the result. The entropy of the teacher distribution; optimization is equivalent to minimizing the cross entropy. , Indicates the number of semantic anchors; The total loss of the student model is: ； in, This represents the total loss of the student model. This indicates the loss caused by hard labels. This indicates the proportion of total loss caused by hard labels; Step 404: Calculate the uncertainty of the student model based on the semantic relation distribution of the student model in the multimodal semantic embedding space; When the uncertainty of the student model is greater than a preset threshold, hard labels provided by the image classification dataset are introduced to strengthen the training of the student model and update the parameters of the student model; when the uncertainty is lower than the threshold, the parameters of the student model are updated based on semantic relation distillation loss. The formula for calculating the uncertainty of the student model: ； in, Indicates sample The prediction uncertainty Indicates the number of semantic anchors. Represents the student model for the first Predicted probability of each semantic anchor point; Step 405: Perform federated learning on the parameters of the student model from multiple end devices to generate updated global student model parameters and distribute them to each end device. Step 406: Following the process from Steps 401 to 405, perform multiple rounds of federated learning training to obtain a converged student model.
9. The image recognition method based on model distillation and federated learning according to claim 8, characterized in that, In step 405, the federated learning aggregation adopts a weighted average strategy, and the aggregation process can be represented as follows: ； in, Indicates the first The global student model weights obtained after round-robin communication Indicates the first k The weights of the local student model trained by each end device in the t-th round of communication. Indicates the first k The number of data samples used by each edge device for local training. This represents the total number of edge devices participating in federated learning; The parameters of the federated learning framework include the number of federated learning communication rounds. Number of local training rounds on the end side Number of semantic anchors Semantic relation: distillation temperature Hard label supervision loss weight The percentage of clients participating in the calculation of sample entropy .