Federal learning tactile signal cross-modal generation method based on Byzantine robustness
By introducing Byzantine robust federated learning methods and fingerprint recognition mechanisms into the tactile Internet, identifying and eliminating Byzantine clients, combining closed-loop learning to correct the feedforward network output, the problem of insufficient robustness of federated learning system is solved, and the accuracy and complexity of tactile friction generation is achieved.
Patent Information
- Application Number
- CN202510574077.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-06
- Publication Date
- 2025-08-12
AI Technical Summary
In the tactile Internet, the federated learning system is not robust enough to be vulnerable to Byzantine attacks, affecting training effects and reducing model performance. At the same time, existing methods are difficult to achieve a balance between high accuracy and low complexity.
Byzantine-based robust federated learning method is adopted, by establishing a global closed-loop learning model, using feedforward and feedback networks for cross-modal generation, and introducing fingerprint recognition mechanisms to identify and eliminate Byzantine clients, combining closed-loop learning methods to correct the feedforward network output to ensure the robustness and accuracy of the model.
Effectively resist Byzantine attacks, ensure the robustness of the global model, and at the same time achieve a good balance between the accuracy and complexity of tactile friction generation, avoiding the contamination of the model by abnormal updates.
Smart Images
Figure CN120474758A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of tactile Internet technology, and in particular to a cross-modal generation method of tactile signals based on Byzantine robust federated learning. Background Art
[0002] The Tactile Internet, a new type of industrial network, provides real-time tactile feedback through reliable, responsive, and intelligent connectivity. A key technical aspect of the Tactile Internet is the acquisition, compression, transmission, and display of tactile information. This article specifically explores methods for acquiring tactile friction data. In Tactile Internet applications, this data can be transmitted to remote surface haptic devices to display shapes and textures on touchscreens.
[0003] An effective approach to capturing tactile friction is cross-modal generation, using machine learning tools to train a mapping model from visual data (images) to tactile friction. However, in the distributed tactile internet, this model training is often insufficient due to limited local data. Federated learning addresses this issue by allowing multiple clients to collaboratively learn a shared prediction model while keeping all training data locally. This means that sensitive data does not need to be transmitted over the network, reducing the risk of data leakage.
[0004] Currently, there has been progress in integrating FL with cross-modal generators to achieve tactile friction in the Tactile Internet. While this approach has performed well, it presents several challenges: the robustness of federated learning systems needs to be enhanced to resist malicious attacks. For example, federated aggregation can be compromised by Byzantine clients providing extreme or incorrect parameters, which can affect training and degrade global model performance. Furthermore, achieving both high accuracy and low complexity requires improving the performance of local models. To overcome these shortcomings in current practical applications, we propose a Byzantine-robust federated learning approach for cross-modal generation of tactile signals. Summary of the Invention
[0005] In view of the shortcomings of the existing technology, the purpose of the embodiments of the present invention is to provide a cross-modal generation method of tactile signals based on Byzantine robust federated learning to solve the problems in the above-mentioned background technology.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A cross-modal generation method for tactile signals based on Byzantine robust federated learning includes the following steps:
[0008] Step 1: The server establishes a global closed-loop learning model and initializes the model parameters;
[0009] Step 2: The server starts the client scheduling process and sends global model parameters and communication round information to each client;
[0010] Step 3: The client uses the feedforward network to map the input grayscale image to the friction spectrum, and reconstructs the grayscale image through the feedback network;
[0011] Step 4: The client updates the local feedforward network and feedback network parameters by stochastic gradient descent, taking the feedforward network loss and feedback loss as constraints.
[0012] Step 5: The client extracts the temporary fingerprint information of this round and uploads the updated feedforward network parameters and temporary fingerprint information to the server;
[0013] Step 6: The server collects temporary fingerprints of each client in the first several rounds of communication and takes the intersection to establish a global intrinsic fingerprint;
[0014] Step 7: In the subsequent defense phase, the server calculates the fingerprint retention rate of each client, identifies clients with a retention rate below a preset threshold as Byzantine clients, and removes their updates;
[0015] Step 8: The server aggregates the feedforward network parameters uploaded by the retained non-Byzantine clients by taking a weighted average of the data volume of each client.
[0016] As a further solution of the present invention, in step 1, the specific operations of the server side establishing a global closed-loop learning model and initializing model parameters are:
[0017] The server defines the feedforward network FF and feedback network FB of the global closed-loop learning model. Both are encoder-decoder structures. The encoder reduces the dimensionality of the image through multiple convolutional blocks. Each convolutional block contains a convolution layer, a ReLU activation function, and a batch normalization layer.
[0018] Each level of the decoder consists of a transposed convolution layer, a batch normalization layer, and an activation function ReLU;
[0019]
[0020] The server initializes the closed-loop learning network parameters w(0) and θ(0), and establishes the client scheduling process.
[0021] As a further solution of the present invention, in step 2, the specific operation of the client scheduling process is:
[0022] In each round of communication, the server sends the current global closed-loop model parameters and communication round t to the client; the client Receive the global closed-loop model parameters and communication round [w(t)=w(0),θ(t)=θ(0),t=1] from the server, and update the local model parameters w c (t),θ c (t) and the number of rounds of communication τ, where
[0023] w c (τ)←w(t)
[0024] θ c (τ)←θ c (t)
[0025] τ←t
[0026] The client uses the model parameters to train the local closed-loop cross-modal generation model.
[0027] As a further solution of the present invention, in step 3, the client uses a feedforward network to map the input grayscale image to a friction spectrum, and the specific operation of reconstructing the grayscale image through a feedback network is as follows:
[0028] Client From M c Surface materials collect visual and tactile data; each material m∈{1,...,M c}Include visual and tactile samples, respectively represented as grayscale image sets and friction coefficient set Therefore, the local dataset can be expressed as The complete dataset can be defined as
[0029] Client c uses short-time Fourier transform from the time version Calculation of friction spectrum For the nth sample of the mth material, the grayscale image and friction spectrum are expressed as and in The size is larger than size;
[0030] The client uses the feedforward network FF to convert the grayscale image Convert to the generated friction spectrum in is the feedforward network parameter of client c, which contains I components; the feedback network FB is used to generate the corresponding grayscale image from the generated friction spectrum, and the generated grayscale image is compared with the real grayscale image Compare this to establish the feedback constraints: in is the feedback network parameter of client c, which contains J components.
[0031] As a further solution of the present invention, the specific operation of the client updating the model parameters by stochastic gradient descent in step 4 is:
[0032] The client randomly extracts small batches of data sets And establish the following loss functions as constraints for the feedforward network FF and the feedback network FB respectively:
[0033]
[0034] Using the loss function, the network parameters are updated by stochastic gradient descent:
[0035]
[0036] Where α is the learning rate,
[0037] As a further solution of the present invention, the specific operation of the client extracting the temporary fingerprint information of this round in step 5 is:
[0038] The temporary fingerprint is defined as follows: At each round t, for each client c, η represents a small hyperparameter chosen close enough to zero so that there is at least one Satisfy the following formula;
[0039]
[0040] All included Collection It is called temporary local fingerprint;
[0041] The server obtains the temporary fingerprint information of each client c in round t according to this definition Client c extracts the temporary fingerprint information of this round And the updated feedforward network parameters w c (t) and temporary fingerprint information are uploaded to the server.
[0042] As a further solution of the present invention, the specific operation of the server in step 6 to establish the global intrinsic fingerprint is:
[0043] The server can Obtain the global fingerprint generated by all clients at time t, which can be used to build the intrinsic fingerprint of the training model;
[0044] The intrinsic fingerprint is defined as follows: Let T * ∈{1,...,T} represents the establishment time, and the following conditions are met;
[0045]
[0046] At time T * , Defined as intrinsic fingerprint.
[0047] As a further solution of the present invention, the specific operation of the server in step 7 to identify and eliminate Byzantine clients is as follows:
[0048] After the system enters the defense phase, at each inspection time t'∈{T * +1,...,T}, the server receives local temporary fingerprints from all clients according to the temporary fingerprint definition And calculate the fingerprint retention rate R between these fingerprints and the intrinsic fingerprint c (t'), when the fingerprint retention rate of client c is R c (t') He is identified as a suspected Byzantine client at t' when the following conditions are met;
[0049]
[0050] where ξ is chosen based on the inclusion relation between two fingerprints in the absence of Byzantine behavior.
[0051] As a further solution of the present invention, the specific operation of the server aggregating the feedforward network parameters uploaded by the non-Byzantine client in step 8 is:
[0052] make is the Byzantine client set, Representing the remaining clients, the server aggregates the feedforward network parameters of normal clients using the updated aggregation formula:
[0053]
[0054] The server sends the updated feedforward network parameters to each client, starting a new round of communication. After multiple communications, a generator that realizes cross-modal generation of tactile signals is obtained.
[0055] In summary, the embodiments of the present invention have the following beneficial effects compared with the prior art:
[0056] The present invention not only ensures the Byzantine robustness of federated learning, but also introduces a fingerprint recognition mechanism to identify and eliminate Byzantine clients. The server quickly establishes an internal fingerprint and detects abnormal behavior by comparing the redundant parameter changes in the model uploaded by the client, thereby avoiding abnormal updates from polluting the global model. Even if there is a malicious attack in the client, the tactile friction generated by this method can still maintain a close match with the actual tactile sensation. For the local client, a closed-loop learning method is used to generate fingertip tactile friction from the visual image. By connecting the output of the feedback network with the output of the feedforward network, the feedback network continuously evaluates and corrects the output error of the feedforward network, thereby constraining the training direction of the feedforward network. The present invention not only ensures the robustness of the federated system, but also achieves a good balance between accuracy and complexity.
[0057] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 Flowchart of the method of the present invention.
[0059] Figure 2 This is a parameter transfer flow chart of the method of the present invention.
[0060] Figure 3 This is the model loss when fingerprint detection and other Byzantine defense methods are deployed in the training phase under different proportions of Byzantine clients in the present invention.
[0061] Figure 4 A comparison of the tactile friction accuracy of the closed-loop model and existing methods in the fingerprint detection framework. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0063] The specific implementation of the present invention is described in detail below with reference to specific embodiments.
[0064] In one embodiment, a cross-modal generation method of tactile signals based on Byzantine robust federated learning is described. Figures 1 to 4 , including the following steps:
[0065] Step 1: The server establishes a global closed-loop learning model and initializes the model parameters;
[0066] Step 2: The server starts the client scheduling process and sends global model parameters and communication round information to each client;
[0067] Step 3: The client uses the feedforward network to map the input grayscale image to the friction spectrum, and reconstructs the grayscale image through the feedback network;
[0068] Step 4: The client updates the local feedforward network and feedback network parameters by stochastic gradient descent, taking the feedforward network loss and feedback loss as constraints.
[0069] Step 5: The client extracts the temporary fingerprint information of this round and uploads the updated feedforward network parameters and temporary fingerprint information to the server;
[0070] Step 6: The server collects temporary fingerprints of each client in the first several rounds of communication and takes the intersection to establish a global intrinsic fingerprint;
[0071] Step 7: In the subsequent defense phase, the server calculates the fingerprint retention rate of each client, identifies clients with a retention rate below a preset threshold as Byzantine clients, and removes their updates;
[0072] Step 8: The server aggregates the feedforward network parameters uploaded by the retained non-Byzantine clients by taking a weighted average of the data volume of each client.
[0073] For further information, see Figures 1 to 4 In step 1, the server establishes a global closed-loop learning model and initializes the model parameters as follows:
[0074] The server defines the feedforward network FF and feedback network FB of the global closed-loop learning model. Both are encoder-decoder structures. The encoder reduces the dimensionality of the image through multiple convolutional blocks. Each convolutional block contains a convolution layer, a ReLU activation function, and a batch normalization layer.
[0075] Each level of the decoder consists of a transposed convolution layer, a batch normalization layer, and an activation function ReLU;
[0076]
[0077] The server initializes the closed-loop learning network parameters w(0) and θ(0), and establishes the client scheduling process.
[0078] For further information, see Figures 1 to 4 In step 2, the specific operation of the client scheduling process is:
[0079] In each round of communication, the server sends the current global closed-loop model parameters and communication round t to the client; the client Receive the global closed-loop model parameters and communication round [w(t)=w(0),θ(t)=θ(0),t=1] from the server, and update the local model parameters w c(t),θ c (t) and the number of rounds of communication τ, where
[0080] w c (τ)←w(t)
[0081] θ c (τ)←θ c (t)
[0082] τ←t
[0083] The client uses the model parameters to train the local closed-loop cross-modal generation model.
[0084] For further information, see Figures 1 to 4 In step 3, the client uses the feedforward network to map the input grayscale image into a friction spectrum, and the specific operation of reconstructing the grayscale image through the feedback network is as follows:
[0085] Client From M c Surface materials collect visual and tactile data; each material m∈{1,...,M c}Include visual and tactile samples, respectively represented as grayscale image sets and friction coefficient set Therefore, the local dataset can be expressed as The complete dataset can be defined as
[0086] Client c uses short-time Fourier transform from the time version Calculation of friction spectrum For the nth sample of the mth material, the grayscale image and friction spectrum are expressed as and in The size is larger than size;
[0087] The client uses the feedforward network FF to convert the grayscale image Convert to the generated friction spectrum in is the feedforward network parameter of client c, which contains I components; the feedback network FB is used to generate the corresponding grayscale image from the generated friction spectrum, and the generated grayscale image is compared with the real grayscale image Compare this to establish the feedback constraints: in is the feedback network parameter of client c, which contains J components.
[0088] For further information, see Figures 1 to 4 In step 4, the specific operation of the client updating the model parameters by stochastic gradient descent is:
[0089] The client randomly extracts small batches of data sets And establish the following loss functions as constraints for the feedforward network FF and the feedback network FB respectively:
[0090]
[0091] Using the loss function, the network parameters are updated by stochastic gradient descent:
[0092]
[0093] Where α is the learning rate,
[0094] For further information, see Figures 1 to 4 In step 5, the specific operation of the client to extract the temporary fingerprint information of this round is:
[0095] The temporary fingerprint is defined as follows: At each round t, for each client c, η represents a small hyperparameter chosen close enough to zero so that there is at least one Satisfy the following formula;
[0096]
[0097] All included Collection It is called temporary local fingerprint;
[0098] The server obtains the temporary fingerprint information of each client c in round t according to this definition Client c extracts the temporary fingerprint information of this round And the updated feedforward network parameters w c (t) and temporary fingerprint information are uploaded to the server.
[0099] For further information, see Figures 1 to 4 The specific operation of the server in step 6 to establish the global intrinsic fingerprint is:
[0100] The server can Obtain the global fingerprint generated by all clients at time t, which can be used to build the intrinsic fingerprint of the training model;
[0101] The intrinsic fingerprint is defined as follows: Let T * ∈{1,...,T} represents the establishment time, and the following conditions are met;
[0102]
[0103] At time T * , Defined as intrinsic fingerprint.
[0104] For further information, see Figures 1 to 4 The specific operations of the server in step 7 to identify and remove Byzantine clients are as follows:
[0105] After the system enters the defense phase, at each inspection time t'∈{T * +1,...,T}, the server receives local temporary fingerprints from all clients according to the temporary fingerprint definition And calculate the fingerprint retention rate R between these fingerprints and the intrinsic fingerprint c (t'), when the fingerprint retention rate of client c is R c (t') He is identified as a suspected Byzantine client at t' when the following conditions are met;
[0106]
[0107] where ξ is chosen based on the inclusion relation between two fingerprints in the absence of Byzantine behavior.
[0108] For further information, see Figures 1 to 4 In step eight, the server aggregates the feedforward network parameters uploaded by the non-Byzantine client in the following specific operations:
[0109] make is the Byzantine client set, Representing the remaining clients, the server aggregates the feedforward network parameters of normal clients using the updated aggregation formula:
[0110]
[0111] The server sends the updated feedforward network parameters to each client, starting a new round of communication. After multiple communications, a generator that realizes cross-modal generation of tactile signals is obtained.
[0112] In this embodiment, the server defines a global closed-loop learning model consisting of a feedforward network (FF) and a feedback network (FB), both of which are encoder-decoder structures. The encoder reduces the dimensionality of the image through multiple convolutional blocks, each of which contains a convolutional layer, a ReLU activation function, and a batch normalization layer. Similar to the encoder, each level of the decoder includes a transposed convolutional layer, a batch normalization layer, and a ReLU activation function. The difference between the two networks is that, to improve representational power, the output features of each transposed convolutional layer of the feedforward network (FF) are concatenated with the corresponding output of the convolutional layer, thereby integrating the features extracted by the encoder. This concatenation does not occur in the feedback network (FB).
[0113] The applicability of the proposed cross-modal tactile signal generation method based on Byzantine robust federated learning to tactile data generation is analyzed through simulation experimental data. The pytorch framework of Python software is used in the simulation experiment. The specific content is as follows:
[0114] The simulation uses Python multi-threading to simulate the real environment, creating a parameter aggregation thread and C client scheduling threads. The client scheduling thread is in a suspended state.
[0115] The client-side closed-loop learning model's feedforward network (FF) and feedback network (FB) were built using the Python PyTorch framework. Following the typical convolutional neural network architecture, the encoder reduces the image dimensionality in multiple stages. Each stage consists of a convolutional layer, a Reluctant Unit (ReLU) activation function, and a normalization layer. Similar to the encoder, each decoder stage includes a transposed convolutional layer, a batch normalization layer, and a Reluctant Unit (ReLU) activation function.
[0116] Start the client scheduling thread: get the initial model parameters and communication round [w(t) = w(0), θ(t) = θ(0), t = 1], assign them to the closed-loop model, and use the grayscale image as input to update the client local parameter w using stochastic gradient descent c (t) and local temporary fingerprint Sent to the server.
[0117] The server starts the parameter aggregation thread and calculates the fingerprint retention rate R of each client based on the intrinsic fingerprint. c (t'), clients with fingerprint retention rates below a threshold are identified as suspected Byzantine clients, and the parameters uploaded by these clients are removed. This completes the tactile data training task under multi-user collaboration.
[0118] The HapTex database was used as simulation data to evaluate the Byzantine robust federated learning-based cross-modal tactile signal generation method. The HapTex database contains 10 categories of surface materials, each of which contains 12 types of materials, for a total of 120 materials. For each material, the dataset provides grayscale image samples and tactile friction samples.
[0119] First, more visual and tactile data samples are obtained by randomly cropping from existing samples. For visual data, a sub-image of size 1024×1024 is cropped from each grayscale image using a horizontally moving sliding window. For tactile data, the friction coefficient consisting of 1280 data points is extracted using a moving sliding window corresponding to the size of a 1024×1024 image. The friction spectrum is then calculated using STFT. Finally, we obtain 600 paired visual samples (1024×1024) and tactile samples (257×11) for each material and randomly divide them into training, validation, and test sets in a ratio of 8:1:1. The training samples of all materials are randomly distributed to 100 customers to simulate a realistic and common federated learning scenario.
[0120] The hyperparameters used in the simulation are: communication round T = 2000, local epoch E = 3, learning rate α = 0.0001, η = 1 × 10 -11 ,ξ=0.8.
[0121] After 2000 communications, save the trained network. Use the trained network to classify the test set and get the classification accuracy of the test set (Figure 1). Take Velvet from the HapTex dataset as an example, Figure 3 The model loss of the fingerprint detection-based Byzantine client identification method and its three competitors under different proportions of Byzantine clients is intuitively compared. Figure 4 As shown in Figure 3, we take cotton from the HapTex dataset as an example and intuitively compare the accuracy of tactile friction generated by the fingerprint detection-based combined federated learning and closed-loop learning method with its three competitors.
[0122] The analysis of the experimental results shows that the method proposed in this invention has excellent resistance to Byzantine attacks and cross-modal generation capabilities. Figure 3 It can be concluded that the fingerprint detection method is completely resistant to attacks and has the best model performance regardless of the proportion of Byzantine clients. In contrast, although other methods can initially mitigate attacks when the Byzantine proportion is low, they fail when the proportion exceeds 50%, resulting in complete model contamination. Figure 4 It can be concluded that the closed-loop model generates tactile friction that outperforms the other four methods. This superior performance is primarily attributed to the feedback mechanism in the closed-loop model. As the feedforward network processes data, it further optimizes its parameters, improving the accuracy of generated tactile friction.
[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A cross-modal generation method of tactile signals based on Byzantine robust federated learning, characterized in that: The following steps are involved: Step 1: The server establishes a global closed-loop learning model and initializes the model parameters; Step 2: The server starts the client scheduling process and sends global model parameters and communication round information to each client; Step 3: The client uses the feedforward network to map the input grayscale image to the friction spectrum, and reconstructs the grayscale image through the feedback network; Step 4: The client updates the local feedforward network and feedback network parameters by stochastic gradient descent, taking the feedforward network loss and feedback loss as constraints. Step 5: The client extracts the temporary fingerprint information of this round and uploads the updated feedforward network parameters and temporary fingerprint information to the server; Step 6: The server collects temporary fingerprints of each client in the first several rounds of communication and takes the intersection to establish a global intrinsic fingerprint; Step 7: In the subsequent defense phase, the server calculates the fingerprint retention rate of each client, identifies clients with a retention rate below a preset threshold as Byzantine clients, and removes their updates; Step 8: The server aggregates the feedforward network parameters uploaded by the retained non-Byzantine clients by taking a weighted average of the data volume of each client.
2. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 1 is characterized in that: In step 1, the server establishes a global closed-loop learning model and initializes model parameters as follows: The server defines the feedforward network FF and feedback network FB of the global closed-loop learning model. Both are encoder-decoder structures. The encoder reduces the dimensionality of the image through multiple convolutional blocks. Each convolutional block contains a convolution layer, a ReLU activation function, and a batch normalization layer. Each level of the decoder consists of a transposed convolution layer, a batch normalization layer, and a ReLU activation function; The server initializes the closed-loop learning network parameters as w(0) and θ(0), and establishes the client scheduling process.
3. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 2 is characterized in that: In step 2, the specific operations of the client scheduling process are: In each round of communication, the server sends the current global closed-loop model parameters and communication round t to the client; the client Receive the global closed-loop model parameters and communication round [w(t)=w(0),θ(t)=θ(0),t=1] from the server, and update the local model parameters w c (t),θ c (t) and the number of rounds of communication τ, where w c (τ)←w(t) i c (t)←θ c (t) τ←t The client uses the model parameters to train the local closed-loop cross-modal generation model.
4. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 3 is characterized in that: In step 3, the client uses the feedforward network to map the input grayscale image into a friction spectrum, and the specific operation of reconstructing the grayscale image through the feedback network is as follows: Client From M c Surface materials collect visual and tactile data; each material m∈{1,...,M c }Include visual and tactile samples, respectively represented as grayscale image sets and friction coefficient set Therefore, the local dataset can be expressed as The complete dataset can be defined as Client c uses short-time Fourier transform from the time version Calculation of friction spectrum For the nth sample of the mth material, the grayscale image and friction spectrum are expressed as and in The size is larger than size; The client uses the feedforward network FF to convert the grayscale image Convert to the generated friction spectrum in is the feedforward network parameter of client c, which contains I components; Then the feedback network FB is used to generate the corresponding grayscale image from the generated friction spectrum, and the generated grayscale image is compared with the real grayscale image. Compare this to establish the feedback constraints: in is the feedback network parameter of client c, which contains J components.
5. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 4 is characterized in that: The specific operation of the client in step 4 to update the model parameters through stochastic gradient descent is: The client randomly extracts small batches of data sets And establish the following loss functions as constraints for the feedforward network FF and the feedback network FB respectively: Using the loss function, the network parameters are updated by stochastic gradient descent: Where α is the learning rate, 6. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 5 is characterized in that: The specific operation of the client in step 5 to extract the temporary fingerprint information of this round is: The temporary fingerprint is defined as follows: At each round t, for each client c, η represents a small hyperparameter chosen close enough to zero so that there is at least one Satisfy the following formula; All included Collection It is called temporary local fingerprint; The server obtains the temporary fingerprint information of each client c in round t according to this definition Client c extracts the temporary fingerprint information of this round And the updated feedforward network parameters w c (t) and temporary fingerprint information are uploaded to the server.
7. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 6 is characterized in that: The specific operation of the server in step 6 to establish the global intrinsic fingerprint is: The server can Obtain the global fingerprint generated by all clients at time t, which can be used to build the intrinsic fingerprint of the training model; The intrinsic fingerprint is defined as follows: Let T * ∈{1,...,T} represents the establishment time, and the following conditions are met; At time T * , Defined as intrinsic fingerprint.
8. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 7 is characterized in that: The specific operations of the server in step 7 to identify and remove Byzantine clients are as follows: After the system enters the defense phase, at each inspection time t'∈{T * +1,...,T}, the server receives local temporary fingerprints from all clients according to the temporary fingerprint definition And calculate the fingerprint retention rate R between these fingerprints and the intrinsic fingerprint c (t'), when the fingerprint retention rate of client c is R c (t') He is identified as a suspected Byzantine client at t' when the following conditions are met; where ξ is chosen based on the inclusion relation between two fingerprints in the absence of Byzantine behavior.
9. The cross-modal generation method of tactile signals based on Byzantine robust federated learning according to claim 8 is characterized in that: The specific operation of the server aggregating the feedforward network parameters uploaded by the non-Byzantine client in step 8 is: make is the Byzantine client set, Representing the remaining clients, the server aggregates the feedforward network parameters of normal clients using the updated aggregation formula: The server sends the updated feedforward network parameters to each client, starting a new round of communication. After multiple communications, a generator that realizes cross-modal generation of tactile signals is obtained.