A semantic information federated learning method and system for digital twinning in an industrial internet of things scenario
By using semantic communication and differential privacy techniques in semantic information federated learning in the Industrial Internet of Things (IIoT), the problems of communication overhead and data heterogeneity in federated learning are solved, the model accuracy and interpretability are improved, and intelligent decision-making and real-time data interaction are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2023-11-28
- Publication Date
- 2026-07-14
AI Technical Summary
In the context of the Industrial Internet of Things (IIoT), federated learning suffers from problems such as high communication overhead, data heterogeneity, and imbalanced samples, making it difficult to achieve intelligent decision-making.
Federated learning utilizes semantic feature information from semantic communication, extracts semantic information through deep learning neural networks, processes it on edge servers, and combines differential privacy technology for privacy protection, reducing data sharing and balancing data contributions.
Reduce communication overhead, improve model accuracy and interpretability, promote collaborative learning of digital twin models, and achieve intelligent decision-making and real-time data interaction.
Smart Images
Figure CN117669699B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a semantic information federated learning method and system for digital twins in the context of industrial Internet of Things (IoT), belonging to the field of semantic communication and machine learning technology. Background Technology
[0002] Federated learning is a distributed machine learning approach where multiple participants (typically devices or users) collaborate to train a global model, rather than centralizing the dataset in one place. In federated learning, each participant has its own local data but does not share the original data. Instead, they collaboratively train the global model by sharing gradients or other information about the model parameters. In digital twin modeling within Industrial Internet of Things (IIoT) scenarios, massive amounts of diverse data are required, posing a challenge to the communication overhead of federated learning. Furthermore, some participants may have more or less data in the federated learning process, leading to imbalanced sample problems.
[0003] The Industrial Internet of Things (IIoT) involves a large amount of data interaction and transmission, and involves different types of devices, sensors, and control systems during the transmission process. These devices are usually provided by different manufacturers and use different communication protocols and data formats, thus creating a data heterogeneity problem. Furthermore, different devices and systems need to make intelligent decisions based on specific working environments and tasks. Summary of the Invention
[0004] The purpose of this invention is to provide a federated learning method for semantic information in digital twins to address the problems mentioned in the background section.
[0005] This invention considers extracting semantic feature information from semantic communication using a deep learning neural network and using it as training data for federated learning on terminal devices. By incorporating semantic information into federated learning, the amount of data uploaded can be reduced. Only important features or feature summaries need to be shared, rather than all the original data, thus reducing communication overhead. Furthermore, semantic feature information helps balance data contributions, making each terminal device's contribution more meaningful. Considering privacy, security, and the uniformity of semantic information, semantic information processing is performed on an edge server, and differential privacy technology is added for privacy protection. Therefore, this invention can solve the heterogeneity problem and make intelligent decisions based on semantic information through federated learning of semantic information.
[0006] This invention also provides a semantic information federated learning system for digital twins in the context of industrial Internet of Things (IoT) scenarios.
[0007] Terminology Explanation:
[0008] 1. Industrial Internet of Things (IIoT) refers to the application of Internet of Things (IoT) technology in the industrial field. It connects, monitors, collects, and analyzes data from various devices, sensors, tools, and other physical objects to achieve intelligent, automated, and optimized industrial systems. Devices can be sensors, actuators, control systems, robots, etc., which can collect and exchange data in real time for remote monitoring and control, thereby improving production efficiency, reducing costs, optimizing resource utilization, and enhancing the flexibility and adaptability of the entire industrial production process.
[0009] 2. Convolutional Neural Networks (CNNs) are a class of deep learning neural networks specifically designed for processing and analyzing data with a grid structure. This is similar to Transformer neural networks; "transformer" typically refers to a neural network structure based on a self-attention mechanism.
[0010] To achieve the above objectives, the present invention provides the following technical solution:
[0011] A federated learning method for semantic information of digital twins in an industrial IoT scenario includes:
[0012] The data of the participants in the federated learning of industrial IoT terminal devices is input into the multimodal neural network in the edge server for data input selection; the data of the participants in the federated learning of industrial IoT terminal devices consists of images and text information collected by sensors;
[0013] Extract semantic information;
[0014] Add differential privacy protection to the extracted semantic information;
[0015] Model training is performed based on semantic information;
[0016] After the model training is completed, global model aggregation and model evaluation are performed. Based on the evaluation accuracy, the training is repeated or the next step is initiated.
[0017] Once the model evaluation meets the requirements, the digital twin model is generated.
[0018] Further preferred, the multimodal neural network is a convolutional neural network.
[0019] According to a preferred embodiment of the present invention, the original data is stored on the device of the local participant, and the semantic information extraction process is performed on the edge server. The local participant uploads the original data to the edge server, and the edge server extracts semantic information from the data.
[0020] According to a preferred embodiment of the present invention, extracting semantic information includes:
[0021] 1) Multimodal data input: Multimodal neural networks receive input data from different modalities;
[0022] 2) Modal feature extraction: For each input modality, the fusion model uses the corresponding feature extractor (e.g., CNN neural network for images, Transformer neural network for text) to extract the semantic feature representation of each modality; relevant features are extracted from the source message, as shown in Equation (1):
[0023] F = σ(W) T S+b T (1)
[0024] In equation (1), F represents the semantic features transmitted from the industrial IoT terminal device to the edge server, and W represents the semantic features transmitted from the terminal device to the edge server. T and b T These are the parameters from which features are extracted from the source message S, and σ(·) is the sigmoid activation function;
[0025] 3) Feature transmission: The extracted features F are transmitted to the semantic encoder for semantic encoding, and further converted into a semantic representation suitable for transmission in communication.
[0026] According to a preferred embodiment of the present invention, the fusion model uses a corresponding feature extractor to extract the semantic feature representation of each modality, which means: if the data is image data, it is input into a CNN neural network for feature extraction; if the data is text data, it is input into a Transformer neural network for feature extraction; the feature extraction process is shown in equation (3):
[0027] X = C α (S β (s)) (3)
[0028] In equation (3), S β (·) is a semantic encoder with parameter set β, and C α (·) is a channel encoder with parameter set α; s is the input sentence or image, and X is the encoded symbol;
[0029] The signal Y received at the receiver is shown in equation (4):
[0030] Y = HX + N (4)
[0031] In equation (4), Y represents the signal received after preliminary processing, H represents the channel gain between the transmitter and receiver, and N is the noise.
[0032] The received signal Y is decoded as shown in equation (5):
[0033]
[0034] In equation (5), The semantic information that needs to be recovered. It is a channel decoder with a parameter set δ; It is a semantic decoder with a parameter set χ; the superscript -1 indicates the decoding operation; the signal is obtained after feature extraction.
[0035] According to a preferred embodiment of the present invention, noise is added to the gradient information of the local model (the model of the local participants) to achieve differential privacy for federated learning, as shown in equation (6):
[0036]
[0037] In equation (6), These are the parameters after adding noise, F(n) is the local parameter model of the nth round, and z(n) is a random variable that follows a Gaussian distribution.
[0038] According to a preferred embodiment of the present invention, model training based on semantic information includes:
[0039] The gradient descent algorithm used in the local model training process is shown in equation (7):
[0040]
[0041] In equation (7), F represents semantic information, F(t) represents the local model parameters at step t, and L(F(t)) represents the loss function. Let η represent the gradient, and η be the learning rate.
[0042] According to a preferred embodiment of the present invention, after the model training is completed, global model aggregation and model evaluation are performed. During the aggregation process, federated averaging is performed to update the global model, as shown in equation (8):
[0043]
[0044] In equation (8), F global (t+1) is the next step parameter of the global model, F i (t) is the parameter of the i-th participant, w i It is the weight of participant i.
[0045] According to a preferred embodiment of the present invention, after the model evaluation meets the requirements, the modeling is completed and a digital twin model is generated. If the requirements are not met, the global model is distributed and retrained iteratively until the requirements are met. The mean squared error (MSE) is used to evaluate the model, as shown in equation (9):
[0046]
[0047] In equation (9), F true For the actual value, F pred is the predicted value, and n is the number of samples.
[0048] According to a preferred embodiment of the present invention, the generated digital twin model is shown in equation (10):
[0049] DT i =Γ(F1,F2,F3,…,F n (10)
[0050] In equation (10), F represents semantic information, and DT i This refers to a digital twin model, T(F1, F2, F3, ..., F...). n () refers to a model composed of semantic features.
[0051] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement steps of a semantic information federated learning method for digital twins in an industrial Internet of Things (IoT) scenario.
[0052] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a semantic information federated learning method for digital twins in an industrial Internet of Things (IoT) scenario.
[0053] A semantic information federated learning system for digital twins in an industrial Internet of Things (IIoT) scenario includes:
[0054] The data input selection module is configured to: input the data of the federated learning participants of the industrial IoT terminal devices into the multimodal neural network in the edge server for data input selection; the data of the federated learning participants of the industrial IoT terminal devices are images and text information collected by sensors;
[0055] The semantic information extraction module is configured to extract semantic information.
[0056] A differential privacy protection module has been added and configured to add differential privacy protection to the extracted semantic information.
[0057] The digital twin model generation module is configured to: train the model based on semantic information; after the model training is completed, perform global model aggregation and model evaluation, and repeat the training or proceed to the next step based on the evaluation accuracy; after the model evaluation meets the requirements, the digital twin model is generated.
[0058] The beneficial effects of this invention are as follows:
[0059] 1. Better model accuracy: Using semantic information helps to improve richer and higher-level feature representations, which can more accurately capture the behavior of entities or systems. The accuracy of digital twin models is crucial for simulating and predicting the state and behavior of entities or systems, so semantic information can provide more powerful modeling capabilities.
[0060] 2. Model Explanation and Interpretability: Semantic information helps generate more interpretable digital twin models, making the model's decisions easier to understand and explain, which can improve the model's credibility and usability.
[0061] 3. Model Collaboration: By using shared semantic information on different devices or systems, collaborative learning of digital twin models can be promoted, enabling models to better collaborate and optimize together in a federated learning environment. Attached Figure Description
[0062] Figure 1 A flowchart illustrating a federated learning method for semantic information in digital twins provided by this invention;
[0063] Figure 2 The flowchart of a semantic information federated learning method for digital twins provided by this invention is shown. Detailed Implementation
[0064] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.
[0065] Example 1
[0066] A federated learning method for semantic information of digital twins in an industrial IoT scenario, such as... Figure 1 and Figure 2 As shown, it includes:
[0067] The data of the participants in the federated learning of industrial IoT terminal devices is input into the multimodal neural network in the edge server for data input selection; the data of the participants in the federated learning of industrial IoT terminal devices consists of images and text information collected by sensors;
[0068] Extract semantic information;
[0069] Differential privacy protection is added to the extracted semantic information; considering privacy, security and semantic information consistency issues, semantic information is processed on the edge server and differential privacy technology is added for privacy protection.
[0070] Model training is performed based on semantic information;
[0071] After the model training is completed, global model aggregation and model evaluation are performed. Based on the evaluation accuracy, the training is repeated or the next step is initiated.
[0072] Once the model evaluation meets the requirements, the digital twin model is generated.
[0073] The resulting digital twin model is used to guide intelligent decision-making in the Industrial Internet of Things and accelerates communication efficiency through semantic information to achieve real-time dataset interaction.
[0074] Multimodal neural networks are convolutional neural networks.
[0075] Example 2
[0076] The semantic information federated learning method for digital twins in an industrial IoT scenario described in Example 1 differs in that:
[0077] A method is adopted to extract semantic information on the server. In this method, the raw data is kept on the local participant's device, while the semantic information extraction process is carried out on the edge server. The local participant uploads the raw data to the edge server, and the edge server extracts the semantic information from the data. This method can reduce communication overhead, and the uniform extraction of semantic information by the server can ensure the consistency of the extraction method.
[0078] Extract semantic information, including:
[0079] 1) Multimodal data input: Multimodal neural networks receive input data from different modalities; for example, they can simultaneously accept image, text and audio inputs, each with different feature representations.
[0080] 2) Modal feature extraction: For each input modality, the fusion model uses the corresponding feature extractor (e.g., CNN neural network for images, Transformer neural network for text) to extract the semantic feature representation of each modality; relevant features are extracted from the source message, as shown in Equation (1):
[0081] F = σ(W) T S+b T (1)
[0082] In equation (1), F represents the semantic features transmitted from the industrial IoT terminal device to the edge server, and W represents the semantic features transmitted from the terminal device to the edge server. T and b T These are the parameters from which features are extracted from the source message S, and σ(·) is the sigmoid activation function;
[0083] 3) Feature transmission: The extracted features F are transmitted to the semantic encoder for semantic encoding, and further converted into a semantic representation suitable for transmission in communication.
[0084] The fusion model uses the corresponding feature extractor to extract the semantic feature representation of each modality, which means: if the data is image data, it is input into the CNN neural network for feature extraction; if the data is text data, it is input into the Transformer neural network for feature extraction; the feature extraction process is shown in equation (3):
[0085] X = C α (S β (s)) (3)
[0086] In equation (3), S β (·) is a semantic encoder with parameter set β, and C α (·) is a channel encoder with a parameter set α; the semantic encoder is responsible for converting information into a semantic representation suitable for transmission in communication. This process typically involves converting raw information (such as text or images) into a vector representing its semantic content. The channel encoder is responsible for converting semantically represented information into a form suitable for transmission over a specific communication channel. This process involves encoding the information to improve the reliability and efficiency of transmission over the channel. s is the input sentence or image, and X is the encoded symbol;
[0087] In this invention, wireless transmission is used, and the signal Y received at the receiver is as shown in equation (4):
[0088] Y = HX + N (4)
[0089] In equation (4), Y represents the signal received after initial processing, H represents the channel gain between the transmitter and receiver, and N is the noise. The transmitter is responsible for transmitting information from the source to the target, converting the electrical signal into a wireless signal suitable for propagation in the air. It typically includes a modulator, a power amplifier, and an antenna. The electrical signal is modulated by the modulator to carry information within the signal. The power amplifier amplifies the modulated signal and then radiates it through the antenna, which is called a wireless signal. The receiver typically includes an antenna, a low-noise amplifier, and a demodulator. After receiving the signal, it amplifies the signal and reduces noise. The demodulator demodulates the signal, converting it back into an electrical signal, and then performs subsequent processing to finally extract the carried information.
[0090] The received signal Y is decoded as shown in equation (5):
[0091]
[0092] In equation (5), The semantic information that needs to be recovered. It is a channel decoder with a parameter set δ; the channel decoder is responsible for decoding the encoded information that has passed through the channel and restoring the original information; It is a semantic decoder with a parameter set χ; the semantic decoder decodes the encoded semantic information received from the communication channel into components that are closer to the original semantic content; the superscript -1 indicates the decoding operation; the signal is obtained after feature extraction.
[0093] To achieve differential privacy in federated learning, noise is added to the gradient information of the local model (the model of the local participants), as shown in Equation (6):
[0094]
[0095] In equation (6), These are the parameters after adding noise, F(n) is the local parameter model of the nth round, and z(n) is a random variable that follows a Gaussian distribution.
[0096] Model training based on semantic information includes:
[0097] The gradient descent algorithm used in the local model training process is shown in equation (7):
[0098]
[0099] In equation (7), F represents semantic information, F(t) represents the local model parameters at step t, and L(F(t)) represents the loss function. Let η represent the gradient, and η be the learning rate.
[0100] After the model training is completed, global model aggregation and model evaluation are performed. During the aggregation process, federated averaging is performed to update the global model. The global model is formed by summing up the local model parameter updates of each participant, as shown in equation (8).
[0101]
[0102] In equation (8), F globaｌ (t+1) is the next step parameter of the global model, F i (t) is the parameter of the i-th participant, w i It is the weight of participant i.
[0103] Once the model evaluation meets the requirements, the modeling is completed and a digital twin model is generated. If the requirements are not met, the global model is distributed and retrained iteratively until the requirements are met. In this invention, the mean squared error (MSE) is used to evaluate the model, as shown in equation (9):
[0104]
[0105] In equation (9), F true For the actual value, F predis the predicted value, and n is the number of samples.
[0106] The generated digital twin model is shown in equation (10):
[0107] DT i =Γ(F1,F2,F3,…,F n (10)
[0108] In equation (10), F represents semantic information, and DT i This refers to a digital twin model, T(F1, F2, F3, ..., F...). n () refers to a model composed of semantic features.
[0109] Each manufacturing equipment's digital twin contains semantic information about its status, performance characteristics, and operating conditions. By semantically modeling this information to describe the equipment's attributes and relationships, a semantically rich description can be created for each equipment. Each manufacturing equipment maintains its local digital twin and collaboratively trains the model using a federated learning framework. Federated learning algorithms allow equipment to train models locally and then share the model parameters with the global model via encryption or other secure means, enabling global model optimization without centralized data aggregation. Based on semantic information-based federated learning, the global model can optimize the performance of the entire digital twin system while taking semantic information into account. The optimization results can be fed back to each equipment, thereby improving the overall system performance.
[0110] Example 3
[0111] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the semantic information federated learning method for digital twins in the industrial Internet of Things scenario described in Embodiment 1 or 2.
[0112] Example 4
[0113] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the semantic information federated learning method for digital twins in the industrial Internet of Things scenario described in Embodiment 1 or 2.
[0114] Example 5
[0115] A semantic information federated learning system for digital twins in an industrial Internet of Things (IIoT) scenario includes:
[0116] The data input selection module is configured to: input the data of the federated learning participants of the industrial IoT terminal devices into the multimodal neural network in the edge server for data input selection; the data of the federated learning participants of the industrial IoT terminal devices are images and text information collected by sensors;
[0117] The semantic information extraction module is configured to extract semantic information.
[0118] A differential privacy protection module is added and configured to add differential privacy protection to the extracted semantic information. Considering privacy, security and semantic information consistency issues, semantic information is processed on the edge server and differential privacy technology is added for privacy protection.
[0119] The digital twin model generation module is configured to: train the model based on semantic information; after training, perform global model aggregation and evaluation, and repeat training or proceed to the next step based on the evaluation accuracy; once the model evaluation meets the requirements, the digital twin model is generated. The final generated digital twin model is used to guide intelligent decision-making in the Industrial Internet of Things (IIoT) and accelerates communication efficiency through semantic information to achieve real-time dataset interaction. The multimodal neural network is a convolutional neural network.
Claims
1. A semantic information federated learning method for digital twins in an industrial Internet of Things (IIoT) scenario, characterized in that, include: Data from federated learning participants in industrial IoT terminal devices is input into a multimodal neural network in an edge server for data input selection; The data from participants in federated learning for industrial IoT terminal devices consists of images and text information collected by sensors. Extract semantic information; Add differential privacy protection to the extracted semantic information; Model training is performed based on semantic information; After the model training is completed, global model aggregation and model evaluation are performed. Based on the evaluation accuracy, the training is repeated or the next step is initiated. Once the model evaluation meets the requirements, the digital twin model will be generated. Multimodal neural networks are convolutional neural networks; The raw data is kept on the local participant's device, while the semantic information extraction process is carried out on the edge server. The local participant uploads the raw data to the edge server, and the edge server extracts semantic information from the data. The fusion model uses corresponding feature extractors to extract semantic feature representations for each modality. This means: if the data is image data, it is input into a CNN neural network for feature extraction; if the data is text data, it is input into a Transformer neural network for feature extraction. The extraction process is shown in equation (3): （3） In equation (3), It has a parameter set The semantic encoder, and It has a parameter set Channel encoder; It is the input sentence or image. It is an encoded symbol; Signal received at the receiver As shown in equation (4): （4） In equation (4), Y represents the signal received after preliminary processing, H represents the channel gain between the transmitter and receiver, and N is the noise. For the received signal Decoding is performed as shown in equation (5): （5） In equation (5), The semantic information that needs to be recovered. It has a parameter set Channel decoder; It has a parameter set The semantic decoder; the superscript -1 indicates the decoding operation; the signal is obtained after feature extraction. .
2. The semantic information federated learning method for digital twins in an industrial IoT scenario according to claim 1, characterized in that, Extract semantic information, including: 1) Multimodal data input: Multimodal neural networks receive input data from different modalities; 2) Modal feature extraction: For each input modality, the fusion model uses the corresponding feature extractor to extract the semantic feature representation of each modality; relevant features are extracted from the source message, as shown in Equation (1): （1） In equation (1), These are the semantic features transmitted from industrial IoT terminal devices to edge servers. and These are parameters used to extract features from the source message S. It is the sigmoid activation function; 3) Feature transfer: Transferring the extracted features The data is transmitted to a semantic encoder for semantic encoding, and then further converted into a semantic representation suitable for transmission in communication.
3. The semantic information federated learning method for digital twins in an industrial IoT scenario according to claim 1, characterized in that, To achieve differential privacy in federated learning, noise is added to the gradient information of the local model, as shown in Equation (6): （6） In equation (6), These are the parameters after adding noise. It is the first The local parameter model of the wheel, It is a random variable that follows a Gaussian distribution.
4. The semantic information federated learning method for digital twins in an industrial IoT scenario according to claim 1, characterized in that, Model training based on semantic information includes: The gradient descent algorithm used in the local model training process is shown in equation (7): （7） In equation (7), F represents semantic information, F(t) represents the local model parameters at step t, and L(F(t)) represents the loss function. Represents the gradient. This is the learning rate.
5. The semantic information federated learning method for digital twins in an industrial IoT scenario according to claim 1, characterized in that, After the model training is completed, global model aggregation and model evaluation are performed. During the aggregation process, federated averaging is performed to update the global model, as shown in Equation (8): （8） In equation (8), These are the next parameters for the global model. It is the parameter of the i-th participant. It is the weight of participant i; Once the model evaluation meets the requirements, the modeling is completed and a digital twin model is generated. If the requirements are not met, the global model is distributed and retrained iteratively until the requirements are met. The mean squared error (MSE) is used to evaluate the model, as shown in equation (9): （9） In equation (9), This is the actual value. The value is the predicted value, and n is the number of samples. The generated digital twin model is shown in equation (10): （10） In equation (10), F represents semantic information. It refers to digital twin models. It refers to a model composed of semantic features.
6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the semantic information federated learning method for digital twins in the industrial Internet of Things scenario as described in any one of claims 1-5.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the semantic information federated learning method for digital twins in the industrial Internet of Things scenario as described in any one of claims 1-5.
8. A semantic information federated learning system for digital twins in an industrial Internet of Things (IoT) scenario, characterized in that, include: The data input selection module is configured to: input the data of the federated learning participants of the industrial IoT terminal devices into the multimodal neural network in the edge server for data input selection; the data of the federated learning participants of the industrial IoT terminal devices are images and text information collected by sensors; The semantic information extraction module is configured to extract semantic information. A differential privacy protection module has been added and configured to add differential privacy protection to the extracted semantic information. The digital twin model generation module is configured to: train the model based on semantic information; after the model training is completed, perform global model aggregation and model evaluation, and repeat the training or proceed to the next step based on the evaluation accuracy; after the model evaluation meets the requirements, the digital twin model is generated. The raw data is kept on the local participant's device, while the semantic information extraction process is carried out on the edge server. The local participant uploads the raw data to the edge server, and the edge server extracts semantic information from the data. The fusion model uses corresponding feature extractors to extract semantic feature representations for each modality. This means: if the data is image data, it is input into a CNN neural network for feature extraction; if the data is text data, it is input into a Transformer neural network for feature extraction. The extraction process is shown in equation (3): （3） In equation (3), It has a parameter set The semantic encoder, and It has a parameter set Channel encoder; It is the input sentence or image. It is an encoded symbol; Signal received at the receiver As shown in equation (4): （4） In equation (4), Y represents the signal received after preliminary processing, H represents the channel gain between the transmitter and receiver, and N is the noise. For the received signal Decoding is performed as shown in equation (5): （5） In equation (5), The semantic information that needs to be recovered. It has a parameter set Channel decoder; It has a parameter set The semantic decoder; the superscript -1 indicates the decoding operation; the signal is obtained after feature extraction. .