Privacy protection type cross-language feature fusion semantic communication system training method

Abstract

The invention relates to a privacy protection type cross-language feature fusion semantic communication system training method, and belongs to the technical field of communication. The method comprises the following steps: S1, initializing a compound word sequence index and a model of a communication user side; s2, performing personalized training of the text semantic communication system according to the language of the local user; s3, after the local user is trained for a certain round, uploading local language enhanced model parameters based on cross-language feature fusion; s4, cross-language feature fusion of different language text users is carried out on the server, and cross-language feature fusion model parameters are issued to the users; and S5, repeating the steps S2 to S4 until a specific aggregation round, thereby obtaining the cross-language feature fusion multi-language semantic communication system. According to the method, the multi-language function of the text semantic communication system can be effectively expanded through the compound word sequence index on the premise that the model scale is not expanded at all, and meanwhile the data privacy of a user side can be protected.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology and relates to a privacy-preserving cross-language feature fusion semantic communication system training method. Background Art

[0002] Semantic communication is considered as the second level of communication, which relies on the semantic knowledge shared between the transmitter and the receiver. With the integration of artificial intelligence technology and communication systems, semantic communication has surpassed traditional communication in efficiency and performance, and is expected to become a new communication paradigm. Text data widely carries the semantic information of human communication, and some research on semantic communication focuses on the text modality. However, most of the current research is limited to one language, and the models designed can only transmit one language, which is incompatible with the real environment of multiple languages. On the other hand, in practical training, users may be reluctant to share their text data, which creates obstacles for traditional training methods. Therefore, it is of practical significance to design a text semantic communication system that can transmit multiple languages ​​while protecting the privacy of users with different language backgrounds.

[0003] Federated learning is a very promising training method that can train a global model for all users' devices without obtaining the user's own data, thereby protecting data privacy. However, there is no method for using federated learning for training in existing semantic communication systems. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a privacy-preserving cross-language feature fusion semantic communication system training method, which uses federated learning for training to achieve multilingual semantic transmission without expanding the model scale and protecting user data privacy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A privacy-preserving cross-language feature fusion semantic communication system training method specifically includes the following steps:

[0007] S1: Initialize the compound word sequence index and model of the communication client;

[0008] S2: Conduct local language personalized training of the text semantic communication system based on the language of the local user;

[0009] S3: After the local user has trained for a specific round, the local language enhanced network model parameters based on the cross-language feature fusion are obtained, and the user language coefficient of the local language enhanced network model parameters is adaptively adjusted according to the number of local languages;

[0010] S4: The cloud server receives the local language enhanced network model parameters from users of each language and performs cross-language feature fusion to obtain new cross-language feature fusion model parameters, and sends the parameters to local users of different languages;

[0011] S5: Repeat steps S2 to S4 until a limited number of rounds of cross-language feature fusion are completed to obtain a multilingual semantic communication system with cross-language feature fusion.

[0012] Further, step S1 specifically includes: the compound word sequence index established includes a group of word sequences and multiple groups of words in different languages, and the existing corpora in different languages ​​are processed according to the frequency of word occurrence to form a dictionary L i =[l 1 ,l 2 ,...,l N ] and the corresponding word sequence E = [e 1 ,e 2 ,...,e N ]; where L i represents the dictionary of the i-th language, l i represents the words in the dictionary of the i-th language, e j Represents a digital sequence, that is, the digital sequence e j Can correspond to different L at the same time i l i , that is, the sequence to word is a one-to-many compound mapping; according to the user's language usage, the corresponding language dictionary is issued; different language users have different language dictionaries but they share a set of compound word sequence indexes.

[0013] Further, step S2 specifically includes the following steps:

[0014] S21: Local users load the cross-language feature fusion model from the cloud server The model parameters are divided into two parts: the encoding module parameters at the transmitter and the decoding module parameters at the receiving end

[0015] S22: The input text data is first mapped through the compound word sequence index, that is, S i →E i , where i represents the language category;

[0016] S23: Local training first performs forward propagation inference at the transmitter, that is, T(e; w e )→x, where T represents the encoder, e represents the word sequence, and w eRepresents the encoder network model parameters; the encoded bit data x is sent and received in a simulated Gaussian channel. Due to the existence of channel noise, the received bit data is represented by y, and then the decoding operation is performed at the receiving end, that is, Among them, R represents the decoder, represents the word sequence received by the receiver, w d Represents the decoder network model parameters; finally, reverse mapping is performed through the compound word sequence index: At this point, forward reasoning is complete;

[0017] S24: Before the back propagation training phase begins, the loss function is first calculated. The loss function used by the local user is the cross entropy loss function, which can be expressed as:

[0018]

[0019] Among them, local users of different languages ​​correspond to different i, q (s i ) represents the i-th word s i The true probability of sending sentence s, p(s i ) represents the i-th word s i The predicted probability in sentence s; w e represents the encoder network model parameters, w d represents the decoder model parameters, They represent the sentences sent by the sender and the sentences received by the receiver during the training process respectively;

[0020] S25: Use the SGD (stochastic gradient descent) optimizer for back propagation and update the parameters of the local language enhanced network model based on multilingual feature fusion Different users of different languages ​​perform forward reasoning and back propagation at the same time. Due to the different language characteristics, the language features they learn are also different. The local language enhanced network model parameters output by each user are personalized.

[0021] Furthermore, in step S4, the cross-language feature fusion of the cloud server specifically includes: since different users have different language data, it is necessary to adaptively adjust the language coefficient of the user uploaded model according to the number of languages, so the cross-language feature fusion model can be expressed as:

[0022]

[0023] Among them, M represents the number of languages ​​in the network, N represents the number of users of the same language; α represents the language coefficient, |D| represents the number of sentences in all languages, and |D ij | represents the number of sentences in user ij; represents the local language enhancement network model parameters of user ij, which is composed of the encoder parameters of the user and decoder parameters constitute; Represents the parameters of a new cross-language feature fusion model.

[0024] Furthermore, in step S4, the objective function of cross-language feature fusion is expressed as:

[0025]

[0026] Among them, f ij (·) represents the loss function of the cross-language feature fusion model under the local data of user ij. The purpose of each cross-language feature fusion can be expressed as: finding the minimum value of the sum of the loss functions of the multilingual text semantic communication system under users of each language, that is, making the system better adapted to multilingual transmission.

[0027] Further, step S5 specifically includes: repeating steps S2 to S4, each time the number of iterations is increased by 1, until the number of iterations reaches the set value and the MBLEU score reaches the expected value, stopping the training iteration, and outputting the last round of cross-language model parameters

[0028]

[0029] Among them, BLEU i is the BLEU score of the i-th language transmitted in the system, and MBLEU is weighted summation based on the language coefficient α in the network; represents the length of the received sentence, L represents the length of the sent sentence, N represents the comparison in groups of several words, and p n Indicates precision.

[0030] The beneficial effects of the present invention are:

[0031] (1) The present invention integrates cross-language features of the semantic communication system of text users in different languages, thereby expanding the system from an original single-language text semantic communication system to a multi-language text semantic communication system.

[0032] (2) The present invention utilizes a set of compound word sequence indexes, without increasing the length of the word book at all, so that the model can achieve the purpose of cross-language feature fusion without expanding.

[0033] (3) The present invention achieves the transmission of multilingual text by only uploading local enhanced language model parameters for cross-language feature fusion, thereby protecting the privacy of local user data.

[0034] (4) The present invention designs an evaluation index for multilingual text semantic transmission, which can better reflect the multilingual transmission capability of the system and is consistent with the practical effect.

[0035] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 A flowchart of the privacy-preserving cross-language feature fusion semantic communication system training method proposed by the present invention;

[0038] Figure 2 A system structure diagram of the method proposed in the present invention;

[0039] Figure 3 This is a structural diagram of the multilingual text semantic communication system obtained by the method proposed in the present invention;

[0040] Figure 4 It is a training loss curve diagram of the method proposed in the present invention;

[0041] Figure 5 It is a performance diagram of the text transmission capability of different languages ​​during the training process of the method proposed in the present invention;

[0042] Figure 6 The multilingual transmission scores of the proposed method and the comparative method under different signal-to-noise ratios when the text sentences in different languages ​​are evenly distributed;

[0043] Figure 7 It is the multilingual transmission score of the proposed method and the comparative method under different signal-to-noise ratios when the text sentences in different languages ​​are unevenly distributed. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] See also Figure 1 to Figure 7 ,The present invention proposes a privacy-preserving cross-language feature fusion semantic communication system training method, such as Figure 1 As shown, the specific steps include:

[0046] S1: Establish a set of compound word sequence indexes based on text data of users of different languages ​​[L 1 ,L 2 ,...,L N ]→E, and the initial featureless model It is distributed to each user terminal according to the user language.

[0047] In step S1, a set of compound word sequence indexes is established, and the process is as follows:

[0048] (1) The compound word sequence index established includes a set of word sequences and multiple groups of words in different languages. The existing corpora in different languages ​​are segmented according to the frequency of word occurrence to form a dictionary L i =[l 1 ,l 2 ,...,l N ] and the corresponding word sequence E = [e 1 ,e 2 ,...,e N ]; where L i represents the dictionary of the i-th language, l i represents the words in the dictionary of the i-th language, e j It represents a sequence of numbers.

[0049] (2) According to the specific mapping relationship, different L i can be mapped to the word sequence E, thus forming a set of compound word sequence indexes. According to the usage of the user's language, the corresponding language dictionary is issued. The language dictionaries of users of different languages ​​are different, but they share a set of compound word sequence indexes.

[0050] S2: Local users use compound word sequence indexes to adaptively convert text sentences into sequences based on their own language indexes, and use local language text to adjust the parameters of the cross-language fusion network model. (When t=0, it is the initial featureless model parameter) Personalized training for the local language is performed, and the semantic transmission capability of the local language is further enhanced based on the cross-language model parameters sent from the cloud.

[0051] In step S2, the process of local users training a single language text semantic communication system is as follows:

[0052] (1) Local users load cross-language feature fusion model parameters from the cloud server The model parameters are divided into two parts: the encoding module parameters w at the transmitter and e and the decoding module parameter w at the receiving end d .

[0053] (2) The input personalized text data is first mapped through the compound word sequence index, that is, S i →E i , where i represents the language category;

[0054] (3) Local training first performs forward propagation inference at the transmitter, that is, T(e; w e )→x, where T represents the encoder, e represents the word sequence, and w e Represents the encoder network model parameters; the encoded bit data x is sent and received in a simulated Gaussian channel. Due to the existence of channel noise, the received bit data is represented by y, and then the decoding operation is performed at the receiving end, that is, Among them, R represents the decoder, represents the word sequence received by the receiver, w d Represents the decoder network model parameters; finally, reverse mapping is performed through the compound word sequence index: At this point, forward reasoning is complete.

[0055] (4) Before the back-propagation training phase begins, the loss function is first calculated. The loss function used by the local user is the cross entropy loss function, which can be expressed as:

[0056]

[0057] Local users of different languages ​​correspond to different i, q (s i ) represents the i-th word s i The true probability of sending sentence s, p(s i ) represents the i-th word s i The predicted probability in sentence s; w e represents the encoder network model parameters, w d represents the decoder model parameters, They represent the sentences sent by the sender and the sentences received by the receiver during the training process respectively;

[0058] (5) Use the SGD optimizer for back propagation and update the local enhanced model parameters based on multilingual feature fusion Different users of different languages ​​perform forward reasoning and back propagation at the same time. Due to the different language characteristics, the language features they learn are also different. The local language enhanced model parameters output by each user are personalized.

[0059] S3: After reaching a certain round of local training, the local language enhanced network model parameters based on cross-language fusion are obtained And adaptively adjust the parameters of the local language enhanced personalized network model according to the number of local languages User language coefficient α ij ;

[0060] S4: The cloud server receives the local language enhanced personalized network model parameters from users of each language and performs cross-language feature fusion to obtain new cross-language feature fusion model parameters. And send the parameter to local users of different languages.

[0061] In step S4, the process of cross-language feature fusion in the cloud server is as follows:

[0062] (1) Since different users have different language data, it is necessary to adaptively adjust the language coefficient of the user-uploaded model according to the number of languages. Therefore, the cross-language fusion model can be expressed as:

[0063]

[0064] Where, represents the number of languages ​​in the network, N represents the number of users of the same language; α represents the language coefficient, |D| represents the number of sentences in all languages, |D ij | represents the number of sentences in user ij; represents the local language enhanced model parameters of user ij, which is composed of the encoder parameters of the user and decoder parameters constitute; Represents the parameters of a new cross-language feature fusion model.

[0065] (2) The purpose of cross-language feature fusion is specifically expressed as follows:

[0066]

[0067] Among them, f ij(·) represents the loss function of the cross-language feature fusion model under the local data of user ij. The purpose of each cross-language feature fusion can be expressed as: finding the minimum value of the sum of the loss functions of the multilingual text semantic communication system under users of each language, that is, making the system better adapted to multilingual transmission.

[0068] S5: Repeat steps S2 to S4 until a limited number of rounds of cross-language feature fusion have passed, and then stop this step. Thus, without the need for users to upload data, a semantic communication system capable of transmitting multiple languages ​​is obtained through a cross-language feature fusion training mechanism.

[0069] In step S5, in terms of the performance evaluation of multilingual text transmission, an evaluation parameter for the multilingual text semantic communication system is designed, specifically:

[0070]

[0071] Among them, BLEU i is the BLEU score of the i-th language transmitted in the system, and MBLEU is weighted summed according to the language coefficient α in the network. In the expression of BLEU score, represents the length of the received sentence, L represents the length of the sent sentence, N represents the comparison in groups of several words, and p n Indicates precision.

[0072] Repeat steps S2, S3, and S4. Each time the number of iterations is increased by 1, until the number of iterations reaches the set value and the MBLEU score reaches the expected value, stop the training iteration, and output the parameters of the last round of cross-language model.

[0073] Comparative experiment:

[0074] The present invention lists a typical embodiment, such as Figure 2 As shown, users of three different languages ​​perform semantic communication system training with cross-language feature fusion, and their data are stored locally, and compound word sequence indexes are issued. Local enhanced model parameters are uploaded after every two rounds of training, and the number of cross-language feature fusion rounds of the cloud server is set to 80. Furthermore, the present invention performs simulation analysis in the case of multilingual text users by using the proposed method;

[0075] like Figure 4 As shown, the present invention gives a training loss function curve diagram of a multilingual text semantic communication system based on federated learning trained using the Chinese, English and German data sets in the WMT18 data set, records the training convergence process, and shows that with the increase of iteration rounds, the training loss gradually decreases. When the iteration rounds are greater than 60, the training loss stabilizes at around 0.2. Figure 5The graph shows how the system's ability to transfer texts in three languages ​​changes with the number of iterations during training.

[0076] Figure 6 The figure shows the comparison of the multilingual evaluation score (MBLEU) of the model trained by the multilingual text semantic communication system based on federated learning proposed in the present invention and other methods under different signal-to-noise ratios when the text data in different languages ​​are evenly distributed. Figure 6 It can be seen that in low signal-to-noise comparison, the multilingual text semantic communication system training method based on federated learning proposed in the present invention is superior to other training methods. This proves the superiority of the proposed method on multilingual text data;

[0077] Figure 7 The figure shows the comparison of the multilingual evaluation score (MBLEU) of the model trained by the multilingual text semantic communication system based on federated learning proposed in the present invention and other methods under different signal-to-noise ratios when the text data in different languages ​​are unevenly distributed. Figure 7 It can be seen that the method proposed by the present invention is still superior to other solutions.

[0078] comprehensive Figure 6 and Figure 7 Analysis shows that the multilingual text semantic communication system training method based on federated learning proposed in the present invention can effectively expand the usage scenarios of the text semantic communication system and achieve superior model performance.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A privacy-preserving cross-language feature fusion semantic communication system training method, characterized in that: The method specifically comprises the following steps: S1: Initialize the compound word sequence index and model of the communication client; S2: Conduct local language personalized training of the text semantic communication system based on the language of the local user; S3: After the local user has trained for a specific round, the local language enhanced network model parameters based on the cross-language feature fusion are obtained, and the user language coefficient of the local language enhanced network model parameters is adaptively adjusted according to the number of local languages; S4: The cloud server receives the local language enhanced network model parameters from users of each language and performs cross-language feature fusion to obtain new cross-language feature fusion model parameters, and sends the parameters to local users of different languages; S5: Repeat steps S2 to S4 until a limited number of rounds of cross-language feature fusion are completed to obtain a multilingual semantic communication system with cross-language feature fusion.

2. The semantic communication system training method according to claim 1, characterized in that: Step S1 specifically includes: establishing a compound word sequence index including a group of word sequences and multiple groups of words in different languages, processing the existing corpora in different languages ​​according to the frequency of word occurrence, and forming a dictionary L i =[l1,l2,...,l N ] and the corresponding word sequence E = [e1,e2,...,e N ]; where L i represents the dictionary of the i-th language, l i represents the words in the dictionary of the i-th language, e j Represents a digital sequence, that is, e j At the same time, it corresponds to different L i l i , that is, the sequence to word is a one-to-many compound mapping; according to the user's language usage, the corresponding language dictionary is issued; different language users have different language dictionaries but they share a set of compound word sequence indexes.

3. The semantic communication system training method according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21: Local users load the cross-language feature fusion model from the cloud server The model parameters are divided into two parts: the encoding module parameters at the transmitter and the decoding module parameters at the receiving end S22: The input text data is first mapped through the compound word sequence index, that is, S i →E i , where i represents the language category; S23: Local training first performs forward propagation inference at the transmitter, that is, T(e; w e )→x, where T represents the encoder, e represents the word sequence, and w e Represents the encoder network model parameters; the encoded bit data x is sent and received in a simulated Gaussian channel. Due to the existence of channel noise, the received bit data is represented by y, and then the decoding operation is performed at the receiving end, that is, Among them, R represents the decoder, represents the word sequence received by the receiver, w d Represents the decoder network model parameters; finally, reverse mapping is performed through the compound word sequence index: At this point, forward reasoning is complete; S24: Before the back propagation training phase begins, the loss function is first calculated. The loss function used by the local user is the cross entropy loss function, which is expressed as: Among them, local users of different languages ​​correspond to different i, q (s i ) represents the i-th word s i The true probability of sending sentence s, p(s i ) represents the i-th word s i The predicted probability in sentence s; w e represents the encoder network model parameters, w d represents the decoder model parameters, s, They represent the sentences sent by the sender and the sentences received by the receiver during the training process respectively; S25: Use SGD optimizer for back propagation and update the parameters of the local language enhanced network model based on multilingual feature fusion Different users of different languages ​​perform forward reasoning and back propagation at the same time. Due to the different language characteristics, the language features they learn are also different. The local language enhanced network model parameters output by each user are personalized.

4. The semantic communication system training method according to claim 3, characterized in that: In step S4, the cross-language feature fusion of the cloud server specifically includes: adaptively adjusting the language coefficient of the model uploaded by the user according to the number of languages. The cross-language feature fusion model is expressed as: Among them, M represents the number of languages ​​in the network, N represents the number of users of the same language; α represents the language coefficient, |D| represents the number of sentences in all languages, and |D ij | represents the number of sentences in user ij; represents the local language enhancement network model parameters of user ij, which is composed of the encoder parameters of the user and decoder parameters constitute; Represents the parameters of a new cross-language feature fusion model.

5. The semantic communication system training method according to claim 4, characterized in that: In step S4, the objective function of cross-language feature fusion is expressed as: Among them, f ij (·) represents the loss function of the cross-language feature fusion model under the local data of user ij. The purpose of each cross-language feature fusion is to find the minimum value of the sum of the loss functions of the multilingual text semantic communication system under users of each language.

6. The semantic communication system training method according to claim 4, characterized in that: Step S5 specifically includes: repeating steps S2 to S4, with the number of iterations increasing by 1 each time, until the number of iterations reaches the set value and the MBLEU score reaches the expected value, stopping the training iteration, and outputting the last round of cross-language model parameters Among them, BLEU i is the BLEU score of the i-th language transmitted in the system, and MBLEU is weighted summation based on the language coefficient α in the network; represents the length of the received sentence, L represents the length of the sent sentence, N represents the comparison in groups of several words, and p n Indicates precision.

Images