Distributed data processing method and device based on privacy protection

Abstract

The invention relates to the field of data processing, and discloses a distributed data processing method and device based on privacy protection, and the method comprises the steps: obtaining input data and output data; preprocessing the input data to obtain preprocessed input data; determining a residual difference based on the preprocessed input data; determining a language model based on the preprocessed input data, the output data and the residual difference; training a language model based on the loss function, and determining initial model parameters of the language model; and obtaining an updated model parameter to update the language model, and processing the new preprocessed input data by using the updated language model to obtain an output result. Through the distributed parallel data processing technology, the operation process is accelerated, the calculation time and resource consumption can be remarkably reduced, the data processing efficiency is improved, the safety risk of private data in the social field in the centralized storage and processing process is reduced, data leakage is prevented, and privacy protection of the data is enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a distributed data processing method and device based on privacy protection. Background Art

[0002] Private user data is often superior to public data in terms of quantity and quality. The reason is that private data sets usually contain more comprehensive and detailed features and follow stricter standards during data collection. For example, in the fields of medicine, finance, and social networks, private data can provide more personalized training materials.

[0003] Nevertheless, the use of private data in the social field also faces huge challenges. Private data in the social field in the prior art is usually stored in isolated data silos. For example, user data on mobile devices is stored on the local device, involving a large amount of personal privacy information. This makes it an important challenge to safely obtain and use these private data without infringing on the user's privacy rights. The centralized storage of private data in the social field not only violates the laws and regulations on data privacy protection, but also may lead to the risk of data leakage, posing a hidden danger to the user's personal safety. Therefore, it is necessary to propose a new data processing method for achieving privacy protection, which is used to reduce the risk of data leakage and ensure the privacy and security of user data.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a distributed data (for example, multimedia data) processing method and device based on privacy protection. Through distributed parallel processing, the calculation process is accelerated, the computing time and resource consumption can be significantly reduced, the data processing efficiency is improved, and the security risks of data in the social field during centralized storage and processing are reduced, which effectively prevents data leakage and information theft, and ensures the privacy and security of user data.

[0006] In a first aspect, an embodiment of the present disclosure provides a distributed data processing method based on privacy protection, the method comprising:

[0007] Acquire input data of a client and output data corresponding to the input data, wherein the input data is multimedia data including one or more of pictures, texts, voices and videos;

[0008] Preprocessing the input data to obtain preprocessed input data, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1;

[0009] Determine a residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1;

[0010] determining a language model based on the preprocessed input data, output data corresponding to the input data, and the residual difference;

[0011] Training the language model based on the loss function to determine initial model parameters of the language model;

[0012] Updating the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters;

[0013] The language model is updated based on the updated model parameters, and the new preprocessed input data is processed using the updated language model to obtain an output result.

[0014] In a second aspect, an embodiment of the present disclosure further provides a distributed data processing device based on privacy protection, the device comprising:

[0015] A client module, used to obtain input data of a client and output data corresponding to the input data, preprocess the input data to obtain preprocessed input data, wherein the input data is multimedia data including one or more of pictures, texts, voices and videos, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1;

[0016] A language model training module, used to determine the residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determine the language model based on the preprocessed input data, the output data corresponding to the input data and the residual difference; train the language model based on the loss function to determine the initial model parameters of the language model;

[0017] A server module, configured to update the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters;

[0018] The result acquisition module is used to update the language model based on the updated model parameters, and use the updated language model to process new pre-processed input data to obtain an output result.

[0019] In the third aspect, an embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the distributed data processing method based on privacy protection as described above.

[0020] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed data processing method based on privacy protection as described above.

[0021] The distributed data processing method based on privacy protection provided by the embodiment of the present disclosure includes: obtaining the input data of the client and the output data corresponding to the input data; preprocessing the input data to obtain the preprocessed input data, wherein the input data is multimedia data including one or more of pictures, texts, voices and videos, wherein the preprocessed input data includes the preprocessed input data at time t and the preprocessed input data at time t-1; based on the preprocessed input data at time t and the preprocessed input data at time t-1, determining the residual difference; determining the language model based on the preprocessed input data, the output data corresponding to the input data and the residual difference; training the language model based on the loss function to determine the initial model parameters of the language model; updating the initial model parameters based on the model parameters and the residual difference to obtain the updated model parameters; updating the language model based on the updated model parameters, and using the updated language model to process the new preprocessed input data to obtain the output result. The present disclosure reduces the security risks of private data in the social field in the centralized storage and processing process through distributed processing technology, effectively prevents data leakage and information theft, and enhances the privacy protection of user data. At the same time, this method allows effective data collaborative processing while isolating data, solving the privacy protection problem of user data on local devices such as mobile devices. At the same time, the distributed processing method can process data in parallel on multiple clients, which speeds up the calculation process compared to traditional centralized server processing, significantly reduces computing time and resource consumption, and improves data processing efficiency, especially when processing large-scale private data sets. The distributed nature of this method allows it to be applied across fields, not only in the social field, but also in different fields. Private data can be effectively used under the premise of protecting privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 The present invention is a flowchart of a distributed data processing method based on privacy protection in an embodiment of the present invention.

[0024] Figure 2It is a structural diagram of a distributed data processing device based on privacy protection in an embodiment of the present disclosure.

[0025] Figure 3 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] In the context of how to reasonably use private data, the federated learning mechanism is introduced, so that the model can be trained locally on multiple user devices without uploading the user's private data to the central server. The core idea of ​​federated learning is to keep the data on the user's device and protect personal privacy by training the model locally. After the user's device is trained on its local data, it only needs to send update information such as model parameters and gradients to the server. The server aggregates these parameters to generate a global model without obtaining the user's original data. In this way, private data can be used to effectively update the model while ensuring the security of private data, thereby improving model performance. Therefore, with the help of the federated learning framework, not only can the risk of privacy leakage be effectively reduced, but also the model training process can be accelerated while retaining data privacy, ultimately achieving higher performance.

[0030] Traditional language model training methods based on federated learning frameworks have some significant shortcomings. For example, the computing resource requirements are high. Traditional federated learning frameworks usually require each client to have sufficient computing resources to train large language models with a large number of parameters. However, in many real scenarios, especially on mobile devices or in resource-constrained environments, it is difficult for clients to meet such high computing requirements. This limits the widespread application of large language models and prevents many potential users from participating in the model training process. Secondly, there is a security risk of privacy leakage. In the existing separation learning transformer framework, although most of the parameters are concentrated on the server and the client is mainly responsible for the training of the embedding layer and the output layer, the gradient of the embedding layer is still vulnerable to attack. This vulnerable nature allows attackers to reconstruct private data or perform reverse engineering by analyzing the data transmitted by the client using techniques such as beam search, thereby threatening the privacy security of users. Moreover, parallel training is inefficient. In the current federated learning training process, the server only processes the update request of one client at a time. This serial processing method significantly reduces the parallelism of training and results in low efficiency. In the context of the growing demand for data training, how to accelerate the training process and improve the overall efficiency of the system is an urgent problem to be solved. The gradients and losses generated by the client are calculated locally and then returned to the server, which then needs to integrate these updates and generate a global model. Due to the delay in this update process, the performance of the model may not reflect the changes in user needs in a timely manner, affecting the training effect. Based on the above problems, the present invention aims to propose a distributed data processing method based on privacy protection.

[0031] In view of the above problems, the embodiments of the present disclosure provide a distributed data processing method based on privacy protection, which allows effective data collaborative processing while isolating data, and solves the privacy protection problem of user data on local devices such as mobile devices. At the same time, the distributed processing method can process data in parallel, improving the efficiency of data processing.

[0032] Figure 1 : is a flowchart of a distributed data processing method based on privacy protection in an embodiment of the present disclosure. The method can be executed by a distributed data processing device based on privacy protection, which can be implemented in software and / or hardware, and can be configured in an electronic device. Figure 1 As shown, the method may specifically include the following steps:

[0033] S110: Obtain input data from the client and output data corresponding to the input data.

[0034] Specifically, the input data and the output data corresponding to the input data are user privacy data that can be stored locally on the client, and the data between clients can be shared. In this way, each client can process privacy data locally to protect the user's privacy. The client refers to the user's personal device, such as a smartphone, personal computer, tablet computer, etc., which stores the user's data. The input data can be multimedia data, including: the user's nickname, profile, gender, tag, number of fans, etc. on the social platform; messages sent by the user, including text, voice, pictures, videos, etc.; user interaction data, such as likes, comments, forwarding, etc.; hot topics or events, such as Weibo hot search list, but not limited to this, but can include any message or information that can be obtained through the social platform. The output data can be the feature encoding state of the input data. For example, assuming a medical health application, the user's smartphone stores personal health data (such as heart rate, blood pressure, etc.), these data are the input data, and the corresponding output data may be a doctor's health advice or an automatically generated health report type. Among them, the input data can be expressed as: , represents a data point in the input data, Represents the number of data points in the input data. The output data corresponding to the input data can be expressed as: It should be noted that the client stores data at different times. Taking time t as an example, the input data and output data at time t can be expressed as: , ,

[0035] Specifically, the client performs data processing locally, including data preprocessing, model training and updating, and processing of preprocessed input data, without sending the data to a remote server, thereby protecting the user's privacy. Through the allocation of servers, multiple clients can process data in parallel, speeding up the calculation process, significantly reducing computing time and resource consumption, and improving data processing efficiency.

[0036] This embodiment effectively prevents unauthorized requests for hidden states by obtaining input data and output data stored in the local client, thereby alleviating the risk of embedded gradient attacks and improving system security.

[0037] S120: Preprocess the input data to obtain preprocessed input data, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1.

[0038] Based on the above embodiment, it is necessary to first calculate the average value of the data points in the input data and the variance of the data points in the input data, and then use the average value and variance and the logistic distribution density function to calculate the pre-processed input data. Among them, the expression of the pre-processed input data is: ,

[0039] in, Represents preprocessing input data, Represents input data, Represents input data The mean of Represents input data The variance of represents the scaling factor, represents the logistic distribution density function, Represents the distribution function impact factor.

[0040] S130: Determine a residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1.

[0041] Further, based on the above embodiment, the preprocessed input data at time t and the preprocessed input data at time t-1 are obtained, and the residual difference is calculated by the preprocessed input data at time t and the preprocessed input data at time t-1, the aggregation factor and the reconciliation factor, wherein the expression of the residual difference is: ,

[0042] in, represents the residual difference, represents the preprocessed input data at time t, represents the preprocessed input data at time t-1, represents the harmonic factor, represents the first aggregation factor, represents the second aggregation factor, Indicates the number of clients.

[0043] S140, determining a language model based on the preprocessed input data, the output data corresponding to the input data, and the residual difference;

[0044] Optionally, a language model is constructed based on preprocessed input data, output data corresponding to the input data, residual differences, and model parameters. It should be noted that the model parameters are unknown at this time, and the model needs to be continuously trained to update the model parameters. The model parameters include a first model parameter and a second model parameter. The expression of the language model is: ,

[0045] in, Represents the output data corresponding to the preprocessed input data, Indicates the output data corresponding to the input data, represents the first model parameter, represents the second model parameter, represents the trigonometric factors, represents the phase factor, represents the transposed convolution operation, Represents the model weight factor.

[0046] S150: Train the language model based on the loss function to determine initial model parameters of the language model.

[0047] Specifically, a loss function may be constructed based on the output data corresponding to the input data and the output data output by the language model (ie, the output data corresponding to the preprocessed input data), wherein the expression of the loss function is: ,

[0048] in, represents the value of the loss function, Represents preprocessing input data, Indicates the output data corresponding to the input data, Represents the output data corresponding to the preprocessed input data, Represents preprocessing input data The number of data points in .

[0049] In an optional embodiment, the language model is trained based on a loss function to determine initial model parameters of the language model, including: optimizing the loss function so that the initial model parameters are obtained when the value of the loss function is minimized.

[0050] On the basis of the above embodiments, the loss function is optimized, including taking partial derivatives of the first model parameter and the second model parameter based on the loss function, and continuously optimizing the loss function to minimize and stabilize the value of the loss function, to obtain the initial model parameters. It should be noted that when taking partial derivatives of the initial model parameters of the loss function, the initial model parameters are unknown, and the known values ​​of the initial model parameters are obtained by taking partial derivatives and continuously optimizing. The specific partial derivative formula is as follows: , , , ,

[0051] This embodiment determines the initial model parameters of the language model through the loss function, thereby improving the efficiency of model optimization.

[0052] S160: Update the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters.

[0053] It can be understood that the initial model parameters are updated using the preprocessed input data and the residual difference to obtain updated model parameters. The updated model parameters can be stored on the server module and the updated model parameters can be encrypted by the server to reduce the risk of data leakage between the server and the client.

[0054] Specifically, the present disclosure can be implemented but not limited to the following:

[0055] The server is responsible for coordinating data sharing between clients and global updates of the model, which will not be described in detail here. Through the allocation of the server, multiple clients can process data in parallel, which speeds up the calculation process, significantly reduces computing time and resource consumption, and improves data processing efficiency.

[0056] Data sharing coordination: The server does not directly store user data, but coordinates data sharing between clients in a secure way to facilitate data processing.

[0057] Model update: The server can receive model update requests from clients, update the global model based on the information provided by the clients, and then distribute the updated model parameters to each client.

[0058] For example, the server may be a cloud service that does not directly access the user's health data, but can receive model update requests from multiple clients, and transmit model parameter updates back to each client to update the model and perform data processing on the client. In this way, each client can perform private data processing locally, while the server is responsible for coordinating and updating the global model, ensuring user privacy while also achieving effective training and updating of the model. This distributed processing method not only protects user privacy, but also improves the efficiency of data processing and the accuracy of the model.

[0059] S170: updating the language model based on the updated model parameters, and using the updated language model to process new preprocessed input data to obtain an output result.

[0060] Among them, the output result can be the output data corresponding to the pre-processed data of each client. For example, the output result of the first client can be text, pictures, and the output result of the second client can be voice, video, etc., but the present disclosure is not limited to this. Through distributed processing technology, the security risks of private data in the social field in the centralized storage and processing process are reduced, data leakage and information theft are effectively prevented, and the privacy protection of user data is enhanced. At the same time, the method allows effective data collaborative processing while isolating data, solving the privacy protection problem of user data on local devices such as mobile devices. At the same time, the distributed processing method can process data in parallel on multiple clients, improving the efficiency of data processing, especially when processing large-scale private data sets.

[0061] In an optional embodiment, updating the model parameters includes updating the first model parameters and updating the first model parameters, and updating the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters, including:

[0062] Based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the first model parameter, the first formula is used to determine and update the first model parameter;

[0063] Based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the second model parameters, the second formula is used to determine and update the second model parameters.

[0064] Specifically, on the basis of the above embodiment, the initial model parameters are updated based on the input data and preprocessed input data obtained at different times and the initial model parameters obtained after optimization by the loss function, and the updated model parameters are obtained by using the formula. Then, the input data and preprocessed input data at other times and the updated model parameters are used to update the updated model parameters using the formula to obtain new updated model parameters. The update is repeated in this way until the updated model parameters are stable, the model training is completed, and the final updated model parameters are obtained.

[0065] Exemplarily, taking time t as an example, based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the first model parameter, the first formula is used to determine the update of the first model parameter; based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the second model parameter, the second formula is used to determine the update of the second model parameter, wherein the expression of the first formula is: ,

[0066] The expression of the second formula is: ,

[0067] in, represents updating the first model parameter, represents updating the second model parameters, Represents the first model parameter The difference influencing factors, represents the second model parameter The difference influencing factors, represents the chi-square density distribution function.

[0068] The first formula and the second formula in this embodiment update the initial model parameters to obtain updated model parameters, and then use the first formula and the second formula to continuously optimize and update the model parameters, which can achieve parallel training of model parameters, thereby improving the speed and accuracy of model updating, and further improving the computing power of the model.

[0069] In an optional embodiment, the updated model parameters are retained on the server and encrypted to ensure the security of user privacy data.

[0070] This embodiment significantly reduces the risk of data leakage between the server and the client by retaining and encrypting the updated model parameters on the server. In addition, the client and the server jointly own and use the entire language model, further strengthening the overall security of the system. It effectively prevents unauthorized requests for hidden states, thereby mitigating the risk of embedded gradient attacks and improving system security.

[0071] It should be noted that the distributed language model can be deployed in a hierarchical server architecture, which is particularly suitable for cluster environments with multiple machines and GPUs. By initializing the sub-servers for parallel training, the central server is responsible for aggregating and distributing model updates, which can effectively increase the processing speed and avoid server capacity limitations. In single-machine and multi-GPU training scenarios, the framework uses a client batch processing method to connect the client information in series, thereby improving parallel execution capabilities and achieving significant training efficiency improvements.

[0072] The distributed data processing method based on privacy protection provided in this embodiment includes the following steps: obtaining the input data of the client and the output data corresponding to the input data; preprocessing the input data to obtain the preprocessed input data, wherein the preprocessed input data includes the preprocessed input data at time t and the preprocessed input data at time t-1; determining the residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determining the language model based on the preprocessed input data, the output data corresponding to the input data and the residual difference; training the language model based on the loss function to determine the initial model parameters of the language model; updating the initial model parameters based on the preprocessed input data and the residual difference to obtain the updated model parameters; updating the language model based on the updated model parameters, and processing the new preprocessed input data using the updated language model to obtain the output result. The present disclosure can use the distributed language model for training and updating through the privacy data of the client, ensure the security of the data, effectively prevent data leakage and information theft; improve the rate and accuracy of model update, thereby effectively improving the anti-attack ability of the model, protecting the data integrity during the training process, and data preprocessing and post-processing can be performed only on the client, reducing the risk of data leakage.

[0073] Figure 2 FIG. 1 is a schematic diagram of the structure of a distributed data processing device based on privacy protection in an embodiment of the present disclosure. Figure 2 As shown, the device includes a client module 210, a language model training module 220, a server module 230 and a result acquisition module 240.

[0074] The client module 210 is used to obtain input information and output information of the client, preprocess the input data, and obtain preprocessed input data, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1.

[0075] The language model training module 220 is used to determine the residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determine the language model based on the preprocessed input data, the output data corresponding to the input data and the residual difference; train the language model based on the loss function to determine the initial model parameters of the language model;

[0076] The server module 230 is used to update the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters;

[0077] The result acquisition module 240 is used to update the language model based on the updated model parameters, and use the updated language model to process new pre-processed input data to obtain an output result.

[0078] It should be noted that the result acquisition module 240 can exist on the client, or on a third-party platform that can ensure data privacy, or in some cases, can exist in an independent device or equipment in the server that can ensure data privacy. In addition, the type of input data can be found in the description of the method above, which will not be repeated here.

[0079] The expression of the preprocessed input data is: ,

[0080] in, Represents preprocessing input data, Represents input data, Represents input data The mean of Represents input data The variance of represents the scaling factor, represents the logistic distribution density function, Represents the distribution function impact factor.

[0081] Among them, the expression of the residual difference is: ,

[0082] in, represents the residual difference, represents the preprocessed input data at time t, represents the preprocessed input data at time t-1, represents the harmonic factor, represents the first aggregation factor, represents the second aggregation factor, Indicates the number of clients.

[0083] In one embodiment, the initial model parameters include a first model parameter and a second model parameter, and the expression of the language model is: ,

[0084] in, Represents the output data corresponding to the preprocessed input data, Indicates the output data corresponding to the input data, represents the first model parameter, represents the second model parameter, represents the trigonometric factors, represents the phase factor, represents the transposed convolution operation, Represents the model weight factor.

[0085] Optionally, the language model training module 220 is further used to optimize the loss function so that the initial model parameters are obtained when the value of the loss function is minimized, wherein the expression of the loss function is: ,

[0086] in, represents the value of the loss function, Represents preprocessing input data, Indicates the output data corresponding to the input data, Represents the output data corresponding to the preprocessed input data, Represents preprocessing input data The number of data points in .

[0087] Optionally, the server module 230 is further configured to determine and update the first model parameter using a first formula based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the first model parameter;

[0088] Based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the second model parameter, the second model parameter is updated using the second formula, wherein the expression of the first formula is: ,

[0089] The expression of the second formula is: ,

[0090] in, represents updating the first model parameter, represents updating the second model parameters, Represents the first model parameter The difference influencing factors, represents the second model parameter The difference influencing factors, represents the chi-square density distribution function.

[0091] Optionally, the server module 230 is further configured to retain the updated model parameters on the server and encrypt the updated model parameters to ensure the security of user privacy data.

[0092] The distributed data processing device based on privacy protection provided in the embodiment of the present disclosure can execute the steps of the distributed data processing method based on privacy protection provided in the method embodiment of the present disclosure, and the execution steps and beneficial effects are no longer repeated here.

[0093] Figure 3 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 3 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing the embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0094] like Figure 3 As shown, the electronic device 500 may include a processing device 501, a ROM 502, a RAM 503, a bus 504, an input / output (I / O) interface 505, an input device 506, an output device 507, a storage device 508, and a communication device 509. The processing device (e.g., a central processing unit, a graphics processor, etc.) 501 can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage device 508 to the random access memory (RAM) 503 to implement the distributed multimedia data processing method based on privacy protection as described in the embodiment of the present disclosure. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the distributed data processing method based on privacy protection as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0096] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0097] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the input data of the client and the output data corresponding to the input data; preprocesses the input data to obtain the preprocessed input data, wherein the preprocessed input data includes the preprocessed input data at time t and the preprocessed input data at time t-1; determines the residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determines the language model based on the preprocessed input data, the output data corresponding to the input data and the residual difference; trains the language model based on the loss function to determine the initial model parameters of the language model; updates the initial model parameters based on the preprocessed input data and the residual difference to obtain the updated model parameters; updates the language model based on the updated model parameters, and processes the new preprocessed input data using the updated language model to obtain the output result.

[0098] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A distributed data processing method based on privacy protection, characterized in that: The method comprises: Acquire input data of a client and output data corresponding to the input data, wherein the input data is multimedia data including one or more of pictures, texts, voices and videos; Preprocessing the input data to obtain preprocessed input data, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1; Determining a residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determining a language model based on the preprocessed input data, output data corresponding to the input data, and the residual difference; Training the language model based on the loss function to determine initial model parameters of the language model; Updating the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters; The language model is updated based on the updated model parameters, and the new preprocessed input data is processed using the updated language model to obtain an output result.

2. The method according to claim 1, characterized in that: The expression of the preprocessing input data is: ， in, Represents preprocessing input data, Represents input data, Represents input data The mean of Represents input data The variance of represents the scaling factor, represents the logistic distribution density function, Represents the distribution function impact factor.

3. The method according to claim 2, characterized in that The expression of the residual difference is: ， in, represents the residual difference, represents the preprocessed input data at time t, represents the preprocessed input data at time t-1, represents the harmonic factor, represents the first aggregation factor, represents the second aggregation factor, Indicates the number of clients.

4. The method according to claim 3, characterized in that The initial model parameters include a first model parameter and a second model parameter, and the expression of the language model is: ， in, Represents the output data corresponding to the preprocessed input data, Indicates the output data corresponding to the input data, represents the first model parameter, represents the second model parameter, represents the trigonometric factors, represents the phase factor, represents the transposed convolution operation, Represents the model weight factor.

5. The method according to claim 1 or 4, characterized in that: Training the language model based on the loss function and determining initial model parameters of the language model include: The loss function is optimized so that the initial model parameters are obtained when the value of the loss function is minimized.

6. The method according to claim 5, characterized in that The expression of the loss function is: ， in, represents the value of the loss function, Represents preprocessing input data, Indicates the output data corresponding to the input data, Represents the output data corresponding to the preprocessed input data, Represents preprocessing input data The number of data points in .

7. The method according to claim 6, characterized in that The updating of the model parameters includes updating the first model parameters and updating the second model parameters. The initial model parameters are updated based on the preprocessed input data and the residual difference to obtain updated model parameters, including: Based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the first model parameter, a first formula is used to determine and update the first model parameter; Based on the preprocessed input data at time t, the preprocessed input data at time t-1, the residual difference and the second model parameter, a second formula is used to determine and update the second model parameter.

8. The method according to claim 7, characterized in that The expression of the first formula is: ， The expression of the second formula is: ， in, represents updating the first model parameter, represents updating the second model parameters, Represents the first model parameter The difference influencing factors, represents the second model parameter The difference influencing factors, represents the chi-square density distribution function.

9. The method according to claim 1, characterized in that: Also includes: The updated model parameters are retained on the server and encrypted to ensure the security of user privacy data.

10. A distributed data processing device based on privacy protection, characterized in that: The device comprises: A client module, used to obtain input data of a client and output data corresponding to the input data, preprocess the input data to obtain preprocessed input data, wherein the input data is multimedia data including one or more of pictures, texts, voices and videos, wherein the preprocessed input data includes preprocessed input data at time t and preprocessed input data at time t-1; A language model training module, used to determine the residual difference based on the preprocessed input data at time t and the preprocessed input data at time t-1; determine the language model based on the preprocessed input data, the output data and the residual difference; train the language model based on a loss function to determine the initial model parameters of the language model; The server module is used to update the initial model parameters based on the preprocessed input data and the residual difference to obtain updated model parameters. The result acquisition module is used to update the language model based on the updated model parameters, and use the updated language model to process new pre-processed input data to obtain an output result.

Images