Data processing method, system, electronic device and non-volatile storage medium

Abstract

The present application discloses a data processing method, system, electronic device and non-volatile storage medium. The method includes: obtaining a first feature matrix sent by an edge device, wherein the first feature matrix is ​​an intermediate output feature matrix with differential privacy noise; performing denoising on the first feature matrix to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by a denoising model, and the denoising is used to remove differential privacy noise; using the second sub-model in the collaborative reasoning model, reasoning is performed based on the second feature matrix to obtain an inference result, and returning the inference result to the edge device. The present application solves the technical problem that the user privacy information is at risk of leakage because the feature data communicated between the edge device and the server in the related technology may be reconstructed as a model input sample after being received by the server.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular, to a data processing method, system, electronic device and non-volatile storage medium. Background Art

[0002] AIoT (Artificial Intelligence & Internet of Things) scenarios involve a large number of sensor devices and edge nodes. The data collected by these devices needs to be inferred and processed in real time or near real time. For edge devices, limited computing power and bandwidth are difficult to meet the needs of processing huge amounts of data. Therefore, collaborative reasoning is needed to reduce data transmission and computing power requirements. Collaborative reasoning refers to splitting the neural network model according to a certain strategy and then distributing it to the IoT edge devices and cloud servers for joint reasoning and calculation.

[0003] However, in related technologies, the characteristic data communicated between edge devices and servers may be reconstructed into model input samples after being received by the server, thereby causing the risk of leakage of AIoT user privacy information.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, system, electronic device and non-volatile storage medium to at least solve the technical problem that the characteristic data communicated between the edge device and the server in the related technology may be reconstructed into model input samples after being received by the server, resulting in the risk of leakage of user privacy information.

[0006] According to one aspect of an embodiment of the present application, a data processing method is provided, including: obtaining a first feature matrix sent by an edge-side device, wherein a first sub-model in a collaborative reasoning model is provided in the edge-side device, and the first feature matrix is ​​an intermediate output feature matrix with differential privacy noise, and the intermediate output feature matrix is ​​obtained by the edge-side device after reasoning on the inference sample data using the first sub-model; denoising the first feature matrix to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; using the second sub-model in the collaborative reasoning model, reasoning according to the second feature matrix to obtain an inference result, and returning the inference result to the edge-side device.

[0007] Optionally, the collaborative reasoning model includes: a first sub-model and a second sub-model, wherein the first sub-model is a model including the model structure located before the target division position in the collaborative reasoning model, and the second sub-model is a model including the model structure located after the target division position in the collaborative reasoning model. The target division position is determined based on the model structure of the collaborative reasoning model to be trained and the characteristics of the corresponding training data set. When the collaborative reasoning model to be trained is a neural network model, the target division position is the position between the two hidden layers of the neural network model.

[0008] Optionally, the first sub-model is set in the edge device, and the second sub-model and the denoising model are set in the server; wherein, the training steps of the collaborative reasoning model and the denoising model include: obtaining a training data set, wherein the training data set includes: multiple training samples, and inference results corresponding to the training samples; using the first sub-model in the collaborative reasoning model to be trained to determine the intermediate output feature matrix corresponding to the training samples, and adding differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix; using the denoising model to be trained to denoise the first feature matrix to obtain a second feature matrix, and using the second sub-model in the collaborative reasoning model to be trained to determine the inference results corresponding to the second feature matrix; determining a loss function based on the first error and the second error, and updating the model parameters of the collaborative reasoning model and the denoising model based on the loss function, repeating the above training steps until the value of the loss function is less than a preset error threshold, and obtaining the trained collaborative reasoning model and denoising model, wherein the first error is used to characterize the error between the inference result determined by the collaborative reasoning model and the inference result corresponding to the training sample in the training data set, and the second error is used to characterize the error between the intermediate output feature matrix and the second feature matrix.

[0009] Optionally, adding differential privacy noise to the intermediate output feature matrix to obtain the first feature matrix includes: determining an upper limit value of privacy loss and a tolerance for privacy protection, wherein smaller the upper limit value and the tolerance are, the stronger the privacy protection of the data is; determining a target noise variance based on the upper limit value and the tolerance, and generating a random noise sample that obeys a Gaussian distribution with a mean of 0 and a variance equal to the target noise variance; and adding the random noise sample to the intermediate output feature matrix to obtain the first feature matrix.

[0010] Optionally, determining the loss function based on the first error and the second error includes: determining the cross entropy between the inference result determined by the collaborative inference model and the inference result corresponding to the training sample in the training data set, and determining the cross entropy as the first error; calculating the mean square error between each element in the intermediate output feature matrix and each element in the corresponding second feature matrix, and determining the mean square error as the second error; determining the weight coefficients corresponding to the first error and the second error respectively, and determining the loss function based on the first error, the second error, and the weight coefficient.

[0011] Optionally, updating the model parameters of the collaborative reasoning model and the denoising model based on the loss function includes: calculating the gradient of the loss function with respect to the model parameters by performing gradient backpropagation on the loss function, wherein the model parameters include: a first model parameter corresponding to the collaborative reasoning model and a second model parameter corresponding to the denoising model; and updating the model parameters based on the gradient and the learning rate corresponding to the model parameters.

[0012] Optionally, the denoising model includes: an upsampling neural network and a downsampling neural network, wherein the upsampling neural network and the downsampling neural network are connected through a fully connected layer, each of the upsampling neural network and the downsampling neural network contains multiple network groups, each network group contains multiple convolutional layers, the network groups in the downsampling neural network are connected through a pooling layer, and the network groups in the upsampling neural network are connected through a deconvolution layer. The convolution layer in the downsampling neural network is used to increase the number of channels of the intermediate output feature matrix, and the convolution layer in the upsampling neural network is used to reduce the number of channels of the intermediate output feature matrix.

[0013] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining sample data to be inferred, and using the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to be inferred to obtain an intermediate output feature matrix; adding differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and sending the first feature matrix to a server, wherein the server is used to perform denoising processing on the first feature matrix to obtain a second feature matrix, and the second feature matrix is ​​the intermediate output feature matrix obtained after denoising processing by the denoising model, and the denoising processing is used to remove differential privacy noise; receiving the inference result corresponding to the first feature matrix returned by the server, wherein the inference result is obtained by the server using the second sub-model in the collaborative reasoning model to perform reasoning based on the second feature matrix.

[0014] According to another aspect of an embodiment of the present application, a data processing system is also provided, including: an edge-side device and a server, wherein the edge-side device is used to obtain sample data to be inferred, and adopt the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to be inferred to obtain an intermediate output feature matrix; and, add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server; the server is used to receive the first feature matrix, and perform denoising on the first feature matrix to obtain a second feature matrix; and, adopt the second sub-model in the collaborative reasoning model to perform reasoning based on the second feature matrix to obtain an inference result, and return the inference result to the edge-side device, wherein the second feature matrix is ​​the intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise.

[0015] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the data processing method is executed when the program is run.

[0016] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored computer program, wherein a device where the non-volatile storage medium is located executes a data processing method by running the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the data processing method when the computer program is executed by a processor.

[0018] In an embodiment of the present application, a first feature matrix sent by an edge device is obtained, wherein a first sub-model in a collaborative reasoning model is provided in the edge device, and the first feature matrix is ​​an intermediate output feature matrix with differential privacy noise, and the intermediate output feature matrix is ​​obtained by the edge device after reasoning with the first sub-model on the inference sample data; the first feature matrix is ​​denoised to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; the second sub-model in the collaborative reasoning model is used to reason based on the second feature matrix to obtain an inference result, and the inference result is returned to the edge device. By adding differential privacy noise and denoising in the collaborative reasoning process, the purpose of protecting the privacy information of the edge device in the collaborative reasoning process is achieved, thereby solving the technical problem of the risk of leakage of user privacy information caused by the fact that the feature data communicated between the edge device and the server in the related technology may be reconstructed as a model input sample after being received by the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a data processing method provided in an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of a data processing method flow provided according to an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a collaborative reasoning method based on denoising differential privacy provided in an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a network structure of a denoising model provided according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a synchronous training provided according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of the timing of an inference process provided according to an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of another data processing method flow provided according to an embodiment of the present application;

[0027] Figure 8 It is a structural diagram of a data processing system provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] To facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0031] AIoT: Integrating artificial intelligence technology and Internet of Things technology, it generates and collects massive amounts of data from different dimensions through the Internet of Things and stores them in the cloud and edge. It then uses big data analysis to form a higher form of artificial intelligence, realizing the digitization and intelligent connection of all things.

[0032] Differential privacy: A method based on rigorous mathematical proof that uses a randomized response method to ensure that the impact of a single record on a dataset when outputting information is always below a certain threshold. This prevents third parties from determining changes, additions, or deletions (differential attacks) based on output changes. This method is considered to be the most secure of the current perturbation-based privacy protection methods.

[0033] AIoT scenarios involve a large number of sensor devices and edge nodes, and the data collected by these devices requires real-time or near-real-time reasoning and processing. However, AIoT environments are subject to numerous limitations, such as limited computing resources and communication bandwidth at edge nodes. Furthermore, the environmental data collected by AIoT sensors can be enormous. Limited computing power and bandwidth at edge devices make it difficult to process this data locally and transmit it to servers over the network. Therefore, collaborative reasoning is necessary to reduce data transmission and computing power requirements.

[0034] Collaborative reasoning refers to splitting a neural network model according to a certain strategy and then distributing it to IoT edge devices and cloud servers for joint reasoning and calculation. By choosing a reasonable split location, the communication overhead can be reduced while fully utilizing the computing power of edge-cloud devices, thereby reducing the reasoning overhead. Improvements to collaborative reasoning in related technologies mainly focus on designing differential schemes that are more efficient in data transmission and computing performance overhead. However, some studies have shown that the feature data communicated between edge devices and servers may be reconstructed into model input samples after being received by the server, posing a risk of leaking AIoT user privacy information. There is also a lack of effective defense measures against these risks.

[0035] In view of this, how to ensure that the user's privacy data is fully protected during the collaborative reasoning process without significantly affecting the model's reasoning ability has become an important issue that technical personnel in this field need to solve.

[0036] In order to solve the problem that the collaborative reasoning technology in the related art is insufficient in protecting the privacy of edge devices, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0037] According to an embodiment of the present application, an embodiment of a method for data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a data processing method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0043] Under the above-mentioned operating environment, an embodiment of the present application provides a data processing method. Specifically, the application scenario in the embodiment of the present application includes a group of AIoT edge devices (such as AI cameras, etc.) and corresponding server devices, and the two can communicate with each other in a specific manner.

[0044] Edge devices typically collect relatively large amounts of data, but have limited computing resources and are subject to stringent network bandwidth and power consumption constraints. Some intelligent applications cannot independently perform inference, nor can they transmit all collected data to cloud servers. Furthermore, the data collected by these devices also requires privacy protection. Server devices, deployed in the cloud, possess more powerful computing power, but cannot be fully trusted by users. Directly uploading data or data features collected by edge devices to the cloud can pose privacy risks.

[0045] In response to the problem of insufficient performance of AIoT mid-end side devices and the need to protect user data privacy, the embodiments of this application provide relevant solutions. Figure 2 This is a schematic diagram of a method flow for data processing on the server side that can be applied to the above scenario according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0046] Step S202: Obtain a first feature matrix sent by the edge-side device. The edge-side device is configured with a first sub-model of the collaborative inference model. The first feature matrix is ​​an intermediate output feature matrix with differential privacy noise. The intermediate output feature matrix is ​​obtained by the edge-side device after inferring the sample data to be inferred using the first sub-model.

[0047] Step S204: De-noising the first feature matrix to obtain a second feature matrix, where the second feature matrix is ​​an intermediate output feature matrix obtained after the de-noising model performs de-noising processing, and the de-noising processing is used to remove differential privacy noise;

[0048] Step S206 : Use the second sub-model in the collaborative reasoning model to perform reasoning based on the second feature matrix to obtain a reasoning result, and return the reasoning result to the edge-side device.

[0049] Through the above steps, by adding differential privacy noise and denoising during the collaborative reasoning process, the purpose of protecting the privacy information of the edge devices during the collaborative reasoning process is achieved, thereby solving the technical problem of the risk of leakage of user privacy information caused by the fact that the characteristic data of the communication between the edge devices and the server in the related technology may be reconstructed as model input samples after being received by the server.

[0050] The data processing method in steps S202 to S206 of the embodiment of the present application is further introduced below.

[0051] Figure 3 This is a schematic diagram of a collaborative reasoning method based on denoising differential privacy provided in an embodiment of the present application, such as Figure 3As shown, in terms of the training of the collaborative reasoning model and the denoising model, we can first select a reasonable location (target division location) to divide the neural network model based on the specific business type, the structure of the model to be trained, and the characteristics of the training data set to obtain a construction plan for the collaborative reasoning model; and select and build a denoising model; by adding differential privacy noise during the training process, the collaborative reasoning model and the denoising model are trained synchronously, and finally a converged and effective network model is obtained; after that, the front and back halves of the collaborative reasoning model (i.e., the first sub-model and the second sub-model) can be deployed on the edge device and server respectively, and the denoising model can be deployed on the server.

[0052] On the other hand, after the model is deployed, the collaborative reasoning model and denoising model can be used for reasoning. Specifically, the front-layer network of the model (i.e., the first sub-model) is first run on the edge device, and the obtained intermediate output feature matrix is ​​added with differential privacy noise in the same way as training and transmitted to the server; the server receives the noisy intermediate output feature matrix (i.e., the first feature matrix) and denoises it using the denoising model; the server uses the denoised intermediate output feature matrix (i.e., the second feature matrix) as input and performs reasoning in the second half of the collaborative reasoning model (i.e., the second sub-model) to obtain the reasoning result; the server returns the reasoning result to the edge device.

[0053] The following describes in detail the model training and deployment process, as well as the process of using the model for collaborative reasoning. First, the model training and deployment process is introduced.

[0054] In some embodiments of the present application, the collaborative reasoning model includes: a first sub-model and a second sub-model, wherein the first sub-model is a model including the model structure located before the target division position in the collaborative reasoning model, and the second sub-model is a model including the model structure located after the target division position in the collaborative reasoning model. The target division position is determined based on the model structure of the collaborative reasoning model to be trained and the characteristics of the corresponding training data set. When the collaborative reasoning model to be trained is a neural network model, the target division position is the position between the two hidden layers of the neural network model.

[0055] Specifically, based on the specific business type, the structure of the model to be trained, and the characteristics of the training data set, a reasonable location (i.e., the target division location) can be selected to divide (collaborative reasoning model) and obtain a construction plan for the collaborative reasoning model.

[0056] The above-mentioned reasonable position can be selected based on the specific situation. In an embodiment of the present application, the above-mentioned collaborative reasoning model to be trained can be a neural network model, which contains more hidden layers and is used to complete specific business functions through reasoning. In the case that the collaborative reasoning model to be trained is a neural network model, the target division position can be the position between two hidden layers in the model to be trained. The position before this position is the front-layer network model (i.e., the first sub-model), and the position after this position is the second-half model (i.e., the second sub-model). They are deployed on the edge-side device and the server device respectively according to the collaborative reasoning scheme. The model deployed according to the scheme is the collaborative reasoning model.

[0057] At the same time, the embodiment of the present application also needs to construct a denoising model, as follows.

[0058] In some embodiments of the present application, the denoising model includes: an upsampling neural network and a downsampling neural network, wherein the upsampling neural network and the downsampling neural network are connected through a fully connected layer, and each of the upsampling neural network and the downsampling neural network contains multiple network groups, each network group contains multiple convolutional layers, and the network groups in the downsampling neural network are connected through a pooling layer, and the network groups in the upsampling neural network are connected through a deconvolution layer. The convolution layer in the downsampling neural network is used to increase the number of channels of the intermediate output feature matrix, and the convolution layer in the upsampling neural network is used to reduce the number of channels of the intermediate output feature matrix.

[0059] Specifically, in this embodiment, the denoising model is illustrated using the U-Net neural network model as an example. Its input is an intermediate output feature matrix with differential privacy noise, i.e., the above-mentioned first feature matrix, and its output is a feature matrix with the same dimension as the input feature matrix but with the noise removed, i.e., the above-mentioned second feature matrix; wherein, the intermediate output feature matrix is ​​the feature matrix output by the front-layer collaborative reasoning model (first sub-model), which contains information related to the business function of the collaborative reasoning model; differential privacy noise refers to the random noise added to the intermediate output feature matrix during the collaborative reasoning process in order to protect the privacy of user data.

[0060] The above intermediate output feature matrix can be a three-order matrix with three dimensions: length, width, and number of channels. The product of length and width is called the resolution of the intermediate output feature. The intermediate output feature matrix can also be referred to as the intermediate feature. In this embodiment, the U-Net denoising model is a symmetrical U-shaped structure, such as Figure 4 As shown in Figure 1, it consists of a downsampling neural network and an upsampling neural network, and these two networks are connected through a fully connected layer.

[0061] Both the upsampling and downsampling neural networks are divided into n network groups. The resolution of the intermediate features input to each network group is the same. The network groups of the upsampling and downsampling neural networks consist of several convolutional layers. The convolutional layers in the downsampling neural network group increase the number of channels of the intermediate features. The groups are connected by pooling layers, which reduces the resolution of the intermediate features. The convolutional layers in the upsampling neural network group reduce the number of channels of the intermediate features. The groups are connected by deconvolution layers, which increases the resolution of the intermediate features.

[0062] In addition, there are direct connections between corresponding groups of up / down sampling neural networks, which can bypass the intermediate layers and improve the effect of the denoising model.

[0063] It should be noted that the denoising model can also be replaced by any machine learning model that can achieve similar functions, such as a reversible neural network, without any restriction.

[0064] After building the collaborative inference model and denoising model to be trained, you can train the collaborative inference model and denoising model synchronously. The specific steps are as follows.

[0065] In some embodiments of the present application, the first sub-model is set in the edge device, and the second sub-model and the denoising model are set in the server; wherein, the training steps of the collaborative reasoning model and the denoising model include the following steps: obtaining a training data set, wherein the training data set includes: multiple training samples and inference results corresponding to the training samples; using the first sub-model in the collaborative reasoning model to be trained to determine the intermediate output feature matrix corresponding to the training samples, and adding differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix; using the denoising model to be trained to denoise the first feature matrix to obtain a second feature matrix, and using the second sub-model in the collaborative reasoning model to be trained to determine the inference results corresponding to the second feature matrix; determining a loss function based on the first error and the second error, and updating the model parameters of the collaborative reasoning model and the denoising model based on the loss function, repeating the above training steps until the value of the loss function is less than a preset error threshold, thereby obtaining the trained collaborative reasoning model and denoising model, wherein the first error is used to characterize the error between the inference result determined by the collaborative reasoning model and the inference result corresponding to the training sample in the training data set, and the second error is used to characterize the error between the intermediate output feature matrix and the second feature matrix.

[0066] like Figure 5 As shown in the figure, first obtain the training data set, randomly extract n training samples x from the training data set D, and obtain their corresponding annotations y (i.e., the inference results corresponding to the training samples); then, take the training samples x as input, and pass the front-layer collaborative inference model (the first sub-model) f edgePerform inference and obtain the intermediate output feature matrix v=f edge (x);

[0067] Afterwards, differential privacy noise is added to the intermediate output feature matrix. The specific steps are as follows.

[0068] In some embodiments of the present application, adding differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix includes the following steps: determining an upper limit value of privacy loss and a tolerance for privacy protection, wherein the smaller the upper limit value and the tolerance, the stronger the privacy protection of the data; determining a target noise variance based on the upper limit value and the tolerance, and generating a random noise sample that obeys a Gaussian distribution with a mean of 0 and a variance equal to the target noise variance; and adding the random noise sample to the intermediate output feature matrix to obtain the first feature matrix.

[0069] Specifically, Gaussian differential privacy noise is added to the output intermediate features, i.e. Where v′ is the intermediate output feature matrix with (Gaussian) differential privacy noise, that is, the first feature matrix mentioned above, which satisfies (∈, δ)-differential privacy. The smaller ∈ and δ are, the stronger the privacy protection effect of this method is. σ 2 That is, the target noise variance mentioned above, s is the data range or the maximum value of the data value, δ is the tolerance of privacy protection, the smaller δ is, the stronger the privacy protection is, ∈ is the upper limit of privacy loss, the smaller ∈ is, the stronger the privacy protection is, It means that the mean is 0 and the variance is σ 2 I is a k-dimensional Gaussian distributed random noise sample, where k is the dimension of the output intermediate features and I is a unit vector of dimension k.

[0070] It should be noted that the differential privacy mechanism in this step can be replaced by any effective differential privacy mechanism, such as the Laplace mechanism, etc., and this embodiment of the present application does not limit this.

[0071] After obtaining the first feature matrix v′, the denoising model g is used to denoise the intermediate output feature matrix (first feature matrix) v′ with differential privacy noise, that is, in is the intermediate output feature of denoising, that is, the second feature matrix mentioned above, with the same dimension as v; then, the intermediate output feature of denoising, that is, the second feature matrix As input, through the second half of the collaborative reasoning model (second sub-model) f cloud Perform inference and obtain inference results

[0072] After that, the inference results can be calculated The loss between the sample label y (i.e., the first error), and the denoised intermediate output features The loss between the output intermediate feature v (i.e., the second error), where l y and and y are positively correlated, l v and The difference between and v is also positively correlated. The specific steps are as follows.

[0073] In some embodiments of the present application, determining the loss function based on the first error and the second error includes the following steps: determining the cross entropy between the inference result determined by the collaborative inference model and the inference result corresponding to the training sample in the training data set, and determining the cross entropy as the first error; calculating the mean square error between each element in the intermediate output feature matrix and each element in the corresponding second feature matrix, and determining the mean square error as the second error; determining the weight coefficients corresponding to the first error and the second error respectively, and determining the loss function based on the first error, the second error, and the weight coefficient.

[0074] Specifically, if the collaborative inference model is a classifier model, the loss is calculated by the mean square error Among them, |v| is the dimension of the intermediate feature, v i and is the denoising intermediate output feature (second feature matrix) and the i-th observation value (element) of the output intermediate feature matrix v; and the loss is calculated by the cross entropy loss function Among them, M is the number of categories of the collaborative reasoning model, y is the one-hot encoding of the sample annotation, that is, the reasoning result corresponding to the training sample in the training dataset. If the category of the sample is labeled as c, then y c =1, otherwise y c =0, is the confidence of the inference result on category c.

[0075] The loss function can be constructed based on the first error and the second error, and the gradient descent algorithm is used to minimize the loss function to perform model training. The specific steps are as follows.

[0076] In some embodiments of the present application, updating the model parameters of the collaborative reasoning model and the denoising model based on the loss function includes the following steps: calculating the gradient of the loss function with respect to the model parameters by performing gradient backpropagation on the loss function, wherein the model parameters include: a first model parameter corresponding to the collaborative reasoning model and a second model parameter corresponding to the denoising model; and updating the model parameters based on the gradient and the learning rate corresponding to the model parameters.

[0077] Specifically, the stochastic gradient descent algorithm is used to minimize the loss function As the goal, update the collaborative reasoning model once and the parameters θ of the denoising model f and θ g , where α is a hyperparameter (weight coefficient). The larger α is, the greater the weight of the denoising model in training.

[0078] Specifically, we can first perform gradient backpropagation on the loss function J through the collaborative inference model f and the denoising model g to calculate the gradient Then use the gradient to update the model parameters Among them, θ f,i ,θ g,i is the parameter of the model in the i-th round, and η is the learning rate. In addition, the improved stochastic gradient descent algorithm such as AdaGrad and Adam can also be used to update the model.

[0079] The above steps of synchronously training the collaborative reasoning model and the denoising model are continuously repeated until the collaborative reasoning model f and the denoising model g converge, and the trained collaborative reasoning model and denoising model can be obtained. For the trained collaborative reasoning model and denoising model, in this embodiment, the front-layer collaborative reasoning model (that is, the first sub-model in the same reasoning model) can be deployed on the end-side device; the second half of the collaborative reasoning model (that is, the second sub-model in the same reasoning model) and the denoising model can be deployed on the server device, waiting for the user's reasoning needs.

[0080] The embodiment of the present application synchronously trains the collaborative reasoning model and the denoising model, which not only ensures that the trained collaborative reasoning model and the denoising model have high effectiveness, but also makes the two models more adaptable in the reasoning process, thereby improving the denoising effect.

[0081] After completing the training and deployment of the model, the model can be used for denoising differential privacy collaborative reasoning. The specific process is as follows: Figure 6 As shown, the following is a detailed introduction.

[0082] When an inference requirement (sample data to be inferred) occurs, the edge device will perform inference and add differential privacy noise. Specifically, the sample x0 to be inferred is used as input, and the front-layer collaborative inference model (first sub-model) f edge Perform inference and obtain the intermediate output feature matrix v0=f edge (x0), then add Gaussian differential privacy noise to the intermediate output feature matrix v0 to obtain the first feature matrix, i.e. The value of σ should be consistent with the σ set when adding differential privacy noise during model training.

[0083] Then, the intermediate output feature matrix with differential privacy noise, that is, the first feature matrix v ′ 0 is transmitted to the server device in a feature manner. The embodiment of the present application adds Gaussian noise to the intermediate output feature matrix v0 to obfuscate any sensitive information that may be present in the feature matrix, making it difficult for malicious attackers to reconstruct private data from the feature matrix, thereby protecting the privacy of the edge device.

[0084] The server device receives the first feature matrix v ′ 0, the denoising model g is used as the intermediate output feature matrix v with differential privacy noise ′ 0 denoising, and obtain the second characteristic matrix, namely In this embodiment of the present application, the intermediate output feature matrix (first feature matrix) v with differential privacy noise is ′ 0 denoising, which reduces the impact of the differential privacy mechanism on the reasoning effect of the collaborative reasoning model, ensuring the availability of the model while protecting private data.

[0085] Afterwards, the server device outputs the denoised intermediate features (second feature matrix) As input, through the second half of the collaborative reasoning model (second sub-model) f cloud Perform inference and obtain inference results And the reasoning results of the collaborative reasoning model f Transmit the data back to the end-side device.

[0086] This application solution addresses the issue of end-side device data being completely transparent to the server during collaborative reasoning and lacking privacy protection by incorporating a differential privacy mechanism into collaborative reasoning. It also makes the degree of data protection in the collaborative reasoning system quantifiable. Furthermore, by using a server-side denoising model for denoising, it addresses the issue of severely degraded collaborative reasoning effectiveness caused by using only differential privacy mechanisms to protect end-side data privacy, enabling collaborative reasoning to have relatively high usability while minimizing privacy risks.

[0087] According to an embodiment of the present application, another embodiment of a data processing method is also provided. Figure 7 This is a schematic diagram of a method flow for data processing that can be applied to a device on the edge side according to an embodiment of the present application. Figure 7 As shown, the method includes the following steps:

[0088] Step 702: Obtain sample data to be inferred, and use the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to obtain an intermediate output feature matrix;

[0089] Step 704: Add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server. The server is configured to denoise the first feature matrix to obtain a second feature matrix. The second feature matrix is ​​the intermediate output feature matrix obtained after denoising by the denoising model. The denoising is used to remove differential privacy noise.

[0090] Step 706: Receive the inference result corresponding to the first feature matrix returned by the server, wherein the inference result is obtained by the server using the second sub-model in the collaborative inference model to perform inference based on the second feature matrix.

[0091] It should be noted that the data processing method provided in this embodiment is Figure 2 The data processing method shown corresponds to an embodiment, therefore, the relevant explanations and descriptions of the above data processing method are also applicable to the embodiments of this application and will not be repeated here.

[0092] According to an embodiment of the present application, an embodiment of a data processing system is also provided. Figure 8 Schematic diagram of a data processing system according to an embodiment of the present application. Figure 8 As shown, the system includes:

[0093] The edge-side device 80 is configured to obtain sample data to be inferred, and use the first sub-model in the collaborative inference model to perform inference based on the sample data to obtain an intermediate output feature matrix; and add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server.

[0094] Server 82 is configured to receive the first feature matrix and perform denoising on the first feature matrix to obtain a second feature matrix; and, using the second sub-model in the collaborative reasoning model, perform reasoning based on the second feature matrix to obtain a reasoning result, and return the reasoning result to the edge device, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise.

[0095] It should be noted that the various modules in the above-mentioned data processing system can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0096] It should be noted that the data processing system provided in this embodiment can be used to execute Figure 2 or Figure 7The data processing method shown, therefore, the relevant explanations of the above data processing method are also applicable to the embodiments of this application and will not be repeated here.

[0097] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following data processing method by running the computer program: obtaining a first feature matrix sent by the edge-side device, wherein the edge-side device is provided with a first sub-model in the collaborative reasoning model, the first feature matrix is ​​an intermediate output feature matrix with differential privacy noise, and the intermediate output feature matrix is ​​obtained by the edge-side device after reasoning on the inference sample data using the first sub-model; denoising is performed on the first feature matrix to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; using the second sub-model in the collaborative reasoning model, reasoning is performed based on the second feature matrix to obtain an inference result, and the inference result is returned to the edge-side device.

[0098] Alternatively, obtain sample data to be inferred, and use the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to be inferred to obtain an intermediate output feature matrix; add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server, wherein the server is used to perform denoising on the first feature matrix to obtain a second feature matrix, which is the intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; receive the inference result corresponding to the first feature matrix returned by the server, wherein the inference result is obtained by the server using the second sub-model in the collaborative reasoning model to perform reasoning based on the second feature matrix.

[0099] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data processing method described in each embodiment of the present application: obtaining a first feature matrix sent by an edge-side device, wherein a first sub-model in a collaborative reasoning model is provided in the edge-side device, and the first feature matrix is ​​an intermediate output feature matrix with differential privacy noise, and the intermediate output feature matrix is ​​obtained by the edge-side device after reasoning on the inference sample data using the first sub-model; denoising is performed on the first feature matrix to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; using the second sub-model in the collaborative reasoning model, reasoning is performed based on the second feature matrix to obtain an inference result, and the inference result is returned to the edge-side device.

[0100] Alternatively, obtain sample data to be inferred, and use the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to be inferred to obtain an intermediate output feature matrix; add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server, wherein the server is used to perform denoising on the first feature matrix to obtain a second feature matrix, which is the intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove differential privacy noise; receive the inference result corresponding to the first feature matrix returned by the server, wherein the inference result is obtained by the server using the second sub-model in the collaborative reasoning model to perform reasoning based on the second feature matrix.

[0101] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0104] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0107] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining a first feature matrix sent by an edge-side device, wherein the edge-side device is provided with a first sub-model of a collaborative inference model, the first feature matrix being an intermediate output feature matrix with differential privacy noise, and the intermediate output feature matrix being obtained by the edge-side device after inferring sample data to be inferred using the first sub-model; Performing denoising on the first feature matrix to obtain a second feature matrix, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by a denoising model, and the denoising is used to remove the differential privacy noise; Using the second sub-model in the collaborative reasoning model, performing reasoning according to the second feature matrix to obtain a reasoning result, and returning the reasoning result to the edge-side device; Among them, the first sub-model is set in the edge-side device, and the second sub-model and the denoising model are set in the server; the training steps of the collaborative reasoning model and the denoising model include: obtaining a training data set, wherein the training data set includes: multiple training samples, and inference results corresponding to the training samples; using the first sub-model in the collaborative reasoning model to be trained, determining the intermediate output feature matrix corresponding to the training samples, and adding differential privacy noise to the intermediate output feature matrix to obtain the first feature matrix; using the denoising model to be trained, denoising the first feature matrix to obtain the second feature matrix, and using the denoising model to be trained The second sub-model in the collaborative reasoning model is trained to determine the inference result corresponding to the second feature matrix; based on the first error and the second error, a loss function is determined, and based on the loss function, the model parameters of the collaborative reasoning model and the denoising model are updated, and the above training steps are repeated until the value of the loss function is less than the preset error threshold, thereby obtaining the trained collaborative reasoning model and the denoising model, wherein the first error is used to characterize the error between the inference result determined by the collaborative reasoning model and the inference result corresponding to the training sample in the training data set, and the second error is used to characterize the error between the intermediate output feature matrix and the second feature matrix.

2. The data processing method according to claim 1, wherein: The collaborative reasoning model includes: the first sub-model and the second sub-model, wherein the first sub-model is a model including the model structure located before the target division position in the collaborative reasoning model, and the second sub-model is a model including the model structure located after the target division position in the collaborative reasoning model. The target division position is determined based on the model structure of the collaborative reasoning model to be trained and the characteristics of the corresponding training data set. When the collaborative reasoning model to be trained is a neural network model, the target division position is the position between two hidden layers of the neural network model.

3. The data processing method according to claim 2, characterized in that: Adding differential privacy noise to the intermediate output feature matrix to obtain the first feature matrix includes: Determining an upper limit value of privacy loss and a tolerance for privacy protection, wherein the smaller the upper limit value and the tolerance, the stronger the privacy protection of the data; Determine a target noise variance based on the upper limit value and the tolerance, and generate a random noise sample that obeys a Gaussian distribution with a mean of 0 and a variance equal to the target noise variance; The random noise samples are added to the intermediate output feature matrix to obtain the first feature matrix.

4. The data processing method according to claim 2, wherein: Determining the loss function based on the first error and the second error includes: Determining a cross entropy between the inference result determined by the collaborative inference model and the inference result corresponding to the training sample in the training data set, and determining the cross entropy as the first error; Calculating a mean square error between each element in the intermediate output feature matrix and each corresponding element in the second feature matrix, and determining the mean square error as the second error; Determine weight coefficients corresponding to the first error and the second error respectively, and determine the loss function based on the first error, the second error, and the weight coefficients.

5. The data processing method according to claim 2, wherein: Updating model parameters of the collaborative reasoning model and the denoising model according to the loss function includes: Calculating the gradient of the loss function with respect to the model parameters by performing gradient backpropagation on the loss function, wherein the model parameters include: a first model parameter corresponding to the collaborative inference model and a second model parameter corresponding to the denoising model; The model parameters are updated according to the gradient and the learning rate corresponding to the model parameters.

6. The data processing method according to claim 1, wherein: The denoising model includes: an upsampling neural network and a downsampling neural network, wherein the upsampling neural network and the downsampling neural network are connected through a fully connected layer, and each of the upsampling neural network and the downsampling neural network includes multiple network groups, each of the network groups includes multiple convolutional layers, and the network groups in the downsampling neural network are connected through a pooling layer, and the network groups in the upsampling neural network are connected through a deconvolution layer. The convolutional layer in the downsampling neural network is used to increase the number of channels of the intermediate output feature matrix, and the convolutional layer in the upsampling neural network is used to reduce the number of channels of the intermediate output feature matrix.

7. A data processing method, characterized in that: include: Obtaining sample data to be inferred, and using the first sub-model in the collaborative reasoning model to perform reasoning based on the sample data to be inferred, to obtain an intermediate output feature matrix; Adding differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and sending the first feature matrix to a server, wherein the server is configured to perform denoising processing on the first feature matrix to obtain a second feature matrix, where the second feature matrix is ​​the intermediate output feature matrix obtained after denoising processing by a denoising model, and the denoising processing is configured to remove the differential privacy noise; receiving an inference result corresponding to the first feature matrix returned by the server, wherein the inference result is obtained by the server using the second sub-model in the collaborative inference model to perform inference based on the second feature matrix; Among them, the first sub-model is set in the edge-side device, and the second sub-model and the denoising model are set in the server; the training steps of the collaborative reasoning model and the denoising model include: obtaining a training data set, wherein the training data set includes: multiple training samples, and inference results corresponding to the training samples; using the first sub-model in the collaborative reasoning model to be trained, determining the intermediate output feature matrix corresponding to the training sample, and adding differential privacy noise to the intermediate output feature matrix to obtain the first feature matrix; using the denoising model to be trained, denoising the first feature matrix to obtain the second feature matrix, and using the denoising model to be trained The second sub-model in the collaborative reasoning model is used to determine the inference result corresponding to the second feature matrix; based on the first error and the second error, a loss function is determined, and based on the loss function, the model parameters of the collaborative reasoning model and the denoising model are updated, and the above training steps are repeated until the value of the loss function is less than the preset error threshold, thereby obtaining the trained collaborative reasoning model and the denoising model, wherein the first error is used to characterize the error between the inference result determined by the collaborative reasoning model and the inference result corresponding to the training sample in the training data set, and the second error is used to characterize the error between the intermediate output feature matrix and the second feature matrix.

8. A data processing system, characterized in that: include: Edge devices and servers, including: The edge-side device is configured to obtain sample data to be inferred, and use the first sub-model in the collaborative inference model to perform inference based on the sample data to be inferred to obtain an intermediate output feature matrix; and add differential privacy noise to the intermediate output feature matrix to obtain a first feature matrix, and send the first feature matrix to the server; The server is configured to receive the first feature matrix and perform denoising on the first feature matrix to obtain a second feature matrix; and, using a second sub-model in the collaborative reasoning model, perform reasoning based on the second feature matrix to obtain a reasoning result, and return the reasoning result to the edge-side device, wherein the second feature matrix is ​​an intermediate output feature matrix obtained after denoising by the denoising model, and the denoising is used to remove the differential privacy noise; Among them, the first sub-model is set in the edge-side device, and the second sub-model and the denoising model are set in the server; the training steps of the collaborative reasoning model and the denoising model include: obtaining a training data set, wherein the training data set includes: multiple training samples, and inference results corresponding to the training samples; using the first sub-model in the collaborative reasoning model to be trained, determining the intermediate output feature matrix corresponding to the training samples, and adding differential privacy noise to the intermediate output feature matrix to obtain the first feature matrix; using the denoising model to be trained, denoising the first feature matrix to obtain the second feature matrix, and using the denoising model to be trained The second sub-model in the collaborative reasoning model is trained to determine the inference result corresponding to the second feature matrix; based on the first error and the second error, a loss function is determined, and based on the loss function, the model parameters of the collaborative reasoning model and the denoising model are updated, and the above training steps are repeated until the value of the loss function is less than the preset error threshold, thereby obtaining the trained collaborative reasoning model and the denoising model, wherein the first error is used to characterize the error between the inference result determined by the collaborative reasoning model and the inference result corresponding to the training sample in the training data set, and the second error is used to characterize the error between the intermediate output feature matrix and the second feature matrix.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data processing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images