Image processing model training method and system

Abstract

The embodiment of the invention provides an image processing model training method and system, and the image processing model training method is applied to a first training unit, and comprises the steps: receiving an image processing model sent by a second training unit, and carrying out the model training of the image processing model through image training data, obtaining update data of each network layer in the image processing model, the update data being used for performing parameter update on each network layer; according to sensitive image data in the image training data, performing data sensitive detection on each piece of updated data, and determining data sensitive information corresponding to each piece of updated data; performing data desensitization on the update data by using the data sensitive information to obtain desensitized update data of the network layers; sending the desensitization updating data of each network layer to the second training unit for parameter updating; and the problem that sensitive privacy information may be exposed in a distributed machine learning process is avoided.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular, to an image processing model training method; one or more embodiments of this specification also relate to another image processing model training method, an image processing model training system, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of artificial intelligence technology, neural network models have also been widely used in various data processing scenarios. In order to solve problems such as insufficient training data and data silos, distributed machine learning methods can be used for model training, thereby ensuring data security while avoiding problems such as insufficient training data and data silos. For example, federated learning methods can be used.

[0003] Currently, there is still a risk of sensitive data leakage in the process of model training using distributed machine learning methods, which may expose sensitive privacy information. Therefore, how to ensure data security and avoid sensitive data leakage during model training has become an urgent problem that needs to be solved. Summary of the invention

[0004] In view of this, an embodiment of this specification provides an image processing model training method. One or more embodiments of this specification also relate to another image processing model training method, an image processing model training system, an image processing model training device, another image processing model training device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing model training method, which is applied to a first training unit and includes:

[0006] Receiving the image processing model sent by the second training unit, and performing model training on the image processing model using the image training data to obtain update data of each network layer in the image processing model, wherein the update data is used to update parameters of each network layer;

[0007] According to the sensitive image data in the image training data, data sensitivity detection is performed on each update data to determine the data sensitivity information corresponding to each update data;

[0008] Desensitizing the updated data using the data sensitive information to obtain desensitized updated data of each network layer;

[0009] The desensitized update data of each network layer is sent to the second training unit for parameter update.

[0010] According to a second aspect of an embodiment of this specification, there is provided an image processing model training device, applied to a first training unit, comprising:

[0011] A model training module is configured to receive the image processing model sent by the second training unit, and perform model training on the image processing model using the image training data to obtain update data of each network layer in the image processing model, wherein the update data is used to update parameters of each network layer;

[0012] A data sensitivity detection module is configured to perform data sensitivity detection on each update data according to the sensitive image data in the image training data, and determine the data sensitivity information corresponding to each update data;

[0013] A data desensitization module is configured to use the data sensitive information to perform data desensitization on each update data to obtain desensitized update data of each network layer;

[0014] The data sending module is configured to send the desensitized update data of each network layer to the second training unit for parameter updating.

[0015] According to a third aspect of an embodiment of this specification, another image processing model training method is provided, which is applied to a second training unit and includes:

[0016] sending the image processing model to a plurality of first training units;

[0017] receiving desensitized update data of each network layer in the image processing model sent by each first training unit, wherein the desensitized update data is obtained by the first training unit performing data desensitization on the update data of each network layer, and the update data is obtained by the first training unit performing model training on the image processing model using the image training data;

[0018] Using the sensitive image data in the second training unit, perform data sensitivity detection on each desensitized update data to determine data sensitivity information corresponding to each desensitized update data;

[0019] Based on the data sensitive information, determining target desensitized update data from the desensitized update data;

[0020] The target desensitization update data is used to update the parameters of each network layer in the image processing model to obtain an updated image processing model.

[0021] According to a fourth aspect of an embodiment of this specification, another image processing model training device is provided, which is applied to a second training unit and includes:

[0022] A model sending module, configured to send the image processing model to a plurality of first training units;

[0023] a data receiving module configured to receive desensitized update data of each network layer in the image processing model sent by each first training unit, wherein the desensitized update data is obtained by the first training unit performing data desensitization on the update data of each network layer, and the update data is obtained by the first training unit performing model training on the image processing model using the image training data;

[0024] A data sensitivity detection module is configured to use the sensitive image data in the second training unit to perform data sensitivity detection on each desensitized update data to determine the data sensitivity information corresponding to each desensitized update data;

[0025] A data determination module is configured to determine target desensitized update data from the desensitized update data based on the data sensitivity information;

[0026] The parameter updating module is configured to use the target desensitization update data to update the parameters of each network layer in the image processing model to obtain an updated image processing model.

[0027] According to a fifth aspect of the embodiments of this specification, there is provided an image processing model training system, including a server and multiple clients, wherein:

[0028] The server is used to send the image processing model to the multiple clients, and receive desensitized update data of each network layer in the image processing model sent by each client, wherein the desensitized update data is obtained by the client performing data desensitization on the update data of each network layer, and the update data is obtained by the client performing model training on the image processing model using image training data, and the sensitive image data in the server is used to perform data sensitivity detection on each desensitized update data to determine the data sensitive information corresponding to each desensitized update data, and based on the data sensitive information, target desensitized update data is determined from each desensitized update data, and the target desensitized update data is used to update the parameters of each network layer in the image processing model to obtain an updated image processing model;

[0029] The client is used to receive the image processing model sent by the server, and use the image training data to perform model training on the image processing model to obtain the update data of each network layer in the image processing model, wherein the update data is used to update the parameters of each network layer, perform data sensitivity detection on each update data according to the sensitive image data in the image training data, determine the data sensitive information corresponding to each update data, use the data sensitive information to desensitize the each update data, obtain the desensitized update data of each network layer, and send the desensitized update data of each network layer to the server for parameter update.

[0030] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including:

[0031] Memory and processor;

[0032] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of any one of the above methods are implemented.

[0033] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of any one of the above methods when executed by a processor.

[0034] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of any one of the above methods when executed by a processor.

[0035] In one or more embodiments of the present specification, the image processing model training method applied to the first training unit can receive the image processing model sent by the second training unit during the distributed machine learning process, and use local image training data to train the image processing model to obtain update data of each network layer in the image processing model; in the process of sending the updated data to the second training unit for parameter updating, in order to avoid the risk that the updated data may leak sensitive data, it is necessary to perform data sensitivity detection on each updated data based on the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each updated data, and use the data sensitive information to desensitize each updated data, and then send the desensitized update data to the second training unit, thereby avoiding the problem of possible exposure of sensitive privacy information in the distributed machine learning process, ensuring data security, and reducing the risk of sensitive data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1It is a schematic diagram of an application of an image processing model training method provided by an embodiment of this specification;

[0037] Figure 2 is a flowchart of an image processing model training method provided by an embodiment of this specification;

[0038] Figure 3 is a processing flow chart of an image processing model training method provided by an embodiment of this specification;

[0039] Figure 4 is a flowchart of another image processing model training method provided by an embodiment of this specification;

[0040] Figure 5 is a processing flow chart of another image processing model training method provided by an embodiment of this specification;

[0041] Figure 6 is a structural schematic diagram of an image processing model training system provided by an embodiment of this specification;

[0042] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0043] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0044] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0045] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0047] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0048] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0049] First, the terms involved in one or more embodiments of this specification are explained.

[0050] Distributed machine learning: refers to a technology that uses multiple computing nodes for machine learning or deep learning. This distributed training technology can greatly improve the speed and efficiency of training large-scale data sets.

[0051] LDP (Local Differential Privacy): Local differential privacy.

[0052] L2 norm: Also called Euclidean norm or 2-norm.

[0053] Federated Learning (FL): A distributed machine learning method that aims to use distributed devices for model training to achieve large-scale cross-domain data learning and reasoning, and protect user data privacy. Centralized machine learning models need to centralize all data to a central server for training, but this may expose sensitive user privacy information. Federated learning avoids sending user data to a central server by training on local devices, making data available but invisible.

[0054] Image Forgery Detection: A technology designed to detect and locate tampered or forged areas in an image. With the development of digital image processing and editing technology, users can use image editing tools and generative models to edit and tamper with image content, such as image synthesis, copy and paste, deletion, repair, etc. The goal of image tampering detection is to identify tampered areas in an image by analyzing the features and content of the image to reveal the authenticity and integrity of the image.

[0055] Model Inversion Attacks: Unless otherwise specified, the inversion attacks mentioned in this article refer to inversion attacks under the federated learning framework. In federated learning, inversion attacks refer to attacks in which malicious actors attempt to obtain original data or model parameters from the federated learning process. This attack aims to reverse engineer the federated learning model to infer private information contained in local data. Specifically, data inversion attacks may occur in federated learning: attackers may attempt to infer the characteristics or sensitive information of local data by analyzing model parameters or gradient information transmitted during the federated learning process. This attack may lead to the exposure of private data.

[0056] Differential Privacy (DP) algorithm: is a classic privacy protection algorithm. The core idea of ​​this algorithm is to introduce a certain amount of noise in the data release process so that attackers cannot infer individual sensitive information by analyzing the output results.

[0057] With the continuous development of artificial intelligence technology, neural network models have also been widely used in various data processing scenarios; for example, in consumer finance scenarios, financial consumers and financial institutions involve a large number of financial certificates with user identity information, such as income certificates, professional certificates, and identity documents. Due to the temptation of unjust enrichment, some users self-drive or with the help of "agent rights protection" to batch produce false certificates and deliver them to multiple financial institutions; how to identify false certificates has become a common difficulty faced by the industry.

[0058] In order to solve the above-mentioned problems such as insufficient training data and data silos, distributed machine learning methods can be used for model training, thereby ensuring data security while avoiding problems such as insufficient training data and data silos; for example, federated learning methods can be used.

[0059] As a representative technology of cross-domain learning, federated learning can make data available but invisible, and is an ideal solution for joint modeling among multiple institutions. However, the potential risks of privacy leakage and model accuracy degradation in federated learning are still key issues that need to be addressed.

[0060] To address the potential privacy leakage risks and model accuracy degradation issues mentioned above, this manual provides four solutions:

[0061] The first scheme can achieve federated learning image classification through a local differential privacy algorithm, especially for medical images with a large amount of personal privacy information; this scheme attempts to integrate local differential privacy (LDP) into the system to achieve strict privacy protection. However, this scheme only protects each sample with the same mechanism, without considering the privacy preference of each client for privacy-sensitive information (PSI). In this case, excessive privacy protection of privacy-insensitive information may introduce unnecessary noise, thereby reducing model performance.

[0062] The second solution is a decentralized federated learning framework for the image field, which is used to solve the communication and computing bottlenecks in the centralized server federated learning framework.

[0063] The third solution is to study the backdoor attacks in federated learning. Through dimensionality reduction technology and clustering algorithm, a federated learning image classification method that can defend against backdoor attacks is proposed.

[0064] The fourth solution is to input the local privacy region image into the local model, and use the output of the model at the specified layer to locate the gradient region corresponding to the privacy region. Obviously, this solution has many manual hyperparameters, such as the number of local privacy region images input to the network, the selection of key layers, and the design of gradient thresholds. In addition, the gradient localization method proposed in the solution has strong expert experience, and requires a lot of manual verification and adjustment optimization for different local models, and has poor generalization.

[0065] Based on this, in this specification, an image processing model training method is provided. One or more embodiments of this specification also involve another image processing model training method, an image processing model training system, an image processing model training device, another image processing model training device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0066] See also Figure 1 , Figure 1 A schematic diagram of an application of an image processing model training method provided according to an embodiment of the present specification is shown. Figure 1 It can be seen that the server 102 can initialize an image processing model (i.e., a global model) and distribute it to all participating clients 104; each client 104 uses local image data (i.e., a local data set) to train the received global model and generate a model update (i.e., gradient data); in order to avoid leakage of sensitive data, the client 104 will define privacy sensitive information (PSI) and calculate the sensitivity of the gradient data to the PSI area; the gradient data is desensitized according to the sensitivity to obtain desensitized gradient data; finally, the client 104 sends the desensitized gradient data to the server 102 for federated learning to protect the privacy data.

[0067] See also Figure 2 , Figure 2 A flowchart of an image processing model training method provided according to an embodiment of the present specification is shown. The method is applied to a first training unit and specifically includes the following steps.

[0068] Step 202: Receive the image processing model sent by the second training unit, and use the image training data to perform model training on the image processing model to obtain update data for each network layer in the image processing model, wherein the update data is used to update parameters of each network layer.

[0069] The first training unit can be understood as a unit in the image processing model training system. The first training unit can use the locally stored image training data to perform model training on the image processing model sent by the second training unit, obtain the update data of each network layer in the image processing model, and feed back the desensitized update data after data desensitization to the second training unit. The first training unit can be a server, a server cluster, a cloud server, a client, etc. In one or more embodiments provided in this specification, the first training unit can be a client; the first training unit can also be understood as an institution participating in federated learning, such as a banking institution, a financial institution, etc.

[0070] Among them, the second training unit can be understood as a unit in the image processing model training system, and the second training unit can be used to send the image processing model to be trained to multiple first training units, and update the parameters of each network layer in the image processing model based on the desensitized update data fed back by multiple first training units. The second training unit can be a server, a server cluster, a server, a cloud server, etc. In one or more embodiments provided in this specification, the second training unit can be a server, and the image processing model training system can be understood as a federated learning system for image processing models.

[0071] Image training data can be understood as image data for model training of image processing models. For example, the image data can be financial image data (such as report images, contract images), face images, architectural images, images of proof materials (such as images of identity proof materials), etc.

[0072] The image processing model can be understood as a model for processing image data. The image processing model can be a neural network model, a deep learning model, or a large model. For example, the image processing model can be an image classification model, an image information recognition model, a text image tampering detection model, etc., and no specific restrictions are made here. The text image tampering detection model is used to perform text image tampering detection tasks.

[0073] The update data can be understood as data for adjusting the parameters of each network layer in the image processing model. For example, the update data can be gradient data, loss data, etc.; adjusting the parameters of each network layer in the image processing model can be understood as adjusting the weight parameters or matrix parameters of each network layer in the image processing model.

[0074] Step 204: Perform data sensitivity detection on each update data according to the sensitive image data in the image training data to determine the data sensitivity information corresponding to each update data.

[0075] Among them, the sensitive image data can be understood as the sensitive data contained in the image training data. For example, the sensitive image data can be sensitive information in the image training data, such as name information, gender information, identity information, etc.; for example, the sensitive image data can be sensitive image areas in the image training data, such as face areas, official seal areas, name areas, etc. in the image training data.

[0076] Data sensitive information can be understood as information used to characterize whether the updated data is sensitive or the degree of sensitivity. For example, the data sensitive information can be a data sensitivity label, data sensitivity, sensitivity, etc. of the updated data.

[0077] Specifically, the image processing model training method provided in this specification can determine sensitive image data from image training data, and then perform data sensitivity detection on each update data according to the sensitive image data to determine the data sensitivity information corresponding to each update data.

[0078] The method of determining the sensitive image data from the image training data may be to determine a sensitive image region and a non-sensitive image region from the image training data, and determine the sensitive image region as the sensitive image data;

[0079] Alternatively, the method of determining sensitive image data from image training data may be to determine multiple initial sensitive image data from the image training data, perform sensitivity evaluation on each initial sensitive image data, and determine a corresponding sensitivity level for each initial sensitive image data according to the sensitivity; for example, the sensitivity levels are: first-level sensitive data, second-level sensitive data, third-level sensitive data, fourth-level sensitive data, etc.

[0080] The initial sensitive image data having a sensitivity level consistent with a preset sensitivity level (eg, level 3 sensitive data and / or level 4 sensitive data) is used as the sensitive image data determined from the image training data.

[0081] Taking the application of the image processing model training method provided in this specification in the personalized differential privacy protection scenario of federated learning as an example, the image processing model training method is explained, wherein the first training unit is the client of federated learning, the second training unit is the server of federated learning, the update data is gradient data, and the sensitive image data is privacy sensitive information (PSI).

[0082] Based on this, during the model training process, this method needs to define privacy-sensitive information (PSI) and identify and mark the privacy preferences of each client.

[0083] Specifically, in financial scenarios, image training data can be text image data of financial texts (identity certificates, contracts). For proof image materials with official seals, some clients (such as banking institutions participating in joint modeling) may consider names and ID numbers to be the most important private information, and therefore mark them as relatively sensitive privacy-sensitive information (i.e., sensitive image data), while other institutions may consider ID numbers, official seals, and addresses to be the most important private information. In practice, the privacy-sensitive information PSI of different institutions can be accurately identified through manual or automated procedures.

[0084] After the privacy-sensitive information is determined, the privacy-sensitive information can be used to perform data sensitivity detection on the gradient data of each network layer, thereby determining the data sensitivity (ie, data-sensitive information) corresponding to the gradient data.

[0085] It should be noted that, in the process of performing data sensitivity detection on each update data based on the sensitive image data in the image training data, the Shapley value can be used to measure the sensitivity of the input value (i.e., the sensitive image data in the image training data) to the gradient data (i.e., data sensitive information).

[0086] In one or more embodiments provided in this specification, the update data is gradient data;

[0087] The performing data sensitivity detection on each update data according to the sensitive image data in the image training data to determine the data sensitivity information corresponding to each update data includes:

[0088] Determining sensitive image data included in the image training data;

[0089] The sensitive image data and the partial derivative matrix corresponding to each gradient data are used to perform data sensitivity detection on each gradient data to determine the data sensitive information corresponding to each gradient data.

[0090] The partial derivative matrix may be understood as a matrix containing partial derivatives of gradient data with respect to input data (ie, image training data). For example, the partial derivative matrix may be a Jacobian matrix.

[0091] Specifically, the method provided in this specification can determine sensitive image data from the image training data according to a pre-marked sensitive identifier, wherein the sensitive identifier can be a sensitivity level or a sensitive data label.

[0092] Then, the partial derivative matrix corresponding to each gradient data is calculated, and the sensitive matrix area is determined from the partial derivative matrix according to the sensitive image data, and the data sensitive information corresponding to each gradient data is determined according to the sensitive matrix area, thereby achieving accurate determination of the sensitivity of each gradient data (i.e., data sensitive information), which facilitates the subsequent accurate denoising of the gradient data.

[0093] In one or more embodiments provided in this specification, using the sensitive image data and the partial derivative matrix corresponding to the gradient data to perform data sensitivity detection on each gradient data to determine the data sensitive information corresponding to each gradient data includes:

[0094] Calculating the partial derivatives of each gradient data with respect to the image training data, and constructing the partial derivative matrix corresponding to each gradient data based on the partial derivatives;

[0095] Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information;

[0096] Based on the matrix parameters, the data sensitive information corresponding to each gradient data is calculated.

[0097] Using the above example, when the client sends gradient data to the server for parameter update, in order to avoid the risk of gradient data leaking sensitive data, the gradient data needs to be desensitized. In the process of data desensitization, it is first necessary to determine the sensitivity of each gradient data to PSI, that is, each client needs to calculate the sensitivity of the gradient data to the PSI area. The specific implementation method may include the following three steps:

[0098] Step 1: Calculate the Jacobian matrix of the gradient data corresponding to each network in the model, that is, the first-order partial derivative of the gradient with respect to the input data x. The calculation method of the Jacobian matrix can be referred to the following formula (1).

[0099]

[0100] Among them, g in the above formula l (x) represents the gradient of the input data x at layer l.

[0101] Step 2: Different clients determine the privacy preference areas PSI of different data samples, and determine the response area of ​​the privacy-sensitive area in the Jacobian matrix using the following formula (2): to locate.

[0102]

[0103] For each sample point (i.e., image training data), its privacy preference area can be expressed as w×h×c, where w represents the width of the area, h represents the height of the area, and c represents the number of channels in the area.

[0104] In the above formula, ||·|| F represents the L2 norm operation on the matrix elements; through this operation, the privacy-sensitive area PSI can be located in the response area of ​​the Jacobian matrix. The response area can be understood as the matrix parameter area corresponding to the privacy-sensitive area in the Jacobian matrix, and the matrix parameter area contains the matrix parameters of the sensitive image data; for example, the matrix parameter can be a partial derivative.

[0105] In one or more embodiments of the present specification, the first output layer of the network (image processing model) and the input image (i.e., image training data) are consistent in length and width, so the region positioning can be directly completed with the aid of the above formula (2). If the size of the intermediate output layer of the network used is inconsistent with the size of the input image, it can be considered to complete the size alignment by upsampling or downsampling before performing the operation.

[0106] Step 3: Different clients use the following formula (3) to calculate the gradient data corresponding to each network layer and the sensitivity of the privacy-sensitive area, thereby obtaining the sensitivity of each network layer:

[0107] Specifically, the client can determine the privacy-sensitive area, the response area of ​​the Jacobian matrix of a certain network layer

[0108] Then, the response area The matrix parameters in (i.e., the matrix parameters of sensitive image data) are averaged to obtain the PSI score S corresponding to a certain network layer. l (i.e. data sensitive information). The scalar value S l It reflects the quantitative results of a specific client's privacy preference area in its local dataset.

[0109]

[0110] By performing the above operations on each network layer, the PSI score S of each network layer is obtained. l , that is, sensitivity.

[0111] Based on the above steps, it can be seen that during the process of gradient sensitivity scoring, the client can calculate the sensitivity of the gradient to PSI. The specific implementation is to use the Jacobian matrix to evaluate the sensitivity of the gradient to the input data.

[0112] Step 206: Desensitize the updated data using the data sensitive information to obtain desensitized updated data of each network layer.

[0113] Among them, data desensitization can be understood as data noise addition operation or data disturbance operation; desensitized update data can be understood as updated data after desensitization.

[0114] Continuing with the above example, each client can use a random response mechanism to perturb the gradient data locally.

[0115] In one or more embodiments provided in this specification, the data sensitive information is data sensitivity;

[0116] The utilizing the data sensitive information to desensitize the updated data to obtain the desensitized updated data of each network layer includes:

[0117] Calculating data desensitization parameters corresponding to each updated data according to the data sensitivity of each updated data, wherein the data desensitization parameters of each updated data are the same or different;

[0118] Based on the data desensitization parameters corresponding to the updated data, the updated data are desensitized to obtain the desensitized updated data of each network layer.

[0119] Among them, the data desensitization parameter can be understood as a parameter for determining the degree of data desensitization for update data. For example, the data desensitization parameter can be a hyperparameter, a privacy budget, etc.

[0120] It should be noted that the data desensitization parameters for the update data of each network layer may be the same or different.

[0121] Specifically, after determining the data sensitivity, the method provided in this specification can calculate different or the same data desensitization parameters for the update data corresponding to different network layers according to the data sensitivity; then, use the data desensitization parameters to perform different degrees of data desensitization operations on each update data to obtain the desensitized update data of each network layer, thereby avoiding unnecessary data desensitization operations and avoiding performance losses caused by excessive privacy protection.

[0122] In one or more embodiments provided in this specification, the step of performing data desensitization on each update data based on the data desensitization parameter corresponding to each update data to obtain the desensitized update data of each network layer includes:

[0123] Determine a target network layer among the network layers, and determine a target data desensitization parameter corresponding to the target network layer, wherein the target network layer is any one of the network layers;

[0124] Determine the noise intensity information corresponding to the target data desensitization parameter, and use the noise intensity information to perform data noise addition on the update data of the target network layer to obtain the desensitized update data.

[0125] Using the above example, the client needs to allocate a privacy budget during the data noise addition process. The privacy budget allocation means: according to the sensitivity score of the gradient, a different privacy budget (∈ l ), please refer to the following formula (4) for details.

[0126]

[0127] Where ∈ is a hyperparameter representing the overall privacy budget used to enforce local differential privacy. In general, the ∈ l The smaller the value, the greater the noise intensity of the differential privacy plus noise, the stronger the privacy protection strength, and the lower the data availability; conversely, the ∈ l The larger the value, the smaller the noise intensity of differential privacy noise, the weaker the privacy protection strength, and the higher the data availability.

[0128] Then the client needs to perform gradient clipping and noise injection.

[0129] First, the client clips the gradients, transforming data points with large logarithmic values ​​into data points with smaller logarithmic values, thereby limiting the maximum impact of a single data point on the model update.

[0130] Secondly, according to the ∈ l , Gaussian noise is injected into the clipped gradient data to meet the allocated privacy budget and achieve LDP. The calculation formulas (5) and (6) of the steps are as follows:

[0131]

[0132] Based on the above steps, we can see that in order to address the problems of insufficient privacy protection and performance degradation in the existing joint modeling scheme for private images, this scheme proposes a personalized differential privacy protection method for federated learning based on protection preference analysis. Specifically, by allocating privacy budgets layer by layer, unnecessary noise for privacy-insensitive information is reduced, thereby avoiding the performance loss caused by excessive privacy protection.

[0133] Step 208: Send the desensitized update data of each network layer to the second training unit for parameter update.

[0134] Continuing with the above example, the client sends the perturbed gradient data to the server.

[0135] In one or more embodiments of the present specification, the image processing model training method applied to the first training unit can receive the image processing model sent by the second training unit during the distributed machine learning process, and use local image training data to train the image processing model to obtain update data of each network layer in the image processing model; in the process of sending the updated data to the second training unit for parameter updating, in order to avoid the risk that the updated data may leak sensitive data, it is necessary to perform data sensitivity detection on each updated data based on the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each updated data, and use the data sensitive information to desensitize each updated data, and then send the desensitized update data to the second training unit, thereby avoiding the problem of possible exposure of sensitive privacy information in the distributed machine learning process, ensuring data security, and reducing the risk of sensitive data leakage.

[0136] The following combination Figure 3 , taking the application of the image processing model training method provided in this specification in the personalized differential privacy protection scenario of federated learning as an example, the image processing model training method is further explained. Among them, Figure 3 A processing flow chart of an image processing model training method provided in one embodiment of the present specification is shown. The image processing model training method is applied to a federated learning system (i.e., an image processing model training system), which includes a server (Server) and multiple clients (Client).

[0137] Among them, the client refers to multiple clients participating in the federated learning, each of which has a local data set. The server is responsible for managing the training process, including the collection and aggregation of gradients and the updating and distribution of model parameters.

[0138] The image processing model training method specifically includes the following steps.

[0139] Step 302: Initialize the global model.

[0140] Specifically, the server can initialize a global model (i.e., image processing model) and distribute it to all participating clients.

[0141] Step 304: Local model training.

[0142] Specifically, each client uses its local dataset to train the received global model and generate model updates (gradient data).

[0143] Step 306: Privacy sensitive information (PSI) identification.

[0144] Specifically, the client,identifies the PSI in the local training data and assigns a corresponding privacy,preference to each PSI data point (i.e., PSI region).

[0145] The privacy preference can be understood as a label representing that a certain image region in the image training data is private data.

[0146] For example, in financial scenarios, image training data can be text image data of financial texts (identity certificates, contracts). For proof image materials with official seals, some clients (such as banking institutions participating in joint modeling) may consider name and ID number to be the most important private information, and therefore mark them as relatively sensitive privacy-sensitive information PSI, while other institutions may consider ID number, official seal, and address to be the most important privacy-sensitive information PSI. In practice, the privacy-sensitive information PSI of different institutions can be accurately identified through manual or automated procedures.

[0147] Step 308: Gradient sensitivity scoring.

[0148] Specifically, the client can calculate the sensitivity of the gradient to the PSI. In the method provided in this specification, the Jacobian matrix can be used to evaluate the sensitivity of the gradient to the input data. The specific implementation method refers to the following steps:

[0149] (1) Calculate the Jacobian matrix J of the gradient data corresponding to each network in the model l (x), that is, the first-order partial derivative of the gradient with respect to the input data x, the Jacobian matrix J l The calculation method of (x) can refer to the following formula (1).

[0150]

[0151] Among them, g in the above formula l (x) represents the gradient of the input data x at the lth layer; y in the above formula represents the output data of the lth layer of the network; w represents the weight data of the lth layer of the network; is the symbol of partial derivative.

[0152] (2) Different clients determine the privacy preference areas PSI of different data samples, and determine the response area of ​​the privacy-sensitive area in the Jacobian matrix through the following formula (2): to locate.

[0153]

[0154] Where a and b represent the a-th row and b-th column of the Jacobian matrix.

[0155] For each sample point (i.e., image training data), its privacy preference region can be expressed as w×h×c, where w represents the width of the region, h represents the height of the region, and c represents the number of channels in the region.

[0156] In the above formula, ||·|| F represents the L2 norm operation of the matrix elements; through this operation, the privacy-sensitive area PSI can be located in the response area of ​​the Jacobian matrix. The response area can be understood as the matrix parameter area corresponding to the privacy-sensitive area in the Jacobian matrix.

[0157] In one or more embodiments of the present specification, the length and width of the first output layer of the network (image processing model) and the input image (i.e., image training data) are consistent, so the region positioning can be directly completed by means of the above formula (2). If the size of the intermediate output layer of the network used is inconsistent with the input image, it can be considered to complete the size alignment by upsampling or downsampling before performing the operation.

[0158] (3) Different clients use the following formula (3) to calculate the gradient data corresponding to each network layer and the sensitivity of the privacy-sensitive area, thereby obtaining the sensitivity S of each network layer: l :

[0159] Specifically, the client can determine the privacy-sensitive area, the response area of ​​the Jacobian matrix of a certain network layer

[0160] Then, the response area The matrix parameters (i.e. partial derivatives) in are averaged to obtain the PSI score S corresponding to a certain network layer. l This scalar value reflects a specific client's quantification of its privacy preference area in its local dataset.

[0161]

[0162] By performing the above operations on each network layer, the PSI score S of each network layer is obtained. l .

[0163] Step 310: Privacy budget allocation.

[0164] Specifically, according to the sensitivity score S of the gradient data corresponding to each network layer l , the client can allocate different privacy budgets to each network layer (∈ l ), please refer to the following formula (4) for details.

[0165]

[0166] where ∈ is a hyperparameter representing the overall privacy budget used to enforce local differential privacy.

[0167] Generally speaking, the ∈ l The smaller the value, the greater the noise intensity of the differential privacy plus noise, the stronger the privacy protection strength, and the lower the data availability; conversely, the ∈ l The larger the value, the smaller the noise intensity of differential privacy noise, the weaker the privacy protection strength, and the higher the data availability.

[0168] Step 312: Gradient clipping and noise injection.

[0169] First, the client clips the gradient, converting some data points with larger values ​​to data points with average values, thereby limiting the maximum impact of a single data point on the model update.

[0170] Secondly, according to the ∈ l , Gaussian noise is injected into the clipped gradient data to meet the allocated privacy budget and achieve LDP. The calculation formulas (5) and (6) of the steps are as follows:

[0171]

[0172]

[0173] Among them, C in the above formula represents the gradient boundary, which is a manually set hyperparameter. During the process of gradient clipping on the client, C can be used to convert some data points with larger values ​​into data points with values ​​that match the average size, thereby limiting the maximum impact of a single data point on the model update.

[0174] In the above formula, N() represents Gaussian distribution; N(mean, variance). The meaning of this formula is to add a mean of 0 and a variance of (C^2*σ l ^2). This variance is calculated based on the set C and the subsequent allocation, so the variance calculation results of different layers will be different to achieve the effect of adaptive allocation.

[0175] In the above formula In this formula, δ is a hyperparameter that reflects the probability of random failure of the differential privacy algorithm, T represents the communication round, and I represents the unit matrix. With this value, the layered adaptive noise algorithm in this specification satisfies (ε,δ)-DP. This theoretically guarantees the privacy protection strength of the algorithm proposed in this article.

[0176] in, is the gradient data of the lth layer after adding noise; It is the gradient data after the model is noised.

[0177] Step 314: Upload the perturbation gradient.

[0178] Specifically, the client sends the gradient data after noise disturbance to the server.

[0179] Based on the above steps, this image processing model training method provides a personalized differential privacy protection method for federated learning based on protection preference analysis, which aims to solve the privacy leakage problem in federated learning; this method uses a layer-by-layer local differential privacy mechanism to protect the privacy of the gradient uploaded by the client. Specifically, first, define privacy-sensitive information, then calculate the sensitivity of the gradient to the PSI area, and finally, allocate different privacy budgets to each network layer to achieve strict privacy protection of PSI. Through a layer-by-layer local differential privacy mechanism, a different privacy budget is allocated to each network layer according to the sensitive information preference of the local client, reducing noise interference on privacy-insensitive information, thereby achieving personalized differential privacy protection for clients with different preferences, and ultimately enabling the federated solution to achieve an effective balance between privacy protection and model accuracy.

[0180] In addition, an adaptive privacy allocation algorithm is implemented by allocating different privacy budgets to each network layer. This method avoids excessive expert experience and model customization and is a local privacy-preserving federated learning method with stronger generalization and better accuracy.

[0181] See also Figure 4 , Figure 4 A flowchart of another image processing model training method provided according to an embodiment of the present specification is shown. The method is applied to the second training unit and specifically includes the following steps.

[0182] Step 402: Send the image processing model to multiple first training units.

[0183] Taking the application of another image processing model training method in this specification in the personalized differential privacy protection scenario of federated learning as an example, another image processing model training method is explained, wherein the first training unit is the client of federated learning, the second training unit is the server of federated learning, the desensitized update data is the desensitized gradient data, and the sensitive image data is privacy sensitive information (PSI).

[0184] Based on this, the server provided in this specification can initialize a global model and distribute it to all participating clients.

[0185] Step 404: Receive desensitized update data of each network layer in the image processing model sent by each first training unit, wherein the desensitized update data is obtained by the first training unit performing data desensitization on the update data of each network layer, and the update data is obtained by the first training unit performing model training on the image processing model using the image training data.

[0186] Using the above example, the server can collect the desensitized gradient data of all clients. The specific method of determining the desensitized gradient data can refer to the above embodiment.

[0187] Step 406: Using the sensitive image data in the second training unit, perform data sensitivity detection on each desensitized update data to determine the data sensitivity information corresponding to each desensitized update data.

[0188] In one or more embodiments provided in this specification, the desensitized update data is desensitized gradient data;

[0189] The method of using the sensitive image data in the second training unit to perform data sensitivity detection on each desensitized update data to determine data sensitivity information corresponding to each desensitized update data includes:

[0190] Determining sensitive detection image data in the second training unit, and determining the sensitive image data from the sensitive detection image data;

[0191] The sensitive image data and the partial derivative matrices corresponding to the desensitized gradient data are used to perform data sensitivity detection on the desensitized gradient data to determine the data sensitive information corresponding to the desensitized gradient data.

[0192] The above steps can be referred to the embodiment of determining data sensitive information in the above-mentioned image processing model training method, which will not be described in detail here.

[0193] In one or more embodiments provided in this specification, using the sensitive image data and the partial derivative matrix corresponding to each desensitized gradient data to perform data sensitivity detection on each desensitized gradient data to determine the data sensitive information corresponding to each desensitized gradient data includes:

[0194] Calculate the partial derivatives of each desensitized gradient data with respect to the sensitive detection image data, and construct the partial derivative matrix corresponding to each desensitized gradient data based on the partial derivatives;

[0195] Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information;

[0196] Based on the matrix parameters, the data sensitive information corresponding to each desensitized gradient data is calculated.

[0197] The above steps can be referred to the embodiment of determining data sensitive information in the above-mentioned image processing model training method, which will not be described in detail here.

[0198] Using the above example, in response to the problems of insufficient privacy protection and performance degradation in the existing joint modeling scheme for private images, this method proposes a federated learning personalized differential privacy protection method based on protection preference analysis. This method optimizes the gradient aggregation process of the global model through the parameter aggregation mechanism, reduces the noise introduced by local gradient perturbations, and improves model performance; that is, this method proposes a parameter aggregation mechanism based on the distribution of perturbation information in the gradient aggregation step. The specific implementation method is:

[0199] The server collects the gradient data of all clients (i.e., desensitized gradient data) and uses the public data set to evaluate the sensitivity of the gradient data. The specific sensitivity can be determined by the following formula (7).

[0200]

[0201] Among them, PSI represents the calculation project of PSI sensitivity in the above embodiment, that is, the central server uses shareable open source data to quantitatively score the privacy-sensitive area information of the gradient matrices uploaded by different clients to complete the evaluation of the disturbance of the gradients uploaded by different clients.

[0202] Step 408: Based on the data sensitivity information, determine the target desensitized update data from the desensitized update data.

[0203] In one or more embodiments provided in this specification, the data sensitive information is data sensitivity;

[0204] The determining target desensitized update data from the desensitized update data based on the data sensitivity information includes:

[0205] The data sensitivity of each desensitized update data is determined, and the desensitized update data whose data sensitivity is less than or equal to a preset score threshold is determined as the target desensitized update data.

[0206] Specifically, this method can perform weighted aggregation according to the sensitivity of the gradient, and give priority to aggregating the gradients with lower sensitivity in the local gradients, which is specifically determined by the following formula (8); this method gives priority to aggregating the gradients with lower sensitivity, thereby reducing the impact of noise on the performance of the global model.

[0207]

[0208] Step 410: Utilize the target desensitization update data to update the parameters of each network layer in the image processing model to obtain an updated image processing model.

[0209] Continuing with the previous example, the server uses gradient plastic to update the global model parameters and evaluate the model performance.

[0210] In one or more embodiments provided in this specification, after updating the parameters of each network layer in the image processing model by using the target desensitization update data to obtain the updated image processing model, the method further includes:

[0211] In the case where it is determined that the updated image processing model satisfies the model training stop condition, determining the updated image processing model as the trained image processing model;

[0212] When it is determined that the updated image processing model does not meet the model training stop condition, the updated image processing model is used as the image processing model to be trained, and the step of sending the image processing model to multiple first training units is continued until the updated image processing model meets the model training stop condition.

[0213] Continuing with the above example, the server sends the updated model parameters back to all clients for the next round of training. The model training steps are repeated until the model reaches the predetermined performance standard or completes the set training cycle.

[0214] In one or more embodiments of the present specification, the image processing model training method applied to the second training unit may send the image processing model to multiple first training units during distributed machine learning;

[0215] Each first training unit uses local image training data to perform model training on the image processing model to obtain update data of each network layer in the image processing model; in the process of sending the update data to the second training unit for parameter update, in order to avoid the risk that the update data may leak sensitive data, it is necessary to perform data sensitivity detection on each update data according to the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each update data, and use the data sensitive information to desensitize each update data, and then send the desensitized update data to the second training unit;

[0216] After receiving the desensitized update data sent by the first training unit, since the data desensitization processing causes the desensitized update data to contain interference information, in order to avoid the interference information from affecting the performance of the image processing model, the sensitive image data in the second training unit can be used to perform data sensitivity detection on each desensitized update data to determine the data sensitive information corresponding to each desensitized update data, and based on the data sensitive information, determine the target desensitized update data with lower sensitivity from each desensitized update data, and then use a small number of target desensitized update data to update the parameters of each network layer in the image processing model, thereby avoiding excessive interference information from affecting the performance of the image processing model and improving the processing efficiency of the image processing model.

[0217] The above is a schematic scheme of another image processing model training method of this embodiment. It should be noted that the technical scheme of the other image processing model training method belongs to the same concept as the technical scheme of the above-mentioned one image processing model training method. The details of the technical scheme of the other image processing model training method that are not described in detail can be found in the description of the technical scheme of the above-mentioned one image processing model training method.

[0218] The following combination Figure 5 , taking the application of another image processing model training method provided in this specification in the personalized differential privacy protection scenario of federated learning as an example, the another image processing model training method is further described. Among them, Figure 5 A processing flow chart of another image processing model training method provided by an embodiment of the present specification is shown. The other image processing model training method is applied to the server of the federated learning system and specifically includes the following steps.

[0219] Step 502: performing gradient aggregation on the desensitized gradient data sent by each client.

[0220] Specifically, this method proposes a parameter aggregation mechanism based on the distribution of perturbation information in the gradient aggregation step. In practice, the server collects the gradient data of all clients and performs weighted aggregation according to the sensitivity of the gradient data, giving priority to aggregating the gradients with lower sensitivity in the local gradients and updating the global model. The specific execution steps are:

[0221] (1) This mechanism uses public data sets to evaluate the sensitivity of gradients. It can be determined by the following formula (7).

[0222]

[0223] In the above formula, PSI: represents the computational engineering for PSI sensitivity performed by the above client; that is, the central server uses shareable open source data to quantitatively score the privacy-sensitive area information of the gradient matrices uploaded by different clients to complete the evaluation of the disturbance of the gradients uploaded by different clients.

[0224] It should be noted that the sensitivity determined in the above formula (7) is For details, please refer to the steps in the client above to calculate the sensitivity of the gradient to PSI, which will not be described here.

[0225] (2) Prioritize the aggregation of gradients with lower sensitivity to reduce the impact of noise on the global model performance.

[0226] Specifically, this mechanism performs weighted processing on the gradient data according to the sensitivity of the gradient data; for example, the smaller the sensitivity, the greater the weight.

[0227] Then, the gradients are aggregated according to the weights corresponding to the gradient data, and the gradients with lower sensitivity in the local gradients are aggregated preferentially, which can be specifically achieved by the following formula (8).

[0228]

[0229] in, is the weighted gradient data.

[0230] Step 504: Global model update.

[0231] Specifically, the server can use the less sensitive gradients to update the global model parameters and evaluate the model performance.

[0232] Step 506: Model parameter distribution.

[0233] Specifically, the server sends the updated model parameters back to all clients for the next round of training.

[0234] Step 508: Iterative training.

[0235] Specifically, step 502 to step 506 are repeated until the model reaches a predetermined performance standard or completes a set training cycle.

[0236] Based on the above steps, it can be seen that this image processing model training method provides a personalized differential privacy protection method for federated learning based on protection preference analysis, which aims to solve the privacy leakage problem in federated learning; this method introduces a novel parameter aggregation mechanism, which uses public data sets to evaluate the sensitivity of gradients, preferentially aggregates gradients with lower sensitivity in local gradients, reduces the noise introduced by local perturbation gradients, and thus maintains the effectiveness of the global model; it achieves superior performance in both privacy and accuracy.

[0237] Corresponding to the above method embodiment, this specification also provides an image processing model training system embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of an image processing model training system provided by an embodiment of the present specification. Figure 6 As shown, the system server 602 and multiple clients 604, wherein:

[0238] The server 602 is used to send the image processing model to the multiple clients 604, receive the desensitized update data of each network layer in the image processing model sent by each client 604, wherein the desensitized update data is obtained by the client 604 performing data desensitization on the update data of each network layer, and the update data is obtained by the client 604 performing model training on the image processing model using image training data, and perform data sensitivity detection on each desensitized update data using the sensitive image data in the server 602 to determine the data sensitive information corresponding to each desensitized update data, determine the target desensitized update data from each desensitized update data based on the data sensitive information, and use the target desensitized update data to perform parameter update on each network layer in the image processing model to obtain an updated image processing model;

[0239] The client 604 is used to receive the image processing model sent by the server 602, and use the image training data to train the image processing model to obtain the update data of each network layer in the image processing model, wherein the update data is used to update the parameters of each network layer, and perform data sensitivity detection on each update data according to the sensitive image data in the image training data to determine the data sensitive information corresponding to each update data, and use the data sensitive information to desensitize the each update data to obtain the desensitized update data of each network layer, and send the desensitized update data of each network layer to the server 602 for parameter update.

[0240] Among them, the above-mentioned server 602 can be understood as the second training unit in the above-mentioned image processing model training; the above-mentioned client 604 can be understood as the first training unit in the above-mentioned image processing model training.

[0241] One or more embodiments of this specification provide an image processing model training system including a server and multiple clients, wherein the server can send the image processing model to the multiple clients during distributed machine learning;

[0242] Each client uses local image training data to train the image processing model and obtains update data of each network layer in the image processing model; in the process of sending the update data to the server for parameter update, in order to avoid the risk of sensitive data leakage due to the update data, it is necessary to perform data sensitivity detection on each update data according to the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each update data, and use the data sensitive information to desensitize each update data, and then send the desensitized update data to the server;

[0243] After the server receives the desensitized update data, since the data desensitization processing causes the desensitized update data to contain interference information, in order to avoid the interference information from affecting the performance of the image processing model, the server can use the sensitive image data in the server to perform data sensitivity detection on each desensitized update data, determine the data sensitive information corresponding to each desensitized update data, and determine the target desensitized update data with lower sensitivity from each desensitized update data based on the data sensitive information, and then use a small number of target desensitized update data to update the parameters of each network layer in the image processing model, thereby avoiding excessive interference information from affecting the performance of the image processing model and improving the processing efficiency of the image processing model.

[0244] The above is a schematic scheme of an image processing model training system of this embodiment. It should be noted that the technical scheme of the image processing model training system and the technical scheme of any of the above-mentioned image processing model training methods belong to the same concept, and the details not described in detail in the technical scheme of the image processing model training system can be referred to the description of the technical scheme of any of the above-mentioned image processing model training methods.

[0245] Corresponding to the above method embodiment, this specification also provides an image processing model training device embodiment, which is applied to a first training unit and includes:

[0246] A model training module is configured to receive the image processing model sent by the second training unit, and perform model training on the image processing model using the image training data to obtain update data of each network layer in the image processing model, wherein the update data is used to update parameters of each network layer;

[0247] A data sensitivity detection module is configured to perform data sensitivity detection on each update data according to the sensitive image data in the image training data, and determine the data sensitivity information corresponding to each update data;

[0248] A data desensitization module is configured to use the data sensitive information to perform data desensitization on each update data to obtain desensitized update data of each network layer;

[0249] The data sending module is configured to send the desensitized update data of each network layer to the second training unit for parameter updating.

[0250] Optionally, the update data is gradient data;

[0251] The data sensitivity detection module is further configured to:

[0252] Determining sensitive image data included in the image training data;

[0253] The sensitive image data and the partial derivative matrix corresponding to each gradient data are used to perform data sensitivity detection on each gradient data to determine the data sensitive information corresponding to each gradient data.

[0254] Optionally, the data sensitivity detection module is further configured to:

[0255] Calculating the partial derivatives of each gradient data with respect to the image training data, and constructing the partial derivative matrix corresponding to each gradient data based on the partial derivatives;

[0256] Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information;

[0257] Based on the matrix parameters, the data sensitive information corresponding to each gradient data is calculated.

[0258] Optionally, the data sensitive information is data sensitivity;

[0259] The data desensitization module is further configured to:

[0260] Calculating data desensitization parameters corresponding to each updated data according to the data sensitivity of each updated data, wherein the data desensitization parameters of each updated data are the same or different;

[0261] Based on the data desensitization parameters corresponding to the updated data, the updated data are desensitized to obtain the desensitized updated data of each network layer.

[0262] Optionally, the data desensitization module is further configured to:

[0263] Determine a target network layer among the network layers, and determine a target data desensitization parameter corresponding to the target network layer, wherein the target network layer is any one of the network layers;

[0264] Determine the noise intensity information corresponding to the target data desensitization parameter, and use the noise intensity information to perform data noise addition on the update data of the target network layer to obtain the desensitized update data.

[0265] In one or more embodiments of the present specification, the image processing model training device applied to the first training unit can receive the image processing model sent by the second training unit during the distributed machine learning process, and use local image training data to train the image processing model to obtain update data of each network layer in the image processing model; in the process of sending the updated data to the second training unit for parameter updating, in order to avoid the risk that the updated data may leak sensitive data, it is necessary to perform data sensitivity detection on each updated data based on the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each updated data, and use the data sensitive information to desensitize each updated data, and then send the desensitized update data to the second training unit, thereby avoiding the problem of possible exposure of sensitive privacy information in the distributed machine learning process, ensuring data security, and reducing the risk of sensitive data leakage.

[0266] The above is a schematic scheme of an image processing model training device of this embodiment. It should be noted that the technical scheme of the image processing model training device and the technical scheme of the above-mentioned image processing model training method belong to the same concept, and the details not described in detail in the technical scheme of the image processing model training device can be referred to the description of the technical scheme of the above-mentioned image processing model training method.

[0267] Corresponding to the above method embodiment, this specification also provides another image processing model training device embodiment, which is applied to the second training unit and includes:

[0268] A model sending module, configured to send the image processing model to a plurality of first training units;

[0269] a data receiving module configured to receive desensitized update data of each network layer in the image processing model sent by each first training unit, wherein the desensitized update data is obtained by the first training unit performing data desensitization on the update data of each network layer, and the update data is obtained by the first training unit performing model training on the image processing model using the image training data;

[0270] A data sensitivity detection module is configured to use the sensitive image data in the second training unit to perform data sensitivity detection on each desensitized update data to determine the data sensitivity information corresponding to each desensitized update data;

[0271] A data determination module is configured to determine target desensitized update data from the desensitized update data based on the data sensitivity information;

[0272] The parameter updating module is configured to use the target desensitization update data to update the parameters of each network layer in the image processing model to obtain an updated image processing model.

[0273] Optionally, the desensitized update data is desensitized gradient data;

[0274] The data sensitivity detection module is further configured to:

[0275] Determining sensitive detection image data in the second training unit, and determining the sensitive image data from the sensitive detection image data;

[0276] The sensitive image data and the partial derivative matrices corresponding to the desensitized gradient data are used to perform data sensitivity detection on the desensitized gradient data to determine the data sensitive information corresponding to the desensitized gradient data.

[0277] Optionally, the data sensitivity detection module is further configured to:

[0278] Calculate the partial derivatives of each desensitized gradient data with respect to the sensitive detection image data, and construct the partial derivative matrix corresponding to each desensitized gradient data based on the partial derivatives;

[0279] Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information;

[0280] Based on the matrix parameters, the data sensitive information corresponding to each desensitized gradient data is calculated.

[0281] Optionally, the data sensitive information is data sensitivity;

[0282] The data determination module is further configured to:

[0283] The data sensitivity of each desensitized update data is determined, and the desensitized update data whose data sensitivity is less than or equal to a preset score threshold is determined as the target desensitized update data.

[0284] The another image processing model training device also includes a model processing module, which is configured to:

[0285] In the case where it is determined that the updated image processing model satisfies the model training stop condition, determining the updated image processing model as the trained image processing model;

[0286] When it is determined that the updated image processing model does not meet the model training stop condition, the updated image processing model is used as the image processing model to be trained, and the step of sending the image processing model to multiple first training units is continued until the updated image processing model meets the model training stop condition.

[0287] In one or more embodiments of this specification, the image processing model training device applied to the second training unit may send the image processing model to multiple first training units during distributed machine learning;

[0288] Each first training unit uses local image training data to perform model training on the image processing model to obtain update data of each network layer in the image processing model; in the process of sending the update data to the second training unit for parameter update, in order to avoid the risk that the update data may leak sensitive data, it is necessary to perform data sensitivity detection on each update data according to the sensitive image data in the image training data, so as to determine the data sensitive information corresponding to each update data, and use the data sensitive information to desensitize each update data, and then send the desensitized update data to the second training unit;

[0289] After the second training unit receives the desensitized update data, since the data desensitization processing causes the desensitized update data to contain interference information, in order to avoid the interference information from affecting the performance of the image processing model, the sensitive image data in the second training unit can be used to perform data sensitivity detection on each desensitized update data, determine the data sensitive information corresponding to each desensitized update data, and determine target desensitized update data with lower sensitivity from each desensitized update data based on the data sensitive information, and then use a small number of target desensitized update data to update the parameters of each network layer in the image processing model, thereby avoiding excessive interference information from affecting the performance of the image processing model and improving the processing efficiency of the image processing model.

[0290] The above is a schematic scheme of another image processing model training device of this embodiment. It should be noted that the technical scheme of the another image processing model training device and the technical scheme of the another image processing model training method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the another image processing model training device can be referred to the description of the technical scheme of the another image processing model training method mentioned above.

[0291] Figure 7The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0292] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0293] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0294] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0295] Among them, the processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of any one of the above-mentioned image processing model training methods.

[0296] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to any one of the image processing model training method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of any one of the image processing model training method embodiments.

[0297] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of any one of the above-mentioned image processing model training methods when executed by a processor.

[0298] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to any one of the image processing model training method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of any one of the image processing model training method embodiments.

[0299] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any one of the above-mentioned image processing model training methods.

[0300] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of any of the above-mentioned image processing model training methods belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of any of the above-mentioned image processing model training methods.

[0301] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0302] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0303] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0304] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0305] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing model training method, applied to a first training unit, comprising: Receiving the image processing model sent by the second training unit, and performing model training on the image processing model using the image training data to obtain update data of each network layer in the image processing model, wherein the update data is used to update parameters of each network layer; According to the sensitive image data in the image training data, data sensitivity detection is performed on each update data to determine the data sensitivity information corresponding to each update data; Desensitizing the updated data using the data sensitive information to obtain desensitized updated data of each network layer; The desensitized update data of each network layer is sent to the second training unit for parameter update.

2. According to the image processing model training method according to claim 1, the update data is gradient data; The step of performing data sensitivity detection on each update data according to the sensitive image data in the image training data to determine the data sensitivity information corresponding to each update data includes: Determining sensitive image data included in the image training data; The sensitive image data and the partial derivative matrix corresponding to each gradient data are used to perform data sensitivity detection on each gradient data to determine the data sensitive information corresponding to each gradient data.

3. The image processing model training method according to claim 2, wherein the sensitive image data and the partial derivative matrix corresponding to the gradient data are used to perform data sensitivity detection on each gradient data to determine the data sensitive information corresponding to each gradient data, including: Calculating the partial derivatives of each gradient data with respect to the image training data, and constructing the partial derivative matrix corresponding to each gradient data based on the partial derivatives; Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information; Based on the matrix parameters, the data sensitive information corresponding to each gradient data is calculated.

4. According to the image processing model training method of claim 1, the data sensitive information is data sensitivity; The utilizing the data sensitive information to desensitize the updated data to obtain the desensitized updated data of each network layer includes: Calculating data desensitization parameters corresponding to each updated data according to the data sensitivity of each updated data, wherein the data desensitization parameters of each updated data are the same or different; Based on the data desensitization parameters corresponding to the updated data, the updated data are desensitized to obtain the desensitized updated data of each network layer.

5. The image processing model training method according to claim 4, wherein the step of performing data desensitization on each update data based on the data desensitization parameters corresponding to each update data to obtain the desensitized update data of each network layer comprises: Determine a target network layer among the network layers, and determine a target data desensitization parameter corresponding to the target network layer, wherein the target network layer is any one of the network layers; Determine the noise intensity information corresponding to the target data desensitization parameter, and use the noise intensity information to perform data noise addition on the update data of the target network layer to obtain the desensitized update data.

6. An image processing model training method, applied to the second training unit, comprising: sending the image processing model to a plurality of first training units; receiving desensitized update data of each network layer in the image processing model sent by each first training unit, wherein the desensitized update data is obtained by the first training unit performing data desensitization on the update data of each network layer, and the update data is obtained by the first training unit performing model training on the image processing model using the image training data; Using the sensitive image data in the second training unit, perform data sensitivity detection on each desensitized update data to determine data sensitivity information corresponding to each desensitized update data; Based on the data sensitive information, determining target desensitized update data from the desensitized update data; The target desensitization update data is used to update the parameters of each network layer in the image processing model to obtain an updated image processing model.

7. The image processing model training method according to claim 6, wherein the desensitized update data is desensitized gradient data; The method of using the sensitive image data in the second training unit to perform data sensitivity detection on each desensitized update data to determine data sensitivity information corresponding to each desensitized update data includes: Determining sensitive detection image data in the second training unit, and determining the sensitive image data from the sensitive detection image data; The sensitive image data and the partial derivative matrices corresponding to the desensitized gradient data are used to perform data sensitivity detection on the desensitized gradient data to determine the data sensitive information corresponding to the desensitized gradient data.

8. The image processing model training method according to claim 7, wherein the sensitive image data and the partial derivative matrix corresponding to each desensitized gradient data are used to perform data sensitivity detection on each desensitized gradient data to determine the data sensitive information corresponding to each desensitized gradient data, including: Calculate the partial derivatives of each desensitized gradient data with respect to the sensitive detection image data, and construct the partial derivative matrix corresponding to each desensitized gradient data based on the partial derivatives; Determine data region information of the sensitive image data, and determine matrix parameters of the sensitive image data from the partial derivative matrix based on the data region information; Based on the matrix parameters, the data sensitive information corresponding to each desensitized gradient data is calculated.

9. The image processing model training method according to claim 6, wherein the data sensitive information is data sensitivity; The determining target desensitized update data from the desensitized update data based on the data sensitivity information includes: The data sensitivity of each desensitized update data is determined, and the desensitized update data whose data sensitivity is less than or equal to a preset score threshold is determined as the target desensitized update data.

10. The image processing model training method according to claim 6, wherein after the target desensitization update data is used to update the parameters of each network layer in the image processing model to obtain the updated image processing model, the method further comprises: In the case where it is determined that the updated image processing model satisfies the model training stop condition, determining the updated image processing model as the trained image processing model; When it is determined that the updated image processing model does not meet the model training stop condition, the updated image processing model is used as the image processing model to be trained, and the step of sending the image processing model to multiple first training units is continued until the updated image processing model meets the model training stop condition.

11. An image processing model training system, comprising a server and multiple clients, wherein: The server is used to send the image processing model to the multiple clients, and receive desensitized update data of each network layer in the image processing model sent by each client, wherein the desensitized update data is obtained by the client performing data desensitization on the update data of each network layer, and the update data is obtained by the client performing model training on the image processing model using image training data, and the sensitive image data in the server is used to perform data sensitivity detection on each desensitized update data to determine the data sensitive information corresponding to each desensitized update data, and based on the data sensitive information, target desensitized update data is determined from each desensitized update data, and the target desensitized update data is used to update the parameters of each network layer in the image processing model to obtain an updated image processing model; The client is used to receive the image processing model sent by the server, and use the image training data to perform model training on the image processing model to obtain the update data of each network layer in the image processing model, wherein the update data is used to update the parameters of each network layer, perform data sensitivity detection on each update data according to the sensitive image data in the image training data, determine the data sensitive information corresponding to each update data, use the data sensitive information to desensitize the each update data, obtain the desensitized update data of each network layer, and send the desensitized update data of each network layer to the server for parameter update.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images