Distributed sight line estimation method and system capable of protecting privacy

Abstract

The invention provides a distributed sight line estimation method and system capable of protecting privacy. The method comprises the following steps: processing individual sight line data acquired by a client i through a privacy protector Ppi to remove privacy and sight line irrelevant information to obtain an individual sight line map Gi, and sending the individual sight line map Gi to a server; extracting the individual sight map Gi by using a pre-trained sight knowledge extractor Ex on the server to obtain an individual sight embedded Ei, and sending the individual sight embedded Ei back to the client i; processing the individual sight line embedding Ei based on a personalized estimation head Hi on the client i to obtain individual sight line estimation; according to the method, a distributed sight line estimation framework with privacy protection capability is provided, sight line estimation training and reasoning speed can be effectively accelerated, generalization and individuation are improved, and the method has distributed optimization and individualized deployment capability and has a wider application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of gaze estimation, and particularly relates to a privacy-protected distributed gaze estimation method and system. Background Art

[0002] The eyes are the windows of the soul. As a behavioral biometric feature of the eyes, gaze estimation reflects the change in the position of the fixation point when a human observes an object, and has become an important way to understand the user's intentions, mental state, etc. in human-computer interaction. With the development of technologies such as augmented reality, virtual reality, and autonomous driving, and the recent rise of concepts such as the metaverse, the demand for accurate and efficient gaze estimation technology has exploded, and it has gradually become a research hotspot in the field of intelligent human-computer interaction. In recent years, appearance-based gaze estimation methods have developed rapidly and achieved success in many application scenarios. However, current appearance-based gaze estimation methods rely on centralized large-scale datasets for sufficient training to obtain good generalization performance. As gaze data is a behavioral biometric feature of the eyes, the collection and aggregation of this data may cause privacy leakage problems. Specifically, most existing gaze estimation methods use at least one complete eye image for gaze direction extraction, and some even use the entire face for additional head pose calibration to achieve accurate estimation. Because this data contains sensitive information of users, such as eye movement information, facial features, personal identity, individual behavior preferences and mental states, decision-making cognition, etc., there may be privacy leakage risks during data collection, transmission, and processing. With the strict implementation of privacy protection laws such as the Data Security Law and the Personal Information Protection Law, the issues of individual data security and privacy protection in gaze estimation have become unavoidable, and the privacy leakage risk needs to attract sufficient attention from researchers. Once the raw data containing highly private information is transmitted from a local device, an individual loses control of their privacy and thus incurs privacy risks. On the other hand, recent research has verified that most of the privacy information entangled in gaze features has no direct relevance to the gaze estimation task itself, and may even damage the generalization ability of the model.

[0003] Current appearance-based gaze estimation methods generally adopt a deep learning framework and obtain a model through training on a large number of raw eye or face image samples. However, building a large-scale gaze data center has security risks and is restricted by privacy protection laws because the raw gaze data contains a large amount of highly private individual features, and once the raw data is leaked, it is irreversible. On the other hand, the generalization ability of a gaze estimation model lacking a large amount of training data will be greatly reduced in practical applications. At the same time, due to differences in physiological structures among individuals, such as differences in eyeball structures and head postures, a general gaze estimation model cannot be personalized for individuals.

[0004] Therefore, there is an urgent need to develop a distributed gaze estimation method and system that can protect the individual privacy of users and be personalized and adapted to solve the above technical problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a privacy-protected distributed gaze estimation method and system to solve the above technical problems.

[0006] The first aspect of the present invention discloses a privacy-protected distributed gaze estimation method, and the method includes:

[0007] Step S1: Process the individual gaze data obtained by the client i through the privacy protector P p i to remove information irrelevant to privacy and gaze, and obtain an individual gaze map g i , and send the individual gaze map g i to the server;

[0008] Step S2: Use the gaze knowledge extractor E pre-trained on the server x to perform extraction processing on the individual gaze map g i to obtain an individual gaze embedding e i , and send the individual gaze embedding e i back to the client i;

[0009] Step S3: Based on the personalized estimation head H on the client i i process the individual gaze embedding e i to obtain a general gaze estimation.

[0010] According to the method of the first aspect of the present invention, in the step S1, the privacy protector P p i is an information bottleneck based on representation decoupling, and through the privacy protector P p i perform individual information stripping on the individual gaze data, which is specifically expressed as:

[0011] where x i is the original gaze estimation picture of the client i, and N is the number of clients.

[0012] According to the method of the first aspect of the present invention, in the step S2, the gaze knowledge extractor E x is trained based on the individual gaze maps sent by multiple clients, and the gaze knowledge extractor E x performs the extraction of the individual gaze embedding, which is specifically expressed as:

[0013] ei = E x (g i ), i = 1, 2, …, N (2).

[0014] According to the method of the first aspect of the present invention, in the step S3, the personalized gaze estimation head H i When performing general gaze estimation on the individual gaze embedding e i , error compensation shall also be combined with the personalized calibration model, and the final general gaze estimation result can be expressed as:

[0015]

[0016] Wherein, represents the calibration characterization extractor of client i, represents the personalized calibration module of client i, and the subscript lc represents local calibration, o i is the final general gaze estimation result of client i.

[0017] According to the method of the first aspect of the present invention, the method further includes: through the overall loss function of the client i, for the privacy protector P p i , the personalized estimation head H i , the personalized correction model and the calibration characterization extractor perform gradient calculation and model parameter update.

[0018] The second aspect of the present invention discloses a privacy-protected distributed gaze estimation system, which includes a plurality of clients i and a server in a distributed architecture. The client i is provided with a privacy protector P p i and a personalized estimation head H i , and the server is provided with a pre-trained gaze knowledge extractor E x ;

[0019] Wherein, the privacy protector P p i , is configured to process the individual gaze data obtained by the client i to remove privacy and gaze-irrelevant information, obtain the individual gaze map g i , and send the individual gaze map g i to the server;

[0020] The gaze knowledge extractor E x , is configured to perform extraction processing on the individual gaze map g i to obtain the individual gaze embedding e i , and send the individual gaze embedding e iSend it back to the client i;

[0021] The personalized estimation head H i , which is configured to process the individual gaze embedding e i to obtain a general gaze estimation.

[0022] For the system according to the second aspect of the present invention, a personalized calibration model is further provided on the client i The personalized calibration model is used to perform error compensation on the general gaze estimation of the personalized estimation head H i .

[0023] For the system according to the second aspect of the present invention, the server consists of a physically isolated split server and a federated server. The split server is responsible for receiving the individual gaze maps g from each client i i for training the gaze knowledge extractor E x , and the federated server is only responsible for globally aggregating and distributing the privacy protector P p i .

[0024] For the system according to the second aspect of the present invention, the gaze knowledge extractor E x is trained based on the individual gaze maps sent by multiple clients. The extraction of the individual gaze embedding by the gaze knowledge extractor E x is specifically expressed as:

[0025] e i = E x (g i ) (2).

[0026] For the system according to the second aspect of the present invention, when the personalized gaze estimation head H i performs a general gaze estimation on the individual gaze embedding E i , it also needs to perform error compensation in combination with the personalized calibration model. The final general gaze estimation result can be expressed as:

[0027]

[0028] wherein, represents the calibration characterization extractor of client i, represents the personalized calibration module of client i, and the subscript lc represents local calibration, and o i is the final general gaze estimation result of client i.

[0029] For the system according to the second aspect of the present invention, through the overall loss function of the client i for the privacy protector P pi , the personalized estimation head H i , the personalized correction model and the calibration feature extractor perform gradient calculation and model parameter update.

[0030] In the third aspect of the present invention, an electronic device is disclosed. The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in a privacy-protected distributed gaze estimation method according to any one of the first aspects of the present disclosure are implemented.

[0031] In the fourth aspect of the present invention, a computer-readable storage medium is disclosed. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in a privacy-protected distributed gaze estimation method according to any one of the first aspects of the present disclosure are implemented.

[0032] In summary, the solution proposed by the present invention incorporates privacy security into the gaze estimation task and proposes the first gaze estimation framework with privacy protection capabilities. It can jointly train the model among multiple parties and protect individual privacy security in the inference stage, and does not require building a centralized large-scale gaze estimation dataset, thus greatly reducing the risk of individual privacy leakage; a distributed gaze estimation framework is established, and the computational resource requirements for client terminal device deployment are greatly reduced by splitting the model, which can effectively accelerate the gaze estimation training and inference speed; at the same time, it takes into account the enhancement of the generalization performance of the general gaze estimation model in the cloud and the personalized adaptation to the individual physiological structure differences of the client. Compared with traditional gaze estimation, which can only focus on one side or take a compromise solution, it improves the generalization and personalization, has the capabilities of distributed optimization and personalized deployment, and has a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1 is the flow of a privacy-protected distributed gaze estimation method according to an embodiment of the present invention Figure 1 ;

[0035] Figure 2 is the flow of a privacy-protected distributed gaze estimation method according to an embodiment of the present invention Figure 2 ;

[0036] Figure 3 The flow of a privacy - protected distributed gaze estimation method according to an embodiment of the present invention Figure 3 ;

[0037] Figure 4 The technical - solution implementation framework diagram of a privacy - protected distributed gaze estimation method according to an embodiment of the present invention;

[0038] Figure 5 The structural diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] The first aspect of the present invention discloses a privacy - protected distributed gaze estimation method. Figure 1 The flowchart of a privacy - protected distributed gaze estimation method according to an embodiment of the present invention is shown as Figure 1 follows. The method includes:

[0041] Step S1: Process the individual gaze data obtained by client i through the privacy protector P p i to remove information irrelevant to privacy and gaze, and obtain the individual gaze map g i , and send the individual gaze map g i to the server;

[0042] Step S2: Use the gaze knowledge extractor E pre - trained on the server x to perform extraction processing on the individual gaze map G i to obtain the individual gaze embedding e i , and send the individual gaze embedding e i back to client i;

[0043] Step S3: Based on the personalized estimation head H on client i i process the individual gaze embedding e i to obtain a general gaze estimation.

[0044] In step S1, process the individual gaze data obtained by client i through the privacy protector P p i to remove information irrelevant to privacy and gaze, and obtain the individual gaze map gi , and send the individual line-of-sight map g i to the server.

[0045] In some embodiments, in step S1 of the present invention, the privacy protector P p i is an information bottleneck based on representation decoupling. Through the privacy protector P p i perform individual information stripping on the individual line-of-sight data, specifically expressed as:

[0046]

[0047] where x i is the original picture of the line-of-sight estimation of client i, and N is the number of clients.

[0048] In step S2, use the line-of-sight knowledge extractor E pre-trained on the server x to perform extraction processing on the individual line-of-sight map g i to obtain the individual line-of-sight embedding e i , and send the individual line-of-sight embedding e i back to client i.

[0049] In some embodiments, in step S2 of the present invention, the line-of-sight knowledge extractor E x is trained based on the individual line-of-sight maps sent by multiple clients. The extraction of the individual line-of-sight embedding by the line-of-sight knowledge extractor E x is specifically expressed as:

[0050] e i = E x (g i ) (2).

[0051] In step S3, based on the personalized estimation head H on client i i perform processing on the individual line-of-sight embedding e i to obtain a general line-of-sight estimation.

[0052] In some embodiments, as Figure 2 shown, in step S3 of the present invention, the personalized line-of-sight estimation head H i while performing line-of-sight estimation on the individual line-of-sight embedding e i , also combines a personalized calibration model for error compensation. The final general line-of-sight estimation result can be expressed as:

[0053]

[0054] where, represents the calibration representation extractor of client i, Denote the personalized calibration module of client i, where the subscript lc represents local calibration, o i is the final general line-of-sight estimation result for client i.

[0055] In some embodiments, the method of the present invention further includes: calculating the gradient and updating the model parameters for the overall loss function of client i with respect to the privacy protector P p i , the personalized estimation head H i , the said personalized correction model and the calibration feature extractor to perform gradient calculation and model parameter update.

[0056] The distributed line-of-sight personalized estimation framework with privacy protection constructed in the embodiments of the present invention splits the traditional end-to-end line-of-sight estimation model into three parts with different objectives, which are specifically as follows:

[0057] 1) The privacy protector strips the privacy and line-of-sight irrelevant information from the original data to generate a transmission representation that contains sufficient and only line-of-sight related information, namely the line-of-sight map, also known as the "gaze view". The line-of-sight map is sent to the server for interacting line-of-sight knowledge. In this process, in order to achieve privacy protection, the information bottleneck theory based on representation decoupling is introduced.

[0058] 2) The line-of-sight knowledge extractor is trained on the line-of-sight maps from all clients to achieve accurate and highly generalized line-of-sight estimation on a sufficiently diverse data distribution, and by removing client-related "noise" (i.e., privacy and gaze irrelevant information that may lead to performance degradation). The line-of-sight knowledge extractor extracts the generalized line-of-sight embedding representation from the "gaze view" and sends it back to the corresponding client.

[0059] 3) The personalized estimation head is located at the client side, which combines the extracted line-of-sight embedding with personalized calibration to achieve personalized adapted line-of-sight estimation. Among them, the calibration is extracted from the individual privacy information by an additional lightweight personalized calibration model P lc and these information are decoupled through the information bottleneck theory and protected locally. By adding an additional calibration model, it provides personalized adaptation for the differences in eyeball structures.

[0060] Therefore, the entire communication process of the privacy-protected distributed line-of-sight estimation method in the embodiments of the present invention can be summarized as the following steps: First, the privacy protector generates an individual line-of-sight map g by removing line-of-sight irrelevant information i ; then, the line-of-sight knowledge extractor E x extracts the generalized line-of-sight embedding e from the line-of-sight maps from all clients i, the embedding is sent back to client i, where the personalized estimation header H i combines it with the personalized calibration to calculate the final general line-of-sight estimation result, specifically as Figure 3 shown.

[0061] Specifically, the specific implementation technical solution of the privacy-protected distributed line-of-sight estimation method in this embodiment is as follows, as Figure 4 shown:[[]]END]]

[0062] In this embodiment, multiple individuals are allowed to be client users. On the premise of ensuring that the original line-of-sight estimation data they own is not leaked, joint training and personalized line-of-sight estimation are carried out. The privacy-protected distributed line-of-sight estimation framework will be described in detail below through formulas and explanations.

[0063] Suppose there are a total of N individuals participating in the multi-party joint training, and the line-of-sight data set owned by each individual is D i , and the data sets of all individuals can be expressed as where the data set of individual i has M i data samples, and each data sample consists of two parts: the original line-of-sight estimation picture x and the actual true value label y of the line-of-sight direction. Therefore, D i can be expressed as where represents the data distribution of individual i. Due to the existence of physiological structure differences between individuals, the data distributions between different individuals are also different, that is, it is expressed as where represents the true general line-of-sight sample distribution.

[0064] Suppose the total number of optimization rounds in the distributed optimization process is T. When the update round is t, first, for all individual clients {1...N} participating in the optimization of this round, through the information bottleneck based on representation decoupling perform individual privacy stripping.

[0065]

[0066] Then send these privacy-irrelevant line-of-sight maps g i to the server in the cloud to extract the line-of-sight embedding through the line-of-sight knowledge extractor E x .

[0067] e i = E x (g i ) (2)

[0068] Then the server in the cloud will extract the line-of-sight embedding e iSend it back to the corresponding client for gaze estimation. While the personalized gaze estimation head performs gaze estimation, it also compensates for errors through a lightweight personalized calibration model. The final gaze direction result can be expressed as:

[0069]

[0070] where represents the calibration feature extractor of client i, represents the personalized calibration model of client i, and the subscript lc represents local calibration, o i is the final general gaze estimation result of client i. The above represents the entire process of personalized gaze direction prediction for privacy protection. To achieve excellent generalization performance, personalized adaptation, and accurate gaze estimation, it is necessary to optimize the three parts of the model splitting and the lightweight personalized calibration model.

[0071] Before optimizing the model, loss calculation should be performed first. For client i, the overall loss consists of three parts: gaze prediction loss, decoupled information bottleneck loss, and calibration loss. The first part, the gaze prediction loss, represents the deviation between the predicted direction and the actual direction, and is calculated using the predicted gaze direction o i and the actual ground truth label y i .

[0072] l τ = l τ (o i , y i , σ)

[0073]

[0074] where τ and σ are hyperparameters that can be set and adjusted according to needs. Therefore, the gaze prediction loss of client i is fully expanded and expressed as follows:

[0075]

[0076] The second part, the decoupled information bottleneck loss, aims to decouple gaze-related information and gaze-unrelated privacy information. For this purpose, this embodiment uses mutual information constraints to perform the above feature decoupling. For client i, the decoupled information bottleneck loss can be expressed as follows:

[0077]

[0078] where I(*; *) represents mutual information calculation, g i is the gaze-related representation, p i is the gaze-unrelated representation,

[0079] In the third part, calibration loss. In addition to calibrating the predicted line-of-sight angle in the loss function of the first part, an additional regularization loss term is added to prevent the overall deviation of the correction module. Specifically, to ensure that the average prediction of all samples deviates from the customer by zero, the following regularization loss term is added to client i:

[0080]

[0081] In summary, the overall loss function of client i is expressed as:

[0082]

[0083] Among them, α and β are the weight balance parameters between the losses, which can be set and adjusted according to needs. After calculating the loss function of the client, use this loss to calculate the gradient and update the model parameters of the privacy protector, personalized estimation head, and lightweight personalized correction model on the client. Specifically, the gradient of the privacy protector of client i in the t-th round can be calculated as follows:

[0084]

[0085] Among them represents the parameters of the privacy protector of client i at the t-th round, and the corresponding update can be expressed as follows:

[0086]

[0087] Similarly, the gradients of the personalized estimation head and lightweight personalized calibration model of client i in the t-th round can be calculated as follows:

[0088]

[0089] Among them, represents the parameters of the personalized estimation head H of client i i at the t-th round, represents the lightweight personalized correction model of client i at the t-th round.

[0090] Next, the server in the cloud uses the prediction losses from all clients to update the gradient of the generalized line-of-sight knowledge extraction network in the cloud.

[0091]

[0092] Among them represents the parameters of the line-of-sight knowledge extraction network model E in the cloud server x at the t-th round.

[0093] After that, to enhance the privacy protection performance of the client privacy protector, model aggregation and distribution are performed on it through the server.

[0094]

[0095] After each round of training, the server calculates the global privacy protector H g , and sends it to each client i = 1...N to replace the local privacy protector.

[0096] In this embodiment, the server in the cloud consists of two physically isolated servers. One is a splitting server, which is responsible for receiving the "gaze maps" from each client for training the gaze embedding extractor. The other is a federated server, which is only responsible for global aggregation and distribution of the privacy protector model. Using the physically isolated dual-server architecture can effectively prevent attackers from stealing data. Even if one server is compromised, attackers cannot reconstruct individual privacy from it, thus effectively enhancing the privacy protection ability of this framework.

[0097] In summary, the gradient calculation and update of all modules can be completed through the above steps. Among them, the decoupled information bottleneck needs to be further explained. Because it is very difficult and time-consuming to directly calculate the mutual information in the actual application process, therefore, this embodiment adds a discriminator and a face generator to decouple the privacy representation and the gaze representation using adversarial training. It consists of three parts, which can be respectively understood as minimizing the mutual information between the privacy representation and the gaze representation min{I(g i ; p i )}, maximizing the mutual information between the gaze representation and the gaze label max{I(g i ; y i )}, and maximizing the mutual information between the privacy representation combined with the gaze label and the original gaze image max{I(x i ; p i , y i )}.

[0098] To achieve min{I(g i ; p i )}, adversarial training is used to fully decouple the privacy representation and the gaze representation.

[0099]

[0100] Among them, d represents the discriminator, and q represents the data distribution.

[0101] To achieve max{I(g i ; y i)}, a prediction angle deviation similar to the loss in the first part is introduced.

[0102]

[0103] To achieve max{I(x i ; p i , y i )}, a face reconstructor is introduced, which attempts to reconstruct the original gaze estimation data using only the privacy representation and the gaze direction label.

[0104]

[0105] Where Rand represents randomly selecting a gaze representation consistent with the gaze label to form a complete representation with the privacy representation. Re(·) represents the reconstruction model, which uses the complete representation to reconstruct the original gaze estimation image. l rec represents the reconstruction loss, which aims to evaluate the similarity between the image reconstructed by the reconstruction model Re(·) and the real input image. If the similarity is high, it means that the original data can be well reconstructed using only the privacy representation and the gaze direction label, indicating that the gaze representation only contains pure gaze-related representations. In the actual construction of the embodiments of the present invention, the above three are used to replace the representation decoupling information bottleneck loss constraint that completely uses mutual information.

[0106] The embodiments of the present invention propose to construct a distributed gaze estimation framework, which splits the gaze estimation model into three parts: a privacy protector, a gaze knowledge extractor, and a personalized gaze estimation head. Among them, the gaze knowledge extractor with a large model scale and high computational resource consumption is deployed on the cloud server. By fully learning the gaze maps that are irrelevant to individual privacy from each client, a general gaze knowledge extraction ability with good generalization ability can be obtained. In addition, in order to achieve personalized gaze estimation to overcome the estimation deviation caused by individual physiological structure differences, a lightweight personalized individual difference calibration model is deployed on the individual client, which is used to receive only the individual privacy information retained locally for training to minimize the influence of individual eyeball structure changes and head pose changes. The personalized gaze estimation head deployed on the client device receives the general gaze embedding extracted by the cloud gaze knowledge extractor and the personalized individual difference calibration module to provide accurate individual gaze estimation, so as to provide personalized adaptation estimation that overcomes individual physiological differences on the basis of a general gaze model with good generalization.

[0107] In summary, the privacy - protected distributed gaze estimation method of the embodiments of the present invention incorporates privacy security into the gaze estimation task for the first time and proposes the first gaze estimation framework with privacy protection capabilities. It can jointly train the model among multiple parties and protect individual privacy security during the inference phase, and does not require the construction of a centralized large - scale gaze estimation dataset. Therefore, it greatly reduces the risk of individual privacy leakage; it establishes a distributed gaze estimation framework for the first time, and greatly reduces the computational resource requirements for the deployment of client terminal devices by means of model splitting, which can effectively accelerate the training and inference speed of gaze estimation; at the same time, it takes into account the enhancement of the generalization performance of the general gaze estimation model in the cloud and the personalized adaptation to the individual physiological structure differences of the client. Compared with the traditional gaze estimation that can only favor one side or take a compromise solution, the method of the embodiments of the present invention improves both generalization and personalization at the same time, has the capabilities of distributed optimization and personalized deployment, and has a broader application prospect.

[0108] The second aspect of the present invention discloses a privacy - protected distributed gaze estimation system. The system includes:

[0109] The system 100 includes a plurality of clients i and a server in a distributed architecture. A privacy protector P and a personalized estimation head H are provided on the client i p i and a pre - trained gaze knowledge extractor E is provided on the server i ; x

[0110] Among them, the privacy protector P p i , is configured to process the individual gaze data obtained by the client i to remove information irrelevant to privacy and gaze, obtain an individual gaze map g i , and send the individual gaze map g i to the server;

[0111] The gaze knowledge extractor E x , is configured to perform extraction processing on the individual gaze map g i to obtain an individual gaze embedding e i , and send the individual gaze embedding e i back to the client i;

[0112] The personalized estimation head H i , is configured to process the individual gaze embedding e i to obtain a general gaze estimation.

[0113] According to the system of the second aspect of the present invention, a personalized calibration model is also provided on the client i The personalized calibration model is used to calibrate the personalized estimation head H iPerform error compensation for general gaze estimation.

[0114] According to the system of the second aspect of the present invention, the server consists of a physically isolated split server and a federated server. The split server is responsible for receiving the individual gaze maps gi from each client i i for training the gaze knowledge extractor E x , and the federated server is only responsible for globally aggregating and distributing the privacy protector P p i for global aggregation and distribution.

[0115] According to the system of the second aspect of the present invention, the gaze knowledge extractor E x is trained based on the individual gaze maps sent by multiple clients. The gaze knowledge extractor E x performs the extraction of individual gaze embeddings, which is specifically expressed as:

[0116] e i = E x (gi i ) (2).

[0117] According to the system of the second aspect of the present invention, the personalized gaze estimation head H i while performing general gaze estimation on the individual gaze embedding e i , also combines a personalized calibration model for error compensation. The final general gaze estimation result can be expressed as:

[0118]

[0119] wherein, represents the calibration feature extractor of client i, represents the personalized calibration module of client i, and the subscript lc represents local calibration, and o i is the final general gaze estimation result of client i.

[0120] According to the system of the second aspect of the present invention, the privacy protector P p i , the personalized estimation head H i , the said personalized correction model and the calibration feature extractor are subjected to gradient calculation and model parameter update through the overall loss function of client i.

[0121] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor. When the processor executes a computer program, the steps in a privacy-preserving distributed gaze estimation according to any one of the first aspects disclosed in the embodiments of the present invention are implemented.

[0122] Figure 5 The structural diagram of an electronic device according to an embodiment of the present invention is shown as Figure 5 follows. The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a carrier network, near field communication (NFC), or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0123] Those skilled in the art can understand that Figure 5 the structure shown in

[0124] is only the structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0125] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, all possible combinations of the technical features in the above embodiments are not described. However, as long as the combination of these technical features does not conflict, it should be considered as the scope described in this specification. The above embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

[0126] The above are the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A privacy-preserving distributed line-of-sight estimation method, characterized in that: The method comprises: Step S1: Through the privacy protector P p i The individual sight line data obtained by client i is processed to remove privacy and sight-irrelevant information, and the individual sight line graph g is obtained. i , and the individual sight map g i Send to server; Step S2: Using the sight line knowledge extractor E pre-trained on the server x For the individual sight map g i Extraction processing is performed to obtain the individual gaze embedding g i , and embed the individual line of sight into e i Send back to the client i; Step S3: Estimating the personalized header H based on the client i i Embed the individual's line of sight i Processing is performed to obtain a generalized line of sight estimate.

2. A privacy-preserving distributed line of sight estimation method according to claim 1, characterized in that: In step S1, the privacy protector P p i Based on the information bottleneck of representation decoupling, the privacy protector P p i Individual information is stripped from the individual sight line data, which is specifically expressed as follows: Among them, x i is the original image of line of sight estimation of client i; N is the number of clients.

3. The privacy-preserving distributed line-of-sight estimation method according to claim 2, characterized in that: In step S2, the sight line knowledge extractor E x The line of sight knowledge extractor E is trained based on individual line of sight graphs sent by multiple clients. x The extraction of individual line of sight embedding is specifically expressed as: yes i =E x (g i ),i=1,2,…,N (2).

4. The privacy-preserving distributed line-of-sight estimation method according to claim 3, characterized in that: In step S3, the personalized sight line estimation head H i In the individual line of sight embedding i While performing general line of sight estimation, it is also necessary to combine the personalized calibration model for error compensation. The final general line of sight estimation result can be expressed as: in, represents the calibrated representation extractor for client i, represents the personalized calibration module of client i, the subscript lc represents local calibration, and o i is the final general line of sight estimation result of client i.

5. The privacy-preserving distributed line-of-sight estimation method according to claim 4, characterized in that: The method further includes: using the overall loss function of the client i to calculate the privacy protector P p i , the personalized estimation head H i , the personalized correction model and the calibration characterization extractor Perform gradient calculation and model parameter update.

6. A distributed line of sight estimation system for protecting privacy, characterized in that: The system includes multiple clients i and servers in a distributed architecture, and a privacy protector P is provided on the client i. p i and the personalized estimation head H i , the server is provided with a pre-trained sight line knowledge extractor E x ; Wherein, the privacy protector P p i , is configured to process the individual sight line data obtained by client i to remove privacy and sight line irrelevant information, and obtain the individual sight line graph g i , and the individual sight map g i Sending to the server; Gaze Knowledge Extractor E x , is configured to, for the individual sight graph g i Extraction processing is performed to obtain the individual gaze embedding e i , and embed the individual line of sight into e i Send back to the client i; The personalized estimation head H i , is configured to embed the individual line of sight into e i Processing is performed to obtain a generalized line of sight estimate.

7. The privacy-preserving distributed line-of-sight estimation system according to claim 6, characterized in that: The client i is also provided with a personalized calibration model The personalized calibration model For the personalized estimation head H i Error compensation for general line of sight estimation.

8. The privacy-preserving distributed line-of-sight estimation system according to claim 7, characterized in that: The server is composed of physically isolated split servers and federated servers. The split servers are responsible for receiving individual sight graphs g from each of the clients i. i To train the sight knowledge extractor E x , the federal server is only responsible for the privacy protector P p i Perform global aggregation and distribution.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the privacy-preserving distributed line-of-sight estimation method described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the privacy-preserving distributed line-of-sight estimation method described in any one of claims 1 to 5 are implemented.

Images