Image processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the application provides an image processing method and device, electronic equipment and readable storage medium, the method comprises the following steps: obtaining the face image of a user; using a neural network model, based on the face image, obtaining the gaze focus position of the user. Based on the method provided in the embodiment of the application, the accuracy of the estimation of the gaze focus position of the user can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, with the development of science and technology, various electronic devices have become an indispensable part of people's lives. In many application scenarios, it is sometimes necessary to estimate the user's gaze focus, i.e., the point of attention of the user's gaze, when using electronic devices. For example, using gaze to select and launch an application (equivalent to using gaze as a mouse), or pushing advertisements based on gaze position, etc. All of these application scenarios require real-time and accurate estimation of the user's gaze position on the electronic device screen. However, the existing estimation schemes still need to improve the estimation accuracy in practical applications. Summary of the Invention

[0003] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium to improve the accuracy of estimating the location of key points in a user's gaze. The solutions provided by embodiments of this application are as follows:

[0004] In a first aspect, embodiments of this application provide an image processing method based on a neural network model, the method comprising:

[0005] Obtain the user's facial image;

[0006] Using a neural network model, the user's gaze focus position is determined based on facial images.

[0007] Secondly, embodiments of this application provide a method for training a neural network model, the method comprising:

[0008] Obtain the training sample set, which includes images of each sample;

[0009] The initial target neural network model is trained based on each sample image until the loss function converges, resulting in a well-trained target neural network model.

[0010] Thirdly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0011] The image acquisition module is used to acquire images of the user's face.

[0012] The gaze focus position determination module is used to determine the user's gaze focus position based on a facial image using a neural network model.

[0013] Fourthly, embodiments of this application provide a training apparatus for a neural network model, the apparatus comprising:

[0014] The sample acquisition module acquires the training sample set, which includes images of each sample.

[0015] The model training module trains the initial target neural network model based on each sample image until the loss function converges, thus obtaining the trained target neural network model.

[0016] Fifthly, embodiments of this application provide an electronic device, which includes a memory and a processor; wherein the memory stores a computer program; and the processor is used to invoke the computer program to execute the method provided in the first or second aspect of this application.

[0017] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods provided in the first or second aspect of this application.

[0018] The beneficial effects of the technical solution provided in this application will be described in detail later in conjunction with specific embodiments and accompanying drawings, and will not be described here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0020] Figure 1 A flowchart illustrating a training method for a neural network model provided in an embodiment of this application is shown.

[0021] Figure 2 A flowchart illustrating a training method in one example of this application is shown;

[0022] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0023] Figure 4 A schematic diagram of a screen calibrator in one example of this application is shown;

[0024] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting the invention.

[0026] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0027] In scenarios requiring gaze keypoint location estimation, a crucial performance characteristic of gaze estimation schemes is accuracy. The trained estimator must achieve high accuracy not only on the training set but also on the actual test set, demonstrating good generalization performance. Another important performance characteristic is stability. When the user is fixed looking at the same point or makes slight movements near a certain point, the algorithm's estimated gaze position must be accurate and free from significant fluctuations. However, existing algorithms exhibit poor generalization performance. Regarding algorithm stability, current techniques, mostly based on video or multi-frame methods, involve post-processing after prediction, resulting in latency issues. In practical applications, gaze estimation requires fast and accurate results.

[0028] To address the problems existing in the prior art, this application provides a method for training a neural network model. This method fundamentally solves the problem during training by using a single image as input and increasing the stability of gaze estimation without sacrificing real-time performance. Furthermore, this method can be combined with video-based methods. On the other hand, this application also provides an image processing method that can process the initial gaze keypoint positions predicted by the model, improving the accuracy and stability of gaze keypoint positions.

[0029] The following is a detailed description of the solution provided in this application.

[0030] This application provides a method for training a neural network model, such as... Figure 1 As shown, the method may include:

[0031] Step S101: Obtain the training sample set, which includes images of each sample;

[0032] Step S102: Train the initial target neural network model based on each sample image until the loss function converges, and obtain the trained target neural network model.

[0033] In an optional embodiment of this application, step S102 above, training the initial target neural network model based on each sample image, specifically includes:

[0034] Obtain the first neural network model;

[0035] The first neural network model is trained at least twice based on each sample image to obtain the first neural network model after each training.

[0036] The neural network model is trained in each training iteration to predict each sample image, thus obtaining the prediction results for each sample image corresponding to the neural network model trained in each iteration.

[0037] Based on the difference between the prediction results and the actual results of each sample image, the sample images in the training sample set are deleted to obtain the processed sample images.

[0038] In an optional embodiment of this application, when the first neural network model is trained at least twice based on sample images, the sample images for the current training are sample images after deleting a set number or a set proportion of sample images from the sample images used in the previous training, where the difference between the predicted result and the actual result of the sample images is small.

[0039] In other words, the solution provided in this application embodiment can first perform sample image preprocessing using the above optional method before training the model based on sample images, that is, noise reduction processing of the training sample set, filtering out some poor sample images, and then training the target neural network model based on the filtered better sample images, so as to improve the accuracy of the trained model.

[0040] The following example further illustrates the preprocessing scheme for sample images.

[0041] This example utilizes an iterative filtering strategy to remove noise from a portion of the data, improving data quality and thus obtaining a better training model. Let the entire dataset (i.e., the training sample set) contain N samples (x_i, i = 1, ..., N), and each sample has a known true value gt_i, i.e., the true result. The training iterations in this example are M (M ≥ 2). Let the number of samples deleted each time be Nd = N / M, meaning the number of samples used in the next training iteration is the current sample count minus Nd. The specific steps are as follows:

[0042] 1. Initialize the training sample set as a "dataset" (containing N samples). Initialize the neural network parameters. Choose a loss function. Initialize the learning rate (e.g., 0.01). The neural network structure (i.e., the first neural network model) can be a network structure from existing technologies, such as AlexNet. For the loss function used to train the neural network, we use the ordinal loss function, but other loss functions can also be chosen.

[0043] 2. Train the neural network using the training sample set to obtain the model and model parameters after the training is completed.

[0044] 3. For each sample x_i in the training sample set, use the neural network trained in step 2 to calculate the predicted value y_i, and calculate the error between the predicted value and the true value err_i = distance(x_i, y_i). The error metric distance can be Euclidean distance. Sort all errors in ascending order.

[0045] 4. Select the first Nd samples (i.e., the Nd samples with the smallest error) and remove them from the training sample set. That is, the current training sample set size becomes N-Nd*t (where t represents the t-th training iteration in step 2). Save the current neural network parameter model and adjust the learning rate (a general learning rate adjustment algorithm can be selected).

[0046] 5. If N - Nd*t is not zero, return to step 1 and continue execution, that is, repeat the above steps until N - Nd*t is zero, or M iterations of training have been performed.

[0047] 6. Using all the saved M neural network parameter models (i.e., the M-th and M-th neural networks corresponding to the parameters of the M models), calculate the predicted values ​​of the N samples in the entire dataset, and calculate the error between the predicted values ​​and their true values. Thus, each sample yields M error results. The N error results obtained from the same model are sorted in ascending order, resulting in M ​​sequences, each with N values, representing the error of a certain model for all samples. If a sample x ranks in the last r% (r% in this example) of all M sequences, then x is considered a noise sample and is deleted from the dataset. The final clean dataset is the training sample set after deleting N*r% of the N samples.

[0048] In an optional embodiment of this application, training an initial target neural network model based on each sample image includes:

[0049] Each sample image is input into the teacher network model to obtain the output results of each sample image;

[0050] Each output result is used as the true result of the corresponding sample image, and the target neural network model is trained based on each sample image;

[0051] Among them, the teacher network model is any teacher network model randomly selected from the teacher queue;

[0052] Each output result is used as the true result for the corresponding sample image. After each training of the target neural network model based on each sample image, the following steps are also included:

[0053] Add the target neural network model after each training session to the teacher queue;

[0054] In this model, the teacher queue is initially empty, and the actual result of each sample image during initialization is the actual result corresponding to the labeled label of the sample image.

[0055] In an optional embodiment of this application, training an initial target neural network model based on each sample image includes:

[0056] After each training iteration, a portion of the model parameters of the target neural network model are initialized. The other portion of the model parameters and the initialized portion of the model parameters are then used as the new model parameters for the next training iteration of the target neural network model.

[0057] In an optional embodiment of this application, initializing a portion of the model parameters of the target neural network model after each training iteration includes:

[0058] Determine the importance of each filter in the target neural network model;

[0059] Determine the target filters that require parameter initialization based on the importance of each filter;

[0060] The model parameters of each target filter are initialized.

[0061] In an optional embodiment of this application, the model parameters of each target filter are initialized, including:

[0062] The filter parameter matrix of the neural network layer containing the target filter is decomposed to obtain the orthogonal matrix of the filter parameter matrix;

[0063] For the neural network layer containing the target filter, the eigenvectors corresponding to each target filter in the orthogonal matrix of the neural network layer are determined based on the position of each target filter in the corresponding neural network layer.

[0064] Based on the feature vectors corresponding to each target filter in the same neural network layer, determine the L2 norm of the feature vectors of each target filter in the same neural network layer;

[0065] For each target filter, the initialized parameters of the target filter are determined based on the feature vector corresponding to the target filter and the L2 norm of the neural network layer to which the target filter belongs.

[0066] The examples described above in this application provide a training method that can effectively reduce overfitting. This method is based on the basic framework of knowledge distillation and can incorporate two modules (a cosine similarity-based pruning module and an alignment orthogonal initialization module) to optimize the training process, thereby improving the accuracy and stability of the model.

[0067] The following example will provide further information on the content involved in the above examples.

[0068] In this example, let the neural network model be `net`, and its parameters be `W`. The number of iterations is `K`, and the number of times the training data is traversed in each iteration is `L`. The pruning rate is `p%` (i.e., the proportion of filters whose network parameters need to be redefined out of the total number of filters in the model), and the maximum pruning rate per layer is `p_max%` (i.e., for a layer of the model, the proportion of filters whose network parameters need to be redefined out of the total number of filters in that layer does not exceed `p_max%`). The algorithm process of this training method can be expressed as:

[0069] Initialize the teacher queue to empty. Initialize the parameters of net.

[0070]

[0071]

[0072] The process concludes by outputting the last network model in the current teacher queue as the training result. The neural network architecture can use existing techniques, such as AlexNet. The loss function can also use existing techniques, such as ordinal loss.

[0073] The pruning algorithm and the re-initialization algorithm described above are explained below.

[0074] Let the filter parameters of each layer in the neural network model be WF, with shape (Nout, C, Kw, Kh), where Nout is the number of filters in that layer, C is the number of input channels, and Kw and Kh are the width and height of the filter in that layer. The shape of WF is then adjusted to Nout one-dimensional vectors Wfi. i ,i=1,…,Nout, each one-dimensional vector has a dimension of C*Kw*Kh, that is, a vector with 1 row and C*Kw*Kh columns, and Nout represents the number of filters in one layer.

[0075] The specific steps of the pruning algorithm may include:

[0076] 1. Calculate the normalized result according to formula (1). (The norm in the following formula can be the Euclidean norm in the specific implementation, or other normalization methods can be used.)

[0077]

[0078] 2. Calculate Simf (representing the score of all filters in all layers) according to formula (2).

[0079] Simf = {Simf} k The set of all layers is defined as {k = 1, ..., Layer_num}.

[0080]

[0081]

[0082] In the above formulas, Layer_num represents the number of network layers in the model, and Simf k This is represented by the Simf of each filter in the k-th layer, i.e., Simf k This also corresponds to a set, where each element is the Simf of each filter in that layer. For example, if the number of filters in the k-th layer is Nout, then Simf... k This corresponds to a Simf with Nout numbers. For the i-th filter in a layer, it can be based on the filter's... and the j-th filter of the layer to which this filter belongs. By calculating the dot product of the two By obtaining the correlation between the network parameters of the two, and based on this method, the correlation between the network parameters of the i-th filter and each filter in the layer to which the i-th filter belongs can be calculated, resulting in Nout correlations. Then, through... The Simf of the i-th filter can then be obtained.

[0083] It should be noted that the Simf value of each filter represents the importance of that filter; the larger the Simf value, the lower the importance.

[0084] 3. Sort the Simf values ​​of each filter in ascending order. The filters in the last p% are the pruned filters W'. However, the percentage of pruned filters in each layer should not exceed p_max%.

[0085] The specific steps for re-initializing the algorithm may include:

[0086] 1. Perform QR decomposition on each layer WF of W to obtain its orthogonal matrix Worth. Take the value at the corresponding position of W' to obtain a matrix Worth' of the same size as W' (i.e., Worth' is calculated independently for each layer);

[0087] 2. Calculate the parameter Wpra' of W' after aggregating it with the batch normalization (BN) parameters according to the formula (calculated independently for each filter).

[0088]

[0089] Wherein, BNscale and BNvar are both parameters of Batch Normalization, BNscale is the network coefficient of the BN layer, and BNvar is the variance of the network parameters of the BN layer.

[0090] Understandably, in practical applications, if the filter, i.e., the convolutional layer, is not followed by a BN layer, this step can be omitted.

[0091] 3. Calculate the norm of each row of Wpar'. k,i (i.e., the L2 norm of the i-th filter in the k-th layer), and record the maximum and minimum values ​​of all L2 norms obtained in each layer, denoted as max_norm and min_norm respectively.

[0092] 4. Obtain the reinitialized weight Wr according to the following formula. k,i (Calculated for each filter in each layer)

[0093]

[0094] Among them, scalar aligned It can be sampled from a uniform distribution of (min_norm, max_norm).

[0095] In an optional embodiment of this application, training the initial target neural network model based on each sample image may include:

[0096] Determine the prediction loss of the target neural network model for each sample image during each training session, correct each sample image based on the prediction loss, and then conduct the next training of the target neural network model based on the corrected sample images.

[0097] In an optional embodiment of this application, the prediction loss of the target neural network model for each sample image is determined at each training session, and the sample images are corrected based on the prediction loss. Specifically, this may include:

[0098] Determine the prediction loss of the target neural network model for each sample image during each training session;

[0099] Determine the perturbation of the prediction loss to each sample image, and correct each sample image based on the perturbation.

[0100] In an optional embodiment of this application, determining the perturbation of the prediction loss to each sample image and correcting each sample image based on the perturbation includes:

[0101] For each sample image, determine the gradient change of the prediction loss for each pixel in the sample image.

[0102] Based on the gradient change corresponding to each pixel, determine the perturbation of the prediction loss for each pixel;

[0103] The perturbation corresponding to each pixel is superimposed on the original pixel value of the corresponding pixel in the sample image to obtain the corrected sample.

[0104] In an optional embodiment of this application, before determining the prediction loss of the target neural network model for each sample image during each training session, the method includes:

[0105] For each sample image, the sample image is cropped to obtain the global image and local image of the sample image;

[0106] Determine the prediction loss of the target neural network model for each sample image during each training iteration, correct the prediction loss for each sample image based on the prediction loss, and perform the next training iteration of the target neural network model based on the corrected sample images, including:

[0107] The global and local images corresponding to each sample image are used as new sample images to determine the prediction loss of the target neural network model for each new sample image during each training session.

[0108] The prediction loss of each new sample image is used to correct the corresponding new sample image;

[0109] The next training of the target neural network model is based on the revised new sample images.

[0110] The optional embodiments described above in this application also provide a training method that can effectively increase the robustness of the model. It is understood that this method can be used in conjunction with the aforementioned overfitting reduction methods, or it can be used alone, or it can be used based on the results obtained from the aforementioned overfitting reduction methods, i.e., using the results obtained from the aforementioned overfitting reduction methods as the initial values ​​for this part.

[0111] The training method for enhancing the robustness of this model will be illustrated below with an example.

[0112] In this example, the target neural network model is used to predict the location of a user's gaze key points in a face image. The training sample set consists of face images. The flowchart of the training method is as follows. Figure 2 As shown, the specific process may include:

[0113] 1.1 Input a random image X from the dataset. Based on the facial landmark detection results, crop the image into a face image X. f (Global image), Left eye image X l (Partial image), Right eye image X r (Partial Image) Take three images and use bilinear interpolation to resize them to a preset fixed size, then output the result. Assume the preset fixed sizes for the three images are 64x64, 48x64, and 48x64 respectively.

[0114] 1.2 Determine if adversarial images (the modified images in this example) have been generated for this training step: If not, output the original three images; if yes, output the latest three adversarial images.

[0115] 1.3 Input the three images into the neural network model and calculate the network output P. x (The vector representing the positions of key points in the line of sight) = f(Xf, Xl, Xr). Then, the loss is calculated and output using the ranking loss function. The calculation formula is:

[0116] Loss i =-Y′ xi *Log(P xi )-(1-Y′ xi )*Log(1-P xi )

[0117]

[0118] Where i represents vector Px or Y′ x The i-th component, bin_num represents the total number of components, Y′ x This represents the correct output value (i.e., ground truth) corresponding to the input image, which is the representation vector of the actual gaze key point position.

[0119] 1.4 Calculate the gradient of the loss with respect to the three images respectively to obtain three sets of adversarial perturbations. Taking a face image as an example, the calculation formula is:

[0120]

[0121]

[0122] Here, p_bin is the index of the last component in the Px vector that is greater than a set value (e.g., 0.5). Adding the three sets of adversarial perturbations to their corresponding images yields three adversarial images. α is a hyperparameter representing the step size, which can be selected as 1.0. This represents the Laplacian operator, sgn() is the sign function, and k is a set value, such as 4. It's understood that the value of 2k+1 cannot exceed the total number of neurons in the model's output layer. (The formula is used to...) This allows you to calculate the gradient change of each pixel in the image.

[0123] 1.5 Determine if the adversarial steps have reached the preset value step: If not, return to step 1.2, use three adversarial images as input, and continue to this step for judgment; if yes, proceed to step 1.6. The preset value step can be configured according to actual needs, such as 3.

[0124] 1.6 Input three adversarial images into the neural network model and calculate the adversarial loss Loss_adv. The calculation method is the same as for Loss, except that the input images are replaced with X. f adv X l adv X r adv .

[0125] 1.7 Input the adversarial loss Loss_adv and the original loss Loss. Based on a preset percentage c, the two losses are weighted and summed: Loss_total = c*Loss + (1-c)*Loss_adv. This sum is used as the overall loss. The gradients of all parameters of the neural network model are then calculated and backpropagated. Optionally, the preset percentage c can be 80%.

[0126] 1.8 Determine if the training steps have reached the preset maximum step count s: If not, repeat steps 1.1 to 1.7 to perform this determination; if yes, output the parameters of the neural network model and end the training process. The maximum step count s can be set to 200,000 steps.

[0127] In one experiment, a neural network model for estimating user gaze (i.e., the location of key points in user gaze) was trained based on the training method provided in the embodiments of this application. The model trained based on the scheme of the embodiments of this application and the model trained using common training methods were tested on the GAZE_CN_DB and GAZE_STARE_DB databases. The experimental results are shown in the table below:

[0128]

[0129] In the figure, an error of 241 pixels indicates that the number of pixels deviating between the predicted coordinates (predicted location of key points in the line of sight) and the actual coordinates (actual location of key points in the line of sight) is 241. The standard deviation of 63.88 indicates that the standard deviation calculated based on the prediction deviation of each experimental sample is 63.88. As can be seen from the table, compared with existing conventional training methods, the model trained based on the scheme of this application effectively improves the stability of the model's prediction results in terms of performance.

[0130] This application provides an image processing method based on a neural network model, such as... Figure 3 As shown, the method can mainly include:

[0131] Step S110: Obtain the user's facial image;

[0132] Step S120: Using a neural network model, based on the facial image, obtain the user's gaze focus position.

[0133] The image processing method provided in this application embodiment can determine the user's gaze focus position in the image based on the user's facial image, that is, the position of the focus point that the user's eyes are paying attention to in the image.

[0134] In an optional embodiment of this application, a neural network model is used to obtain the user's gaze focus position based on a facial image, including:

[0135] Obtain the position adjustment parameters;

[0136] Using a neural network model, based on facial images, the predicted focus position of the user's gaze is obtained;

[0137] Based on the position adjustment parameters, the predicted line-of-sight focus position is adjusted to obtain the adjusted line-of-sight focus position.

[0138] As an alternative, after obtaining the predicted user's gaze focus position based on the neural network model, the predicted position can be adjusted based on the position adjustment parameters to obtain the user's gaze focus position, thereby improving the accuracy of the gaze focus position.

[0139] In an optional embodiment of this application, the position adjustment parameter may be obtained in the following manner:

[0140] Display the calibration object to the user and obtain the user's current facial image;

[0141] Using a neural network model, based on the current face image, the predicted gaze focus position of the user corresponding to the current face image is obtained;

[0142] Based on the predicted focal point of the user's gaze corresponding to the current facial image, and the position of the calibration object, determine the position adjustment parameters.

[0143] In this scheme, the user can be guided to focus on the calibration point by providing a calibration object, and the user's facial image at this time can be obtained. Based on the predicted gaze focus position of the facial image at this time and the position of the calibration object, the position adjustment parameters can be determined.

[0144] In practical applications, the number of calibration objects can be configured according to actual needs, and there can be one or more. The style of the calibration objects is not limited in this application embodiment; for example, they can be calibration points.

[0145] As an example, Figure 4 The figure shows a schematic diagram of a calibration object. In this example, the calibration object can be one of three specific calibration points shown in the figure. The steps for determining the position adjustment parameter F(x) based on these three calibration points can include:

[0146] 1. Each time a specific calibration point is displayed on the screen of an electronic device (using a mobile phone as an example in this case) to guide the user's gaze to that specific calibration point, n (n≥1) images are captured at that calibration point using the mobile phone's visible light camera. For each of the three specific calibration points, n images are captured respectively. The actual position of each calibration point on the screen is recorded, i.e., its coordinates, denoted as g1, g2, and g3. For each image of a calibration point, a neural network model can predict the user's predicted gaze focus position on the screen, obtaining the predicted gaze focus position, i.e., the predicted coordinates, for each calibration point, denoted as p1, p2, and p3. p1, p2, and p3 correspond to g1, g2, and g3 respectively. In practical applications, when n>1 (e.g., n can be 3), p1, p2, and p3 can be the average of the predicted coordinates of the n images of each of the three specific calibration points.

[0147] 2. After obtaining g1, g2, g3 and p1, p2, p3, the position adjustment parameters can be determined based on these g1, g2, g3 and p1, p2, p3.

[0148] This example provides an expression for an optional position adjustment parameter F(x), as follows:

[0149]

[0150] In the expression, x represents the predicted viewpoint position, i.e., coordinates, that needs to be adjusted when adjusting the position based on the function, and scr is the pre-configured maximum position.

[0151] It should be noted that in practical applications, when calculating the focal point position, the coordinates in different dimensions need to be adjusted separately for each focal point position that needs adjustment. For example, if the focal point position has two directions (such as horizontal X and vertical Y), the adjustment value needs to be obtained from the predicted focal point coordinates in the horizontal direction using the function mentioned above. Based on the adjustment value and the predicted focal point coordinates in the horizontal direction, the adjusted focal point coordinates in the horizontal direction are obtained. Similarly, the adjustment value needs to be obtained from the predicted focal point coordinates in the vertical direction using the function mentioned above. Based on the adjustment value and the predicted focal point coordinates in the vertical direction, the adjusted focal point coordinates in the vertical direction are obtained. Accordingly, the calculations for points p1, p2, p3 and g1, g2, g3 also require the coordinate values ​​in each direction to be used in the corresponding calculations in each direction.

[0152] Based on the solution provided in this application embodiment, when it is necessary to determine the user's gaze focus on the electronic device, the facial image of the user when using the electronic device is collected, the image is input into the neural network model, the predicted gaze focus position x is output, and the corresponding adjustment F(x) is obtained according to the adjustment function. The adjustment F(x) is added to the predicted gaze focus position to obtain the adjusted position F(x)+x. The adjusted position is used as the user's gaze focus position on the electronic device.

[0153] In an optional embodiment of this application, step S120 above, which uses a neural network model to obtain the user's gaze focus position based on a facial image, may include:

[0154] Using a neural network model, based on facial images, the predicted focus position of the user's gaze is obtained;

[0155] Determine the prediction loss for the predicted focal point location;

[0156] Based on the prediction loss, determine the confidence level of the predicted focal point location;

[0157] If the confidence level is greater than the set threshold, the predicted gaze focus position will be determined as the user's gaze focus position.

[0158] If the confidence level is not greater than the set threshold, the predicted gaze focus position is adjusted to obtain the adjusted gaze focus position, or the gaze focus position corresponding to the previous frame of the face image is determined as the user's gaze focus position.

[0159] In an optional embodiment of this application, determining the confidence level of the predicted viewpoint location based on the prediction loss includes:

[0160] Determine the prediction loss for at least two perturbations in the face image;

[0161] The facial image is corrected based on each perturbation to obtain at least two corrected images;

[0162] The predicted gaze focus position for each corrected image is obtained through a neural network model.

[0163] The confidence level is obtained based on the predicted viewpoint position corresponding to each corrected image.

[0164] In an optional embodiment of this application, the confidence level is obtained based on the predicted gaze focus position corresponding to each corrected image, including:

[0165] Based on the predicted viewpoint position corresponding to each corrected image, the standard deviation is determined, and the reciprocal of the standard deviation is used as the confidence level.

[0166] In an optional embodiment of this application, the prediction loss is determined for at least two perturbations to the face image, including at least one of the following:

[0167] Determine the prediction loss relative to the initial gaze focus position of the face image in at least two directions; based on the prediction loss in each direction, determine the perturbation of the face image by the initial gaze focus position in each direction.

[0168] Based on at least two perturbation coefficients, determine at least two perturbations for the initial gaze focus position on the face image.

[0169] In an optional embodiment of this application, a neural network model is used to obtain the user's predicted gaze focus position based on a facial image, including:

[0170] Cropping the face image yields both a global image and a local image of the face.

[0171] The global and local images are input into the neural network model to obtain the user's predicted gaze focus position;

[0172] Determine the prediction loss for at least two perturbations to the face image, including:

[0173] Determine the prediction loss for at least two perturbations in both the global and local images;

[0174] Each perturbation is corrected on a separate facial image to obtain at least two corrected images, including:

[0175] Based on at least two perturbations corresponding to each image in the global and local images, each image is corrected to obtain at least two corrected images for each image.

[0176] Using a neural network model, the predicted gaze focus position for each corrected image is obtained, including:

[0177] Each set of corrected images is input into the neural network model to obtain the predicted gaze focus position corresponding to each set of corrected images. Each set of corrected images includes a corrected image corresponding to each image after cropping the face image.

[0178] Based on the predicted viewpoint location corresponding to each corrected image, a confidence level is obtained, including:

[0179] The confidence level is obtained based on the predicted gaze focus position corresponding to each group of modified images.

[0180] In an optional embodiment of this application, determining the perturbation of the face image by the prediction loss includes:

[0181] Determine the gradient change of the prediction loss for each pixel in the face image;

[0182] Based on the gradient change corresponding to each pixel, determine the perturbation of the prediction loss for each pixel;

[0183] The facial image is corrected based on the perturbation to obtain the corrected image, including:

[0184] The perturbation corresponding to each pixel is superimposed on the original pixel value of the corresponding pixel in the face image to obtain the corrected image.

[0185] The following specific example illustrates the various optional solutions provided in this application. In this example, the input is image data captured by a visible light camera of an electronic device (such as a mobile phone), for example, a single-frame image containing a face. The main flow of the image processing method in this example is as follows:

[0186] 1. The input for this step is the face image to be processed, i.e., a single-frame image X. For a single-frame image X, the image can be cropped into a face image X based on the facial landmark detection results. f Left eye image X l Right eye image X r Three images can be resized to a preset fixed size using bilinear interpolation and then output. For example, a face image X f Left eye image X l And right eye image X r The corresponding preset fixed sizes can be 64px64p, 48px64p, and 48px64p, where p represents pixels.

[0187] Understandably, facial image X f This refers to the global image in this example, specifically the left-eye image X.l And right eye image X r This refers to a partial image in this example.

[0188] 2. Input the three images into the neural network model to obtain the output P. x In this example, the output is the feature vector corresponding to the initial gaze focus position of a single frame image X, P. x The dimension is bin_num, which represents the total number of vector components, corresponding to the number of neurons in the model's output layer, P. x The value of the i-th component can be represented as P. xi .

[0189] 3. Based on P xi Multiple perturbations are identified. In this example, the perturbation is illustrated using a face image Xf. The perturbation corresponding to the face image Xf is represented as follows: Where f represents the face image, l represents the direction. In this example, l has two values, such as 1 and 2. A value of 1 corresponds to a perturbation to the left, and a value of 2 corresponds to a perturbation to the right. g and j correspond to the perturbation coefficients, and their explanations will be described later.

[0190] For face image X f Based on P xi Multiple perturbations can be obtained This results in multiple adversarial images (i.e., corrected face images), which can be represented as: That is, each perturbation is associated with a face image X. f By overlaying (pixel values), the modified face image is obtained.

[0191] In this example, the specific calculation formula is as follows:

[0192] Loss_test_l i = -Log(1-P xi )

[0193] Loss_test_r i =-Log(P) xi )

[0194]

[0195]

[0196]

[0197]

[0198] in,

[0199] In the above expressions, Loss_test_l i Loss_test_r represents the prediction loss of the i-th component in the leftward direction. i `p_i` represents the prediction loss for the i-th component in the rightward direction. `Loss_test_l` represents the prediction loss for the entire viewpoint position in the leftward direction in this example. `Loss_test_r` represents the prediction loss for the entire viewpoint position in the rightward direction in this example. `k` is a set value, such as 4. It's understood that the value of `2k+1` cannot exceed the total number of neurons in the model's output layer. `p_bin` represents the index (neuron number) of the last component in the `Px` vector that is greater than a set value (e.g., 0.5). Indicates a leftward perturbation. This indicates a rightward perturbation. Let α represent the Laplace operator. j Let α represent the step size, and j take different values. j Corresponding to different step sizes, for example, there are two choices for the step size: α1 = 1 and α2 = 2, meaning that the value of j can be 1 or 2. In this case, This indicates a leftward perturbation corresponding to a step size of 1. This represents a leftward perturbation corresponding to a step size of 2; sgn() represents the sign function, and pos g Represents the probability; for different values ​​of g, pos g This represents different probabilities. In this example, there are three possible probabilities: pos1 = 1, pos2 = 0.8, and pos3 = 0.6. That is, the value of g can be 1, 2, or 3. This represents a leftward perturbation with a step size of 1 and a probability of 1. This represents a leftward perturbation corresponding to a step size of 1 and a probability of 0.8, and so on.

[0200] rdm(M,pos) k ) nm Represents the position in matrix M at position pos k The probability of randomly selecting elements. It can be understood that rdm(M, pos) is a random selection of elements. k ) nm This is just to illustrate the meaning of the rdm() function, in the case where random()≤pos g When the function result is the corresponding element value, random() > pos g The element value is 0. Specifically, for Indicates each pixel position at pos gThe probability is randomly selected to determine whether to generate a perturbation, resulting in a perturbation matrix where some pixel positions are perturbed and others are not. For example, at position (m, n), random() > pos. g If the value at position (m, n) in the resulting matrix is ​​0, then the perturbation at that position is 0.

[0201] 3. Calculate the confidence score (the reciprocal of the adversarial standard deviation std_adv).

[0202] The formula for calculating the adversarial standard deviation in this example is as follows:

[0203]

[0204]

[0205]

[0206] Where N represents the total number of adversarial images corresponding to the three images, i.e., j * g * l. This represents the predicted gaze focus position for each set of adversarial images (i.e., each set of modified images). For different values ​​of j, g, and l, This corresponds to the predicted gaze focus position for each set of adversarial images, where mean is the average and var is the variance. In application, each A set of adversarial images corresponds to a set of three images (face image, left eye image, and right eye image), each with one corresponding adversarial image. That is, every three adversarial images serve as a set of inputs to the neural network model. Each set of inputs corresponds to a corrected predicted gaze focus position. In this example, there are a total of N sets of inputs, resulting in N predictions. Based on these N The standard deviation can be calculated, and the reciprocal of the standard deviation gives the confidence level.

[0207] 4. Predict the gaze focus position P of a single frame image X based on the adversarial standard deviation. x Different processing methods are applied. In this step, the input is the predicted view focus position P of the image. x The confidence level is 1 / std_adv, and the output is the processed view focus position.

[0208] Specifically, if 1 / std_adv is greater than the threshold th1, it indicates that the confidence level of the prediction result for the image is high, and the predicted view focus position P is directly output. x If 1 / std_adv is not greater than the threshold th1, it indicates that the confidence level of the prediction result for the image is low. In this case, we perform temporal smoothing on the predicted coordinates, such as Kalman filtering. The threshold th1 can be configured according to actual needs, such as 1 / 63.88.

[0209] Of course, in practical applications, we can also directly process the data based on the standard deviation. Specifically, if std_adv is less than or equal to the threshold th2, it means that the adversarial standard deviation of the prediction result for the image is small, and we consider the confidence level to be high, so we directly output its predicted gaze focus position P. x If std_adv is greater than the threshold th2, it indicates that the confidence level of the predicted result for the image is low. In this case, we perform temporal smoothing on the predicted coordinates, such as Kalman filtering. The threshold th2 can be configured according to actual needs, such as a value of 63.88.

[0210] It is understood that the neural network model for image processing in the embodiments of this application can be trained based on the training method provided in any embodiment of this application. That is, the target neural network model to be trained can be a model for outputting the user's gaze focus position (or the vector representation of the focus position). When training the model, the prediction result of the model is the predicted gaze focus position of the sample image, and the real result is the user's real gaze focus position in the sample image, that is, the real coordinate point of the user on the screen position of the electronic device in the sample image.

[0211] The solution provided in this application can be applied to various electronic devices, such as mobile electronic devices like mobile phones and tablets that only have a single visible light camera. The mobile phone can estimate the user's gaze based on video / images. The solution in this application can effectively improve the performance of user interaction with the device (mobile phone) using their gaze. For example, it can acquire user data, such as images, through the mobile phone camera and then estimate the position on the mobile phone screen that the user is looking at based on the user data.

[0212] It should be noted that, in the embodiments of this application, the explanations of some parameters in the examples of the training method embodiments and the examples of the image processing method can be referred to each other.

[0213] In the embodiments of the training method of this application, a method for denoising the dataset and a training method to reduce overfitting are proposed, thereby improving the generalization performance of the trained model. The embodiments of this application also propose a training method based on adversarial training targeting the ranking loss function (i.e., the aforementioned training method to enhance model robustness), which makes the gaze estimation results of the trained model more stable and solves the jitter problem during the training phase.

[0214] In embodiments of the image processing method of this application, a testing method for obtaining stable prediction results is proposed. This method outputs the standard deviation of the adversarial samples of the test image and the prediction result, and processes the prediction result using this adversarial standard deviation. Embodiments of this application also propose a three-point calibration method for a specific individual; of course, single-point or multi-point calibration methods can also be used, allowing for rapid and efficient adjustment of the prediction result.

[0215] Based on the same principle, this application also provides an image processing apparatus, which includes an image acquisition module and a gaze focus position determination module. Wherein:

[0216] The image acquisition module is used to acquire images of the user's face.

[0217] The gaze focus position determination module is used to determine the user's gaze focus position based on a facial image using a neural network model.

[0218] Optionally, the line-of-sight focus position determination module is specifically used for:

[0219] Obtain the position adjustment parameters;

[0220] Using a neural network model, based on facial images, the predicted focus position of the user's gaze is obtained;

[0221] Based on the position adjustment parameters, the predicted line-of-sight focus position is adjusted to obtain the adjusted line-of-sight focus position.

[0222] Optionally, the position adjustment parameters are obtained in the following way:

[0223] Display the calibration object to the user and obtain the user's current facial image;

[0224] Using a neural network model, based on the current face image, the predicted gaze focus position of the user corresponding to the current face image is obtained;

[0225] Based on the predicted focal point of the user's gaze corresponding to the current facial image, and the position of the calibration object, determine the position adjustment parameters.

[0226] Optionally, the line-of-sight focus position determination module is specifically used for:

[0227] Using a neural network model, based on facial images, the predicted focus position of the user's gaze is obtained;

[0228] Determine the prediction loss for the predicted focal point location;

[0229] Based on the prediction loss, determine the confidence level of the predicted focal point location;

[0230] If the confidence level is greater than the set threshold, the predicted gaze focus position will be determined as the user's gaze focus position.

[0231] If the confidence level is not greater than the set threshold, the predicted gaze focus position is adjusted to obtain the adjusted gaze focus position, or the gaze focus position corresponding to the previous frame of the face image is determined as the user's gaze focus position.

[0232] Optionally, when determining the confidence level of the predicted gaze focus position based on the prediction loss, the gaze focus position determination module is specifically used for:

[0233] Determine the prediction loss for at least two perturbations in the face image;

[0234] Each perturbation is corrected on a separate facial image to obtain at least two corrected images;

[0235] The predicted gaze focus position for each corrected image is obtained through a neural network model.

[0236] The confidence level is obtained based on the predicted viewpoint position corresponding to each corrected image.

[0237] Optionally, when the gaze focus position determination module obtains the confidence level based on the predicted gaze focus position corresponding to each corrected image, it is specifically used for:

[0238] Determine the standard deviation based on the predicted viewpoint position corresponding to each corrected image;

[0239] The reciprocal of the standard deviation is used as the confidence level.

[0240] Optionally, when determining the gaze focus position module, it performs at least one of the following schemes to determine the prediction loss for at least two perturbations in the face image:

[0241] Determine the prediction loss relative to the initial gaze focus position of the face image in at least two directions; based on the prediction loss in each direction, determine the perturbation of the face image by the initial gaze focus position in each direction.

[0242] Based on at least two perturbation coefficients, determine at least two perturbations for the initial gaze focus position on the face image.

[0243] Optionally, when the gaze focus position determination module uses a neural network model to obtain the user's predicted gaze focus position based on a facial image, it is specifically used for:

[0244] Cropping the face image yields both a global image and a local image of the face.

[0245] The global and local images are input into the neural network model to obtain the user's predicted gaze focus position;

[0246] The gaze focus location determination module, when determining the prediction loss for at least two perturbations in a face image, is specifically used for:

[0247] Determine the prediction loss for at least two perturbations in both the global and local images;

[0248] Each perturbation is corrected on a separate facial image to obtain at least two corrected images, including:

[0249] Based on at least two perturbations corresponding to each image in the global and local images, each image is corrected to obtain at least two corrected images for each image.

[0250] The gaze focus position determination module, when obtaining the predicted gaze focus position for each corrected image through a neural network model, is specifically used for:

[0251] Each set of corrected images is input into the neural network model to obtain the predicted gaze focus position corresponding to each set of corrected images. Each set of corrected images includes a corrected image corresponding to each image after cropping the face image.

[0252] The gaze focus position determination module, when obtaining the confidence level based on the predicted gaze focus position corresponding to each corrected image, is specifically used for:

[0253] The confidence level is obtained based on the predicted gaze focus position corresponding to each group of modified images.

[0254] Optionally, the gaze focus location determination module, when determining the perturbation of the prediction loss on the face image, is specifically used for:

[0255] Determine the gradient change of the prediction loss for each pixel in the face image;

[0256] Based on the gradient change corresponding to each pixel, determine the perturbation of the prediction loss for each pixel;

[0257] The facial image is corrected based on the perturbation to obtain the corrected image, including:

[0258] The perturbation corresponding to each pixel is superimposed on the original pixel value of the corresponding pixel in the face image to obtain the corrected image.

[0259] Based on the same principle, this application also provides a training device for a neural network model, which includes a sample acquisition module and a model training module. Wherein:

[0260] The sample acquisition module is used to acquire the training sample set, which includes images of each sample.

[0261] The model training module is used to train the initial target neural network model based on each sample image until the loss function converges, thus obtaining the trained target neural network model.

[0262] Optionally, the model training module is specifically used for:

[0263] Obtain the first neural network model;

[0264] The first neural network model is trained at least twice based on each sample image to obtain the first neural network model after each training.

[0265] The neural network model is trained in each training iteration to predict each sample image, thus obtaining the prediction results for each sample image corresponding to the neural network model trained in each iteration.

[0266] Based on the difference between the prediction results and the actual results of each sample image, the sample images in the training sample set are deleted to obtain the processed sample images.

[0267] Optionally, when the model training module trains the first neural network model at least twice based on sample images, the sample images for the current training are sample images after deleting a set number or a set proportion of sample images from the sample images used in the previous training, where the difference between the predicted results and the actual results of the sample images is small.

[0268] Optionally, the model training module is specifically used for:

[0269] Each sample image is input into the teacher network model to obtain the output results of each sample image;

[0270] Each output result is used as the true result of the corresponding sample image, and the target neural network model is trained based on each sample image.

[0271] Optionally, the teacher network model can be any teacher network model randomly selected from the teacher queue;

[0272] Optionally, after the model training module uses each output result as the true result for each corresponding sample image, and after training the target neural network model once based on each sample image, it is also used for:

[0273] Add the target neural network model after each training session to the teacher queue;

[0274] In this model, the teacher queue is initially empty, and the actual result of each sample image during initialization is the actual result corresponding to the labeled label of the sample image.

[0275] Optionally, when training the initial target neural network model based on each sample image, the model training module is specifically used for:

[0276] After each training iteration, a portion of the model parameters of the target neural network model are initialized. The other portion of the model parameters and the initialized portion of the model parameters are then used as the new model parameters for the next training iteration of the target neural network model.

[0277] Optionally, when initializing a subset of model parameters of the target neural network model after each training iteration, the model training module is specifically used for:

[0278] Determine the importance of each filter in the target neural network model;

[0279] Determine the target filters that require parameter initialization based on the importance of each filter;

[0280] The model parameters of each target filter are initialized.

[0281] Optionally, when initializing the model parameters for each target filter, the model training module is specifically used for:

[0282] The filter parameter matrix of the neural network layer containing the target filter is decomposed to obtain the orthogonal matrix of the filter parameter matrix;

[0283] For the neural network layer containing the target filter, the eigenvectors corresponding to each target filter in the orthogonal matrix of the neural network layer are determined based on the position of each target filter in the corresponding neural network layer.

[0284] Based on the feature vectors corresponding to each target filter in the same neural network layer, determine the L2 norm of the feature vectors of each target filter in the same neural network layer;

[0285] For each target filter, the initialized parameters of the target filter are determined based on the feature vector corresponding to the target filter and the L2 norm of the neural network layer to which the target filter belongs.

[0286] It is understood that the modules provided in the embodiments of this application can have the function of implementing the corresponding steps in the methods provided in the embodiments of this application. This function can be implemented by hardware or by hardware executing corresponding software. The above modules can be software and / or hardware; each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module of the voice translation device, please refer to the corresponding descriptions in the methods of the above embodiments, which will not be repeated here.

[0287] Furthermore, in practical applications, the functional modules of the apparatus in this application embodiment can be run in terminal devices and / or servers according to actual application requirements.

[0288] Based on the same principle, this application also provides an electronic device, which includes a memory and a processor; the memory stores a computer program; the processor is used to call the computer program to execute the method provided in any embodiment of this application.

[0289] Based on the same principle, this application also provides a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the method provided in any embodiment of this application.

[0290] As an alternative, Figure 5 The diagram shows a structural schematic of an electronic device to which an embodiment of this application applies, such as... Figure 5 As shown, the electronic device 4000 may include a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0291] Processor 4001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0292] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI bus or an EISA bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0293] The memory 4003 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0294] The memory 4003 stores a computer program for executing the scheme of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer program stored in the memory 4003 to implement the content shown in any of the foregoing method embodiments.

[0295] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0296] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Using a neural network model, the output gaze position is obtained from the input image of the user's face; Based on the output gaze position, the user's gaze position is obtained through at least one of the following: Determine the user's position adjustment parameters, and adjust the output line-of-sight position using the position adjustment parameters to obtain the user's line-of-sight position; Determine the prediction loss of the output gaze position, determine the confidence of the output gaze position based on the prediction loss, and predict the user's gaze position based on the confidence of the output gaze position. The position adjustment parameters are obtained in the following way: Display the calibration object; acquire the user's adjusted facial image, the adjusted facial image corresponding to the calibration object; Using a neural network model, the gaze position corresponding to the adjusted facial image is obtained based on the adjusted facial image; The position adjustment parameters are determined based on the gaze position corresponding to the adjusted facial image and the position of the calibration object.

2. The method according to claim 1, characterized in that, Predicting the user's gaze position based on the confidence level of the output gaze position includes: If the confidence level is greater than a set threshold, then the output gaze position is determined as the user's gaze position; or, If the confidence level is not greater than a set threshold, the output gaze position is adjusted to obtain the user's gaze position, or the gaze position corresponding to the previous frame of the face image is determined as the user's gaze position.

3. The method according to claim 1, characterized in that: The step of determining the confidence level of the output gaze position based on the prediction loss includes: Determine the prediction loss for at least two sets of perturbations on the face image; The facial images are corrected based on at least two sets of perturbations to obtain at least two corrected facial images; By inputting the at least two corrected facial images into a neural network model, the gaze position corresponding to the output of the at least two corrected images is obtained; The confidence level is obtained based on the gaze position of the output corresponding to the at least two corrected images.

4. The method according to claim 3, characterized in that, The step of obtaining the confidence level based on the output gaze positions corresponding to the at least two corrected images includes: The standard deviation is determined based on the gaze positions of the output corresponding to the at least two corrected images; The reciprocal of the standard deviation is used as the confidence level.

5. The method according to claim 3, characterized in that, The determination of the prediction loss for at least two perturbations to the face image includes at least one of the following: Determine the prediction loss for the output gaze position in each of at least two directions; based on the prediction loss in each of the at least two directions, determine at least two perturbations for the face image in the at least two directions; The at least two perturbations are determined based on at least two perturbation coefficients.

6. The method according to claim 1, characterized in that, Obtain the output gaze position, including: The face image is cropped to obtain a global face image and a local face image. The global face image and the local face image are respectively input into the neural network model to obtain the output gaze position and local gaze position; The prediction loss for determining the output gaze position, and the confidence level of the output gaze position based on the prediction loss, include: Determine the prediction loss of the output gaze position and the prediction loss of the local gaze position; The prediction loss for the output gaze position is determined for at least two perturbations of the global face image, and the prediction loss for the local gaze position is determined for at least two perturbations of the local face image. Based on at least two perturbations corresponding to each image in the global face image and the local face image, the corresponding images are corrected respectively to obtain at least four corrected images; The at least four corrected images are input into a neural network model to obtain the gaze positions of the output corresponding to the at least four corrected images; The confidence level is obtained based on the gaze positions of the outputs corresponding to the at least four corrected images.

7. The method according to claim 3 or 6, characterized in that, Determine the perturbation of the face image to the prediction loss, including: Determine the gradient change of the prediction loss for each pixel in the face image; Based on the gradient change corresponding to each pixel, determine the perturbation of the prediction loss for each pixel; The facial image is corrected based on the perturbation to obtain the corrected image, including: The perturbation corresponding to each pixel is superimposed on the original pixel value of the corresponding pixel in the face image to obtain the corrected image.

8. The method according to any one of claims 1 to 6, characterized in that, The neural network model is trained in the following ways: Obtain the training sample set, which includes images of each sample; The initial neural network model is trained based on the sample images until the loss function converges, resulting in a trained neural network model.

9. The method according to claim 8, characterized in that, The training of the initial neural network model based on the sample images includes: Determine the prediction loss of the neural network model for each sample image during each training session, correct each sample image based on the prediction loss, and then conduct the next training of the neural network model based on the corrected sample images.

10. The method according to claim 8, characterized in that, The training of the initial neural network model based on the sample images includes: Obtain the first neural network model; The first neural network model is trained at least twice based on each sample image to obtain the first neural network model after each training. The neural network model is used to predict each training sample after each training iteration, and the prediction results of each sample image corresponding to the neural network model after each training iteration are obtained. Based on the difference between the prediction results and the actual results of each sample image, the sample images in the training sample set are deleted to obtain the processed sample images.

11. The method according to claim 10, characterized in that, When the first neural network model is trained at least twice based on the sample images, the sample images for the current training are sample images after deleting a set number or a set proportion of sample images from the sample images used in the previous training, where the difference between the predicted results and the actual results of the sample images is small.

12. The method according to claim 8, characterized in that, The training of the initial neural network model based on the sample images includes: Each sample image is input into the teacher network model to obtain the output results of each sample image; Each output result is used as the true result of the corresponding sample image, and the neural network model is trained based on each sample image; The teacher network model is any teacher network model randomly selected from the teacher queue; Each output result is used as the true result for the corresponding sample image. After each training of the neural network model based on each sample image, the following steps are also included: The neural network model after each training iteration is added to the teacher queue; The teacher queue is initialized with an empty model, and the actual result of each sample image during initialization is the actual result corresponding to the labeled label of the sample image.

13. The method according to claim 8, characterized in that, The training of the initial neural network model based on the sample images includes: After each training iteration, a portion of the model parameters of the neural network model are initialized. The other portion of the model parameters and the initialized portion of model parameters are then used as the new model parameters for the next training iteration of the neural network model.

14. The method according to claim 13, characterized in that, The initialization of a subset of model parameters after each training iteration of the neural network model includes: Determine the importance of each filter in the neural network model; Determine the target filters that require parameter initialization based on the importance of each filter; Initialize the model parameters for each target filter; The initialization of the model parameters for each target filter includes: The filter parameter matrix of the neural network layer containing the target filter is decomposed to obtain the orthogonal matrix of the filter parameter matrix; For the neural network layer containing the target filter, the eigenvectors corresponding to each target filter in the orthogonal matrix of the neural network layer are determined based on the position of each target filter in the corresponding neural network layer. Based on the feature vectors corresponding to each target filter in the same neural network layer, determine the L2 norm of the feature vectors of each target filter in the same neural network layer; For each target filter, the initialized parameters of the target filter are determined based on the feature vector corresponding to the target filter and the L2 norm of the neural network layer to which the target filter belongs.

15. The method according to claim 9, characterized in that, The process of determining the prediction loss of the neural network model for each sample image during each training iteration, and correcting each sample image based on the prediction loss, includes: Determine the prediction loss of the neural network model for each sample image during each training session; For each sample image, determine the gradient change of the prediction loss for each pixel in the sample image. Based on the gradient change corresponding to each pixel, determine the perturbation of the prediction loss for each pixel; The perturbation corresponding to each pixel is superimposed with the original pixel value of the corresponding pixel in the sample image to obtain the corrected sample image.

16. An electronic device, characterized in that, Including memory and processor; The memory stores computer programs; The processor is configured to invoke the computer program to execute the method according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 15.

Images