An end-to-end pupil center localization method based on deep voting mechanism

Abstract

The present invention relates to an end-to-end pupil center positioning method based on a deep voting mechanism, and belongs to the field of computer vision applications. The method comprises: S1: obtaining an image set containing a human face, annotating the image set, and then dividing the annotated image set into a training set and a test set; S2: constructing a pupil center positioning model, and inputting the training set into the model for training; S3: inputting the image to be positioned in the test set into the trained pupil center positioning model to obtain the final pupil center positioning coordinates. The present invention realizes end-to-end data stream processing while maintaining high-precision positioning by combining a convolutional neural network with a voting mechanism in classical vision.

Description

Technical Field

[0001] The invention belongs to the field of computer vision applications and relates to an end-to-end pupil center positioning method based on a deep voting mechanism. Background Art

[0002] Pupil center localization is one of the key technologies for various computer vision applications and has been widely used in virtual reality, robotics, brain science, and medical analysis.

[0003] At present, the research on pupil center positioning is mainly divided into head-mounted pupil center positioning and natural pupil center positioning according to its application scenarios.

[0004] Head-mounted pupil center positioning refers to the use of head-mounted camera equipment to collect images of the eye area and then locate the pupil center. The advantage of this method is that the captured images are almost completely distributed around the eyes and have a high resolution. However, due to its requirement for specific head-mounted devices and the lack of universality of the captured images themselves in daily life, the application of this method in daily life is relatively limited. Natural pupil center positioning refers to the use of more general shooting equipment, such as smartphones and webcams, to obtain images under natural conditions and then locate the pupil center. Compared with the previous positioning problem, there are more uncertain factors in natural images that affect the results of pupil center positioning, such as shooting angle and changing lighting.

[0005] In order to overcome the influence of changing factors, many researchers have proposed innovative methods to achieve pupil center positioning in natural images, such as

[0006] Timm, Barth et al. defined the pupil center as the point where image gradients intersect most densely in Accurate Eye Centre Localisation by Means of Gradients. Based on the physiological pigment distribution of the eyeball, the gradient information of the image was used to locate the pupil center.

[0007] Xia and Yu et al. defined the pupil center localization problem as a special semantic segmentation problem in Accurate and Robust Eye Center Localization via Fully Convolutional Networks, and performed weighted prediction of the pupil center for the left and right eye regions respectively for positioning.

[0008] Baek and Choi et al. modeled the changing relationship between eyeball posture and pupil center in the eyeball model-based iris center localization for visible image-based eye-gaze tracking systems, established a priori knowledge base, and then performed the best match based on the collected eyeball posture to finally locate the pupil center.

[0009] The generative adversarial network (GAN) is introduced into the evolutionary neural networks to specifically remove the influence of glasses on the eye area, and then the segmentation method is used to locate the center of the pupil.

[0010] However, most existing methods, including the above methods, divide the entire process into three parts, namely, first detecting the face area, then cutting out the eye area, and finally locating the pupil center in the eye area. This design causes the data flow to become discontinuous, making the final pupil center positioning accuracy dependent on the quality of its front-end components, and also limits the design of the positioning method to the local image, which is not conducive to improving the accuracy and robustness of the positioning method through the overall image information. Summary of the invention

[0011] In view of this, the purpose of the present invention is to provide an end-to-end pupil center positioning method based on a deep voting mechanism, which combines convolutional neural networks with the voting mechanism in classical vision to achieve end-to-end data stream processing while maintaining high-precision positioning.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] An end-to-end pupil center localization method based on a deep voting mechanism uses a neural network to extract semantically informative descriptors from the eye region in an image. Different descriptors give independent predictions, and all predictions are finally integrated to obtain the final localization result. The method specifically includes the following steps:

[0014] S1: Obtain an image set containing human faces, annotate the image set, and then divide the annotated image set into a training set and a test set;

[0015] S2: Build a pupil center positioning model and input the training set into the model for training;

[0016] S3: Input the image to be located in the test set into the trained pupil center location model to obtain the final pupil center location coordinates.

[0017] Furthermore, in step S1, a set of images containing complete human faces in a natural environment is acquired through a common shooting device such as a mobile phone, a camera, etc., wherein the pupil area must be able to be identified by a human annotator.

[0018] Furthermore, in step S1, the annotator uses an annotation tool to annotate the image set, specifically including: for each image I, annotating two groups of six-dimensional vectors, the i-th group of six-dimensional vectors is recorded as in Used to describe the size of the eye area annotation box. represents the center coordinates of the eye area, (w i ,h i ) represents the width and height of the eye area; Information describing the center of the pupil. represents the coordinates of the pupil center, c i Indicates the category to which the sample falling within the eye region annotation box belongs.

[0019] Further, in step S2, a pupil center positioning model is constructed and trained, specifically including the following steps:

[0020] S21: Construct pupil center localization model: Use convolutional neural network as the backbone network for feature extraction, and use a specially designed network structure to construct the pupil center localization algorithm head, including a descriptor classifier and an offset regressor;

[0021] S22: input the training image I into the backbone network, extract the feature set F, input the feature set F into the pupil center positioning algorithm head, and obtain the predicted descriptor score set score and the offset vector set offset through the descriptor classifier and the offset regressor;

[0022] S23: Input the descriptor score set score and the offset vector set offset obtained in step S22 into the loss calculation module to obtain a loss value;

[0023] S24: according to the loss value calculated in step S23, the gradient descent optimization algorithm is used to transmit it back, and the pupil center positioning model is trained, and iterated until convergence;

[0024] S25: Input the sample set into the voting module and the pupil center suppression module in a clockwise manner to obtain the final pupil center positioning result.

[0025] Further, in step S23, the descriptor score set score and the offset vector set offset are input into the loss calculation module to obtain the loss value, which specifically includes: mapping each subset in the feature set F back to the original image according to the following formula according to the corresponding convolution step s,

[0026]

[0027]

[0028] Among them, (x, y) is the coordinate of the feature point in the feature map, (x0, y0) is the sample coordinate mapped back to the original image, if the sample falls within the range of the annotation box B, then this sample is recorded as a positive sample, that is, the local descriptor used to vote for the pupil center; the descriptor score set score and the offset vector set offset are input into the loss calculation module, and the classification loss L is calculated respectively. reg (o,o * ) and regression loss L cls (p,c * ), the calculation formula is as follows:

[0029]

[0030] L cls (p,c * )=-[α(1-p) γ logp+(1-α)p γ log(1-p)]

[0031] Among them, p represents the prediction score corresponding to a sample, c * Indicates the category corresponding to this sample; γ and α are hyperparameters used to adjust the weights of difficult and easy samples and positive and negative samples; o represents the offset vector offset; o * represents the true value of the offset vector, o * The calculation formula is as follows:

[0032]

[0033]

[0034] in, and Respectively represent the distance between the pupil center and the sample (x0, y0) in two dimensions, (x eye ,y eye ) represents the true coordinate of the pupil center;

[0035] The total loss calculation formula of the loss calculation module is as follows:

[0036]

[0037] Among them, L({p,c *},{o,o *}) represents the total loss value of the loss calculation module; λ is a hyperparameter used to balance classification and regression losses; N pos is the total number of descriptors, is an indicator function, which is 1 if and only if the element with coordinates (i, j) on the feature map corresponds to the sample bit descriptor, and 0 otherwise.

[0038] Further, in step S25, the sample set is input into the voting module and the pupil center suppression module to obtain the final pupil center positioning result, which specifically includes the following steps:

[0039] S251: Input the image I to be detected into the model trained in step S24, and obtain the feature set F, the descriptor score set score and the offset vector set offset through the feature extraction module and the pupil center positioning algorithm head, and map the feature set F back to the original image according to their respective step sizes to obtain the sample set sampleSet;

[0040] S252: The sample set sampleSet obtained in step S251 is filtered according to the descriptor score, and the samples with scores greater than the threshold T are selected as descriptors to obtain the descriptor set descriptorSet; the accumulation matrix accMatrix with the same size as the original image is initialized, and for each descriptor, its coordinates are vector-added with the corresponding predicted offset vector to obtain a set of predicted pupil center coordinates (e x ,e y ), add the descriptor score to the bin at the corresponding position of the accumulation matrix dp , the calculation formula is as follows:

[0041]

[0042] Among them, dp represents each specific descriptor, dp x and dp y represents the coordinates of the descriptor, and Indicates the offset predicted by the descriptor;

[0043] S253: sort the accumulated matrix accMatrix obtained in step S252 in descending order according to the value in each bin, and record the sorting result as sortedBin. Take the coordinate of the first bin in sortedBin as EC0, and calculate the Euclidean distance between the remaining bins and EC0. nms The bins are removed from sortedBin to form the suppression set nmsSet. For each bin in nmsSet, its coordinates are weighted to EC0 to obtain the corrected EC0. The calculation formula is as follows:

[0044] EC0+=accMatrix position ·position

[0045] sortedBin=sortedBin\bin

[0046] Among them, position is the coordinate of the corresponding bin in the original accumulation matrix, accMatrix position Represents the accumulated value at the corresponding position on the accumulation matrix accMatrix;

[0047] In the remaining bins of sortedBin, take the coordinates of the first bin and mark it as EC1. Then perform the above operation to obtain the corrected EC1. EC0 and EC1 are returned as the final located pupil center.

[0048] The beneficial effects of the present invention are:

[0049] (1) The present invention designs pupil center positioning into an end-to-end mode, simplifies the overall processing flow, and provides greater flexibility for subsequent research;

[0050] (2) With the help of the deep voting mechanism, the present invention greatly improves the accuracy and robustness of pupil positioning, providing a new theoretical basis for subsequent research.

[0051] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 This is a schematic diagram of a facial image captured in this embodiment;

[0054] Figure 2 For Figure 1 Schematic diagram of annotated faces;

[0055] Figure 3 It is a training flow chart of the end-to-end pupil center positioning method of the present invention;

[0056] Figure 4 It is a flow chart of the end-to-end pupil center positioning method of the present invention in an application scenario;

[0057] Figure 5 This is a structural framework diagram of a positioning system constructed using the method of the present invention;

[0058] Figure 6The following is a demonstration of the effects of a portion of input images obtained using the method of the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] See also Figure 1 to Figure 6 , Figure 1 This is a schematic diagram of a facial image taken in this embodiment. The present invention defines an image block with local semantic information as a sample. Samples that are close enough to the pupil center are selected as independent local descriptors to predict an offset vector from their own coordinates to the pupil center. Figure 2 For Figure 1 Schematic diagram of sampled faces, where the array of black dots is the center point of the sample image block generated by the neural network on the image, which can be understood semantically as the representation of the image block.

[0061] The present invention aims at the pupil center positioning problem in natural images, and mainly comprises the following steps:

[0062] Step 1: Use common shooting equipment, such as mobile phones, cameras, etc., to collect training image sets. The collected images must contain complete faces, and the pupil area must be identifiable by human annotators.

[0063] Step 2: The annotator uses the annotation tool to annotate the image data. Specifically, for each image I, the annotator annotates two groups of six-dimensional vectors. The i-th group of six-dimensional vectors is recorded as in Used to describe the size of the eye area annotation box. represents the center coordinates of the eye area, (w i ,h i ) represents the width and height of the eye area. Information describing the center of the pupil. represents the coordinates of the pupil center, c i Indicates the category to which the sample falling into the eye region annotation box belongs. In this embodiment, c i Always 1.

[0064] Step 3: Follow Figure 3Construct a pupil center positioning model and initialize the feature extraction module using the network weights pre-trained on ImageNet. Input the training image I into the feature extraction network to obtain the feature set F, and input the feature set F into the pupil center positioning algorithm head to obtain the predicted descriptor score set score and the offset vector set offset given for each sample, where the dimensions of the descriptor score matrix and the offset vector matrix are (H×W×1) and (H×W×2) respectively.

[0065] Step 4: This step corresponds to Figure 3 The loss calculation module in the feature set F is mapped back to the original image according to the following formula based on the corresponding convolution step size s.

[0066]

[0067]

[0068] Among them, (x0, y0) is the sample coordinate mapped back to the original image. If the sample falls within the range of the annotation box B, the sample is recorded as a positive sample, that is, the local descriptor used to vote for the pupil center. The descriptor score set score and the offset vector set offset are input into the loss calculation module to calculate the classification loss and regression loss respectively. The calculation formula is as follows:

[0069]

[0070] L cls (p,c * )=-[α(1-p) γ logp+(1-α)p γ log(1-p)]

[0071] Among them, p represents the prediction score corresponding to a sample, c * Indicates the category corresponding to this sample, o represents the offset vector offset; o * represents the true value of the offset vector, o * The calculation formula is as follows:

[0072]

[0073]

[0074] in, and Respectively represent the distance between the pupil center and the sample (x0, y0) in two dimensions.

[0075] The total loss is calculated as follows:

[0076]

[0077] Among them, N pos is the total number of descriptors, is an indicator function, which is 1 if and only if the sample corresponding to the element with coordinates (i, j) on the feature map is a descriptor, and 0 in other cases.

[0078] Step 5: According to the loss value calculated in step 4, use the gradient descent optimization algorithm to return it, train the entire model, and save the model on disk in the form of a weight file after iterating until convergence.

[0079] Step 6: Follow Figure 4 Construct a pupil center positioning model, load the weight file obtained by training in step 5 from the disk, restore the model's state, input the image I to be detected into the model, and obtain the feature set F, the descriptor score set score, and the offset vector set offset after passing through the feature extraction module and the pupil center positioning algorithm head. Map the feature set F back to the original image according to their respective step sizes to obtain the sample set sampleSet.

[0080] Step 7: This step corresponds to Figure 4 The voting module in step 6. The sample set sampleSet obtained in step 6 is filtered according to the descriptor score. The samples with scores greater than the threshold T are selected as descriptors to obtain the descriptor set descriptorSet. Initialize the accumulation matrix accMatrix with the same size as the original image. For each descriptor, add its coordinates to the corresponding predicted offset vector to obtain a set of predicted pupil center coordinates (e x ,e y ), add the score of the descriptor to the bin at the corresponding position of the accumulation matrix, and the calculation formula is as follows:

[0081]

[0082] Among them, dp represents each specific descriptor.

[0083] Step 8. This step corresponds to Figure 4 The pupil center suppression module in step 7 sorts the accumulated matrix accMatrix in descending order according to the value of each bin, and the sorting result is recorded as sortedBin. The coordinates of the first bin in sortedBin are taken as EC0. For all the remaining bins, the Euclidean distance between them and EC0 is calculated, and the bins that exceed the suppression threshold T are nms The bins are removed from sortedBin to form the suppression set nmsSet. For each bin in nmsSet, its coordinates are weighted to EC0 to obtain the corrected EC0. The calculation formula is as follows:

[0084] EC0+=accMatrix position ·position

[0085] sortedBin=sortedBin\bin

[0086] Among them, position is the coordinate of the corresponding bin in the original accumulation matrix.

[0087] In the remaining bins of sortedBin, take the coordinates of the first bin and mark it as EC1. Then perform the above operation to obtain the corrected EC1. EC0 and EC1 are returned as the final located pupil center.

[0088] The algorithm of the present invention achieved excellent positioning performance on the public data sets BioID and GI4E, reaching a maximum positioning accuracy of 69.08% and 94.66% respectively, which are 9.08% and 4.26% higher than the current SOTA methods. Figure 6 What is shown is the positioning results of the present invention on some collected images and some facial expression analysis data sets, such as Figure 6 As shown, the present invention can achieve stable and accurate pupil center positioning in complex environments.

[0089] refer to Figure 5 The positioning system structure framework diagram constructed by the method of the present invention includes: processors (510, 560), network modules (540, 550), storage units (530, 580), computing units 520, display units 590, sensors 570 and other components. Those skilled in the art can understand that Figure 5 The architecture shown in the figure does not constitute a limitation of the system and may include more or fewer modules than shown in the figure.

[0090] Combine the following Figure 5 A detailed introduction to each module of the system:

[0091] The storage units 530 and 580 are used to store software programs, modules, and necessary data. Specifically, the processor 510 implements various functional applications and service provision by running the software programs and modules stored in the memory 530. Due to the high similarity, the following introduction does not make special distinctions unless necessary. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program, etc.; the data storage area may store specific data. Specifically, the storage unit 530 may store a trained model weight file, a request image cache sent by the terminal, and a cache of positioning results. The storage unit 580 may store the request image data obtained from the sensor 570, and the positioning results obtained from the network module 550, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk device, a flash memory device, or other non-volatile solid-state storage device.

[0092] Network modules 540 and 550 are network communication modules of the remote server and the near-end terminal, respectively, and are responsible for remote interaction of requests and positioning results between the server and the terminal. Specifically, the image requested for detection is sent to the network module 540 through the network module 550, and the positioning result is sent through the network module 540 after the positioning is completed, and the network module 550 transmits it to the processor 560 for image rendering. Optionally, the network module can communicate through a wired network or a wireless network. It is understandable that in order to realize the transmission of image data from the network module 550 to the network module 540, the transmission rate or bandwidth of the network needs to meet certain conditions, which have different definitions in different application scenarios.

[0093] The computing unit 520 is a computing device that carries the core process in the present invention, and is used to process the input image data and return the located pupil center. The computing unit can be mainly composed of a storage area 522 and a computing area 521, wherein the storage area 522 is responsible for caching the data streams incoming from the processor 510 and the storage unit 530, including the weights of the pupil center positioning model, and the image data to be positioned; the computing area 521 is responsible for interacting with the internal storage area 522 to directly process the computing stream. For those skilled in the art, it can be understood that the computing unit 520 can adopt but is not limited to currently available high-performance parallel computing devices, such as independent devices encapsulated with high-performance computing chips, or distributed device clusters, and should have different forms in different application scenarios.

[0094] The sensor 570 and the display unit 590 are components of the proximal terminal. The sensor 570 is responsible for collecting image data in a natural environment, and the display unit 590 can be used to display information input by the user or information provided to the user. Specifically, the sensor 570 can be, but is not limited to, an independent camera (Webcam), a camera device integrated in a smartphone, or other available optical sensing devices. It should be noted that the image data collected by the sensor 570 must maintain sufficient resolution. Images with too low resolution will cause the definition of the pupil positioning problem itself to become blurred. The display unit 590 may include an input area 591 and an output area 592. The input area 591 is used to receive user input information, and can be, but is not limited to, a touch panel, a physical keyboard, a mouse, etc. The output area 592 is used to provide interaction with the user, including responding to the user's terminal input and responding to requests for pupil center positioning, and can be, but is not limited to, a liquid crystal display (LCD), an organic light emitting diode (OLED), etc. It can be understood that although in Figure 5 In the embodiment, the input area 591 and the output area 592 are integrated into a display unit 590, but in some embodiments, the input area 591 and the output area 592 can exist as independent modules and have different forms in different application scenarios.

[0095] Processors 510 and 560 are respectively the logic processing units of the remote server and the near-end terminal, wherein the remote server refers to the machine on which the present invention is deployed, and the terminal includes but is not limited to general electronic devices such as smart phones and personal computers. Processor 510 is the scheduling unit of the present invention on the deployment device, responsible for calling and executing the modules or software programs stored in the storage unit 530, and interacting with the computing unit 520 and the network module 540 for data and instructions. Processor 560 is the logic processing unit of the terminal device that initiates the request, responsible for processing the user's request, obtaining the original image from the sensor 570, interacting with the network module 550 and the storage unit 580, and outputting the results to the display unit 590. Due to the high similarity, the following introduction does not make a special distinction. Optionally, the processor may include one or more processing units; preferably, the processor may integrate an application processor and a modem, wherein the application processor is mainly responsible for processing such as operating systems and applications, and the modem is responsible for processing wireless communications. It can be understood that the above-mentioned modem is not a necessary module, or it may not be integrated into the processor.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An end-to-end pupil center localization method based on a deep voting mechanism, characterized in that: The method specifically comprises the following steps: S1: Obtain an image set containing human faces, annotate the image set, and then divide the annotated image set into a training set and a test set; S2: Construct a pupil center positioning model and input the training set into the model for training, which specifically includes the following steps: S21: Construct pupil center localization model: Use convolutional neural network as the backbone network for feature extraction and construct the pupil center localization algorithm head, including descriptor classifier and offset regressor; S22: input the training image I into the backbone network, extract the feature set F, input the feature set F into the pupil center positioning algorithm head, and obtain the predicted descriptor score set score and the offset vector set offset through the descriptor classifier and the offset regressor; S23: Input the descriptor score set score and the offset vector set offset obtained in step S22 into the loss calculation module to obtain a loss value; S24: according to the loss value calculated in step S23, the gradient descent optimization algorithm is used to transmit it back, and the pupil center positioning model is trained, and iterated until convergence; S25: Input the sample set into the voting module and the pupil center suppression module to obtain the final pupil center positioning result, which specifically includes the following steps: S251: Input the image I to be detected into the model trained in step S24, and obtain the feature set F, the descriptor score set score and the offset vector set offset through the feature extraction module and the pupil center positioning algorithm head, and map the feature set F back to the original image according to their respective step sizes to obtain the sample set sampleSet; S252: The sample set sampleSet obtained in step S251 is filtered according to the descriptor score, and the samples with scores greater than the threshold T are selected as descriptors to obtain the descriptor set descriptorSet; the accumulation matrix accMatrix with the same size as the original image is initialized, and for each descriptor, its coordinates are vector-added with the corresponding predicted offset vector to obtain a set of predicted pupil center coordinates (e x , e y ), add the descriptor score to the bin at the corresponding position of the accumulation matrix dp , the calculation formula is as follows: Among them, dp represents each specific descriptor, dp x and dp y represents the coordinates of the descriptor, and Indicates the offset predicted by the descriptor; S253: sort the accumulated matrix accMatrix obtained in step S252 in descending order according to the value in each bin, and record the sorting result as sortedBin. Take the coordinate of the first bin in sortedBin as EC0, and calculate the Euclidean distance between the remaining bins and EC0. nms The bins are removed from sortedBin to form the suppression set nmsSet. For each bin in nmsSet, its coordinates are weighted to EC0 to obtain the corrected EC0. The calculation formula is as follows: EC0+=accMatrix position ·position sortedBin=sortedBin\bin Among them, position is the coordinate of the corresponding bin in the original accumulation matrix, accMatrix position Represents the accumulated value at the corresponding position on the accumulation matrix accMatrix; In the remaining bins of sortedBin, take the coordinates of the first bin as EC1, and then perform the above operation to obtain the corrected EC1. EC0 and EC1 are returned as the final located pupil center; S3: Input the image to be located in the test set into the trained pupil center location model to obtain the final pupil center location coordinates.

2. The end-to-end pupil center positioning method according to claim 1, characterized in that: In step S1, a set of images containing complete human faces in a natural environment is acquired by a general-purpose photographing device, wherein the pupil area must be able to be identified by a human annotator.

3. The end-to-end pupil center positioning method according to claim 1, characterized in that: In step S1, the annotator uses an annotation tool to annotate the image set, specifically including: for each image I, annotate two groups of six-dimensional vectors, and the i-th group of six-dimensional vectors is recorded as in Used to describe the size of the eye area annotation box. represents the center coordinates of the eye area, (w i ,h i ) represents the width and height of the eye area; Information describing the center of the pupil. represents the coordinates of the pupil center, c i Indicates the category to which the sample falling within the eye region annotation box belongs.

4. The end-to-end pupil center positioning method according to claim 1, characterized in that: In step S23, the descriptor score set score and the offset vector set offset are input into the loss calculation module to obtain the loss value, which specifically includes: mapping each subset in the feature set F back to the original image according to the following formula according to their corresponding convolution step size s, Where (x, y) is the coordinate of the feature point in the feature map, (x0, y0) is the sample coordinate mapped back to the original image, if the sample falls within the range of the annotation box B, then this sample is recorded as a positive sample, that is, the local descriptor used to vote for the pupil center; the descriptor score set score and the offset vector set offset are input into the loss calculation module, and the classification loss L is calculated respectively. reg (o,o * ) and regression loss L cls (p,c * ), the calculation formula is as follows: Lcls(p,c*)=-[α(1-p) γ log p+(1-α)p γ log(1-p)] Among them, p represents the prediction score corresponding to a sample, c* represents the category corresponding to the sample; γ and α are hyperparameters used to adjust the weights of difficult and easy samples and positive and negative samples; o represents the offset vector; o * represents the true value of the offset vector, o * The calculation formula is as follows: in, and Respectively represent the distance between the pupil center and the sample (x0, y0) in two dimensions, (x eye ,y eye ) represents the true coordinate of the pupil center; The total loss calculation formula of the loss calculation module is as follows: Among them, L({p,c * },{o,o * }) represents the total loss value of the loss calculation module; λ is a hyperparameter used to balance classification and regression losses; N pos is the total number of descriptors, is an indicator function, which is 1 if and only if the element with coordinates (i, j) on the feature map corresponds to the sample bit descriptor, and 0 in other cases.

Images