Method and apparatus for constructing line-of-sight prediction model, and device and storage medium
Through knowledge distillation theory, cleaning the line of sight labels and updating the model parameters is solved, and the problem of dirty data influence in the line of sight prediction model is improved, and the prediction accuracy and accuracy of model parameters are improved.
Patent Information
- Application Number
- PCT/CN2024/095931
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
The existing line of sight prediction model has a large amount of dirty data during the construction process, resulting in low prediction accuracy, especially because the subjects fail to accurately look at the preset points, resulting in the eye sample image not matching the corresponding line of sight labels, which affects the model. construction accuracy.
Using knowledge distillation theory, the line of sight labels of the eye sample images used in the student model are cleaned through the line of sight reference information output by the teacher model, reducing the impact of dirty data on model construction, and updating the model parameters based on the line of sight prediction information and target line of sight labels to ensure that Accuracy of model parameters.
The prediction accuracy of the line-of-sight prediction model is improved, and the interference of dirty data on model construction is reduced, ensuring the accuracy of model parameters and the accuracy of the final prediction results.
Smart Images

Figure CN2024095931_05122024_PF_FP_ABST
Abstract
Description
A method, device, equipment and storage medium for constructing a sight line prediction model
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority to a Chinese application with application number 202310620447.8, filing date May 29, 2023, entitled “A method, device, equipment and storage medium for constructing a line of sight prediction model”. The disclosed content of the Chinese application is hereby introduced as a whole into this application. Technical Field
[0003] The present disclosure relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for constructing a line of sight prediction model. Background Art
[0004] Gaze prediction technology uses a network structure to output gaze direction based on a given eye image. It is widely used in fields such as human-computer interaction, virtual reality, augmented reality, and medical analysis.
[0005] Summary of the Invention
[0006] According to one aspect of the present disclosure, a method for constructing a sight line prediction model is provided, comprising:
[0007] Input multiple eye sample images into the constructed teacher model to obtain multiple line of sight reference information.
[0008] A plurality of the eye sample images are input into the student model to be constructed to obtain a plurality of sight line prediction information, wherein each of the eye sample images and each of the sight line reference information corresponds to each of the sight line prediction information.
[0009] When the loss determined by the gaze prediction information and the corresponding true gaze label meets the label screening condition, determining that the target gaze label of the eye sample image includes the corresponding gaze reference information; otherwise, determining that the target gaze label of the eye sample image includes the corresponding true gaze label;
[0010] If it is determined based on multiple gaze prediction information and corresponding target gaze labels that the loss of the student model to be constructed does not meet the convergence condition, the model parameters of the student model to be constructed are updated based on multiple gaze prediction information and corresponding target gaze labels; otherwise, the student model to be constructed is determined to be the gaze prediction model.
[0011] According to another aspect of the present disclosure, a sight line prediction method is provided, comprising:
[0012] Acquire an eye image.
[0013] An eye image is input into a sight line prediction model to obtain sight line information. The sight line prediction model is constructed by the sight line prediction model construction method described in the exemplary embodiment of the present disclosure.
[0014] According to another aspect of the present disclosure, there is provided a construction device, comprising:
[0015] The prediction module is used to input multiple eye sample images into the constructed teacher model to obtain multiple line of sight reference information.
[0016] A construction module is used to input multiple eye sample images into the student model to be constructed, obtain multiple line of sight prediction information, each eye sample image and each line of sight reference information corresponds to each line of sight prediction information, if it is determined based on the multiple line of sight prediction information and the corresponding target line of sight labels that the loss of the student model to be constructed does not meet the convergence condition, update the model parameters of the student model to be constructed based on the multiple line of sight prediction information and the corresponding target line of sight labels, if it is determined based on the multiple line of sight prediction information and the corresponding target line of sight labels that the loss of the student model to be constructed meets the convergence condition, determine that the student model to be constructed is the line of sight prediction model.
[0017] When the loss determined between the gaze prediction information and the corresponding true gaze label meets the label screening condition, the target gaze label of the eye sample image is the corresponding gaze reference information.
[0018] According to another aspect of the present disclosure, a sight line prediction device is provided, comprising:
[0019] The acquisition module is used to acquire eye images.
[0020] A prediction module is used to input an eye image into a sight line prediction model to obtain sight line information. The sight line prediction model is constructed by the sight line prediction model construction method described in the exemplary embodiment of the present disclosure.
[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0022] processor; and
[0023] Memory for storing programs.
[0024] The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.
[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the exemplary embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0027] FIG1 shows a flowchart of a method for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure;
[0028] FIG2 shows a flow chart of a sight line prediction method according to an exemplary embodiment of the present disclosure;
[0029] FIG3 shows a schematic block diagram of functional modules of a construction device according to an exemplary embodiment of the present disclosure;
[0030] FIG4 shows a schematic block diagram of functional modules of a sight line prediction device according to an exemplary embodiment of the present disclosure;
[0031] FIG5 shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure;
[0032] FIG6 shows a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. DETAILED DESCRIPTION
[0033] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0035] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0036] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0037] Before introducing the embodiments of the present disclosure, the following definitions are given for the relevant terms involved in the embodiments of the present disclosure:
[0038] Virtual reality (VR), also known as virtual reality or virtual reality technology, is a new and practical technology developed in the 20th century. It encompasses computer, electronic information, and simulation technologies. Its fundamental implementation relies primarily on computer technology, leveraging and integrating the latest advances in high-tech technologies such as 3D graphics, multimedia, simulation, display, and servo technology. Using computers and other devices, it creates a realistic 3D virtual world with multiple sensory experiences, including visual, tactile, and olfactory experiences. This creates an immersive experience for those in the virtual world.
[0039] Knowledge distillation (KD) is a construction method based on the "teacher-student network" concept. As the name suggests, it distills the knowledge contained in an existing teacher model into the student model. By introducing soft targets related to the teacher model as part of the total loss, it guides the construction of the student model and achieves knowledge transfer.
[0040] The loss function, also known as the objective function, is an important equation used to measure the difference between the predicted value and the target value. The higher the output value (loss) of the loss function, the greater the difference. Therefore, the construction of deep neural networks becomes a process of minimizing this loss as much as possible.
[0041] Model parameters refer to the optimal parameters obtained by constructing samples during model construction, which can be understood as the "internal parameters" of the model.
[0042] When a user uses a VR device, the VR device can collect the user's eye image through the eye tracking system, and predict the user's eye line based on the user's eye image, and then use the predicted user's eye line to control the VR device to perform eye tracking to improve the user's comfort.
[0043] Gaze prediction technology uses a network structure to output gaze direction given an eye image. It is widely used in fields such as human-computer interaction, virtual reality, augmented reality, and medical analysis. During application, the subject is guided to gaze at different points. By capturing eye images at these points, data is generated. Each point corresponds to a known gaze direction, which serves as a label for this data. However, in practice, it is often the case that, for example, the subject is guided to gaze at point x, but in reality does not gaze at that point. This results in a large amount of dirty data in the gaze estimation data, significantly impacting model accuracy.
[0044] Specifically, in the related art, the user's eye image can be input into the line of sight prediction model to obtain the user's line of sight. When constructing the line of sight prediction model, a large number of eye sample images with line of sight labels can be used as a data set to construct the line of sight prediction model. When collecting eye sample images with line of sight labels, the subject can be guided to look at the preset gaze point, but the subject may not follow the guidance to look at the preset gaze point for some reason, resulting in a mismatch between the collected eye sample image and the corresponding line of sight label. Based on this, when constructing the line of sight prediction model, there is a certain amount of dirty data in the data set used. These dirty data can be manifested as a mismatch between the eye sample image and the corresponding real line of sight label, resulting in low prediction accuracy of the constructed line of sight prediction model.
[0045] In response to the above problems, the exemplary embodiments of the present disclosure provide a method for constructing a line of sight prediction model and a line of sight prediction method, which can be based on the knowledge distillation theory and use the line of sight reference information output by the teacher model to clean the line of sight labels corresponding to the eye sample images used by the student model, thereby reducing the impact of dirty data on the construction of the line of sight prediction model and improving the prediction accuracy of the line of sight prediction model. It should be understood that the relevant eye images involved in the method for constructing a line of sight prediction model and the line of sight prediction method provided by the exemplary embodiments of the present disclosure can be images containing human eyes or images containing animal eyes, which are not limited here.
[0046] In one or more technical solutions provided in the exemplary embodiments of the present disclosure, during the process of constructing a gaze prediction model, multiple eye sample images are input into the student model to be constructed to obtain multiple gaze prediction information, and then the loss of the student model to be constructed is determined based on the multiple gaze prediction information and the corresponding target gaze labels. If the convergence condition is met, it means that the model parameters of the student model to be constructed are accurate, and the student model to be constructed can more accurately predict the gaze prediction information of the eye sample images. If the convergence condition is not met, it means that the model parameters of the student model to be constructed are not accurate enough, and the student model to be constructed cannot accurately predict the gaze prediction information of the eye sample images. Therefore, the model parameters of the student model to be constructed can be updated based on the multiple gaze prediction information and the corresponding target gaze labels to ensure that the model parameters of the student model to be constructed are accurate.
[0047] Moreover, when the exemplary embodiment of the present disclosure determines the loss of the student model to be constructed, when the loss determined by the line of sight prediction information and the corresponding real line of sight label meets the label screening condition, it means that the difference between the line of sight prediction information and the corresponding real line of sight label is large, and the reasons for this difference may include two aspects. On the one hand, it may be because there is a problem with the model parameters of the student model to be constructed. On the other hand, it may also be because the matching between the real line of sight label and the corresponding eye sample image is poor, that is, the real line of sight label and the corresponding eye sample image are essentially dirty data in the constructed sample. Based on this, the corresponding line of sight reference information output by the constructed teacher model based on the eye sample image can be determined as the target line of sight label of the eye sample image to reduce the interference caused by the poor matching between the real line of sight label and the corresponding eye sample image, thereby improving the accuracy of the determined loss of the student model to be constructed and ensuring that the line of sight prediction model finally obtained has high accuracy.
[0048] The method of the exemplary embodiments of the present disclosure may be performed by an electronic device or a chip applied to the electronic device. The electronic device may be an electronic device with a display function, such as a mobile phone, a tablet computer, a wearable device, an in-vehicle device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology.
[0049] For example, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed by applying wearable technology to daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are full-featured, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.
[0050] FIG1 illustrates a flow chart of a method for constructing a sight line prediction model provided by an exemplary embodiment of the present disclosure. As shown in FIG1 , the method for constructing a sight line prediction model of an exemplary embodiment of the present disclosure includes:
[0051] Step 101: Input multiple eye sample images into the constructed teacher model to obtain multiple sight line reference information. The eye sample images can be eye sample images collected by an image acquisition device, and the eye sample images can be eye pictures or eye videos.
[0052] In practical applications, eye sample images can be captured by the camera in the VR device, and then multiple collected eye sample images can be input into the constructed teacher model for training. The teacher model can be any existing open source neural network model for line of sight prediction, and is not limited here. For example, it can be a Visual Geometry Group (VGG) network, a Deep Residual Network (ResNet) or a Convolutional Neural Network (CNN) network.
[0053] The teacher model construction process can be as follows: by inputting multiple eye sample images with gaze labels into the teacher model to obtain corresponding multiple gaze reference information, solving the loss based on the gaze reference information corresponding to the multiple eye sample images and the true gaze labels, if the loss meets the iteration condition, it means that the gaze reference information of the eye sample images of the teacher model is inaccurate, and the model parameters of the teacher model can be updated based on the gaze reference information corresponding to the multiple eye sample images and the true gaze labels (the model parameters may include weights and offset values). If the loss meets the iteration termination condition, it means that the gaze reference information of the eye sample images predicted by the teacher model is relatively accurate, and the constructed teacher model can be obtained.
[0054] The eye sample images should carry the true gaze labels as accurately as possible to improve the accuracy of the teacher model. The eye sample images used in the teacher model construction process can be the same as or different from the eye sample images input to the teacher model when building the student model. This is not limited here.
[0055] Step 102: Input multiple eye sample images to the student model to be constructed to obtain multiple gaze prediction information. The eye sample images, gaze reference information, and gaze prediction information correspond to each other. The description of inputting multiple eye sample images to the student model to be constructed can be found in the previous description and is not repeated here.
[0056] In the exemplary embodiment of the present disclosure, the multiple eye sample images input to the student model to be constructed and the multiple eye sample images input to the constructed teacher model can be the same. For example, when the eye sample images, the sight line reference information and the sight line prediction information correspond to each other, the same eye sample image is input to the constructed teacher model and the student model to be constructed, and the same eye sample image can correspond to the sight line reference information output by the constructed teacher model and the sight line prediction information output by the student model to be constructed, respectively, and there is also a correspondence between the sight line reference information and the sight line prediction information.
[0057] Considering that the sample eye images used may contain a certain amount of dirty data when carrying real gaze labels, the exemplary embodiments of the present disclosure may also clean the real gaze labels so that the real gaze labels corresponding to the sample eye images used in the loss calculation have a high degree of matching with the sample eye images, thereby improving the accuracy of the student model to be constructed. Based on this, the method of the exemplary embodiments of the present disclosure may also include:
[0058] Step 103: Determine whether the loss determined by the gaze prediction information and the corresponding true gaze label meets the label filtering condition. Here, the true gaze label carried by the eye sample image corresponding to the gaze prediction information can be the true gaze label corresponding to the gaze prediction information. For any eye sample image, the loss can be determined based on the gaze prediction information and the corresponding true gaze label.
[0059] For any eye sample image, its loss can be considered as a single-sample loss. If the single-sample loss meets the label filtering condition, it means that the true gaze label corresponding to the eye sample image used in the loss calculation is significantly different from the eye sample image. The reasons for this difference may include two aspects. On the one hand, it may be due to problems with the model parameters of the student model to be constructed. On the other hand, it may be due to poor matching between the true gaze label and the corresponding eye sample image. In other words, the true gaze label and the corresponding eye sample image are essentially dirty data in the constructed sample, and the gaze prediction information and the corresponding true gaze label cannot be directly used to confirm the convergence of the student model to be constructed. In this case, step 104 can be executed. When the loss determined by the gaze prediction information and the corresponding true gaze label does not meet the label filtering condition, it means that the true gaze label corresponding to the eye sample image used in the loss calculation is well matched with the eye sample image. The gaze prediction information and the corresponding true gaze label can be directly used to confirm the convergence of the student model to be constructed. In this case, step 105 can be executed.
[0060] Step 104: Determine that the target gaze label of the eye sample image includes the corresponding gaze reference information. Here, since the same eye sample image is input into the student model to be constructed and the constructed teacher model respectively, and the constructed teacher model has a higher gaze prediction accuracy, when the loss determined by the gaze prediction information and the corresponding true gaze label meets the label screening condition, the gaze reference information corresponding to the eye sample image can be used to replace the true gaze label carried by the eye sample image, and the gaze reference information corresponding to the eye sample image can be used as the target gaze label to determine the convergence of the student model to be constructed, thereby improving the convergence accuracy of the student model to be constructed.
[0061] It can be seen that the exemplary embodiments of the present disclosure can reduce the construction interference caused by the poor matching between the real gaze label and the corresponding eye sample image when the loss determined by the gaze prediction information and the corresponding real gaze label meets the label screening condition.
[0062] Step 105: Determine whether the target gaze label of the eye sample image includes the corresponding true gaze label. In this case, the loss between the gaze prediction information and the corresponding true gaze label does not meet the label filtering condition. Therefore, there is no need to clean the true gaze label. Therefore, the target gaze label of the eye sample image can be determined to be the corresponding true gaze label.
[0063] Step 106: Determine whether the loss of the student model to be constructed meets the convergence condition based on the multiple gaze prediction information and the corresponding target gaze labels. The student model architecture can be a lightweight model architecture such as SqueezeNet, MobileNet, or ShuffleNet.
[0064] In practical applications, when determining the loss of a student model to be constructed based on multiple gaze prediction information and corresponding target gaze labels, the multiple gaze prediction information and corresponding target gaze labels can be substituted into a loss function to determine the loss of the student model to be constructed. The loss function can be a softmax loss function or other possible loss functions, such as a cross-entropy loss.
[0065] The convergence condition here can be that the loss of the student model to be constructed is less than or equal to the preset loss, or that the loss of the student model to be constructed is stable, that is, the loss of the student model to be constructed remains constant or fluctuates within a certain range during different iterations.
[0066] When the loss of the student model to be constructed does not meet the convergence condition, it means that the gaze prediction information output by the student model is significantly different from the target gaze label and cannot meet the gaze prediction requirements of the reasoning stage (for example, the gaze prediction accuracy does not meet the requirements). Therefore, step 107 can be executed. When the loss of the student model to be constructed meets the convergence condition, it means that the gaze prediction information output by the student model to be constructed is well matched with the target gaze label and meets the gaze prediction requirements of the reasoning stage (for example, the gaze prediction accuracy meets the requirements). Therefore, step 108 can be executed.
[0067] Step 107: Update the model parameters of the student model to be constructed based on the multiple gaze prediction information and the corresponding target gaze labels. Here, the model parameters of the student model to be constructed can be updated using a backpropagation algorithm, and then the student model to be constructed with the updated model parameters is reconstructed based on the next batch of eye sample images. It should be understood that the next batch of eye sample images can be at least partially identical to the current batch of eye sample images, or they can be completely different.
[0068] Step 108: Determine whether the student model to be constructed is a gaze prediction model. For example, if the loss of the student model to be constructed satisfies the convergence condition based on the multiple gaze prediction information and the corresponding target gaze labels, the construction of the student model to be constructed can be terminated directly, and the current student model to be constructed can be used as the gaze prediction model.
[0069] As a possible implementation, the tag screening conditions of the exemplary embodiment of the present disclosure may be limited by threshold comparison or by loss ranking.
[0070] When the label filtering condition of the exemplary embodiment of the present disclosure is limited by a threshold comparison method, the label filtering condition includes: the loss determined based on the gaze prediction information and the corresponding real gaze label is greater than the loss threshold. It should be understood that the loss threshold here can be determined based on experience.
[0071] In practical applications, by judging the relationship between the loss determined by the gaze prediction information and the corresponding true gaze label and the loss threshold, the eye sample image whose loss determined by the gaze prediction information and the corresponding true gaze label is greater than the loss threshold is determined as dirty data. The gaze reference information output by the constructed teacher model with higher gaze prediction accuracy is used as the target gaze label, and the loss is re-determined with the corresponding gaze prediction information, thereby cleaning up the dirty data in the eye sample image and avoiding the influence of dirty data on the accuracy of constructing the student model to be constructed.
[0072] When the label filtering condition of the exemplary embodiment of the present disclosure is limited by loss sorting, the method provided by the exemplary embodiment of the present disclosure may further include: sorting in descending order according to the losses determined based on the line of sight prediction information and the corresponding real line of sight labels, and the label filtering condition includes: the order of the losses determined based on the line of sight prediction information and the corresponding real line of sight labels in the sorting is less than or equal to M, where M represents an integer less than the total number of line of sight prediction information.
[0073] In practical applications, the gaze prediction information and the corresponding true gaze label can be brought into the loss function for solution to obtain the loss corresponding to each eye sample image, and the losses of different eye sample images are sorted in descending order. If the order of the loss determined by the gaze prediction information and the corresponding true gaze label in the sorting is less than or equal to M, it means that the true gaze label may not match its corresponding eye sample image. Therefore, it can be considered that the loss determined by the gaze prediction information and the corresponding true gaze label meets the label filtering condition, and the target gaze label of the eye sample image corresponding to the gaze prediction information is set as the corresponding gaze reference information, thereby avoiding the influence of dirty data on the accuracy of constructing the student model to be constructed.
[0074] The aforementioned M can be considered the number of sample errors in the eye sample image construction set, which can be determined by the eye sample image cleaning rate and the total number of eye sample images. The eye sample image cleaning rate here can reflect the error rate of the eye sample images and can be determined statistically or customized based on actual conditions. The total number of eye sample images can be the number of eye sample images required to update the model parameters of the student model to be constructed, or the number of eye sample images in a batch for constructing the student model.
[0075] Exemplarily, the above M is positively correlated with the cleaning rate of the eye sample images, and M is positively correlated with the total number of the eye sample images.
[0076] Taking M = rate * batch_size as an example, when the cleaning rate of the eye sample image is rate = 0.01 and the total number of eye sample images, batch_size = 1000, M = 10. This means that the number of dirty data that need to be cleaned in this batch of eye sample images is 10. At this time, the gaze prediction information corresponding to the eye sample image and its corresponding true gaze label can be brought into the loss function for solution to obtain the loss corresponding to each eye sample image. Based on the order of the losses of different eye sample images from large to small, the gaze prediction information corresponding to each loss is sorted. Based on the sorted gaze prediction information, the target gaze labels corresponding to the first 10 gaze prediction information are updated to their corresponding gaze reference information, and the target gaze labels corresponding to the remaining 990 gaze prediction information are updated to their corresponding true gaze labels. This is to re-determine the convergence of the student model to be constructed and reduce the impact of dirty data on the accuracy of the student model to be constructed.
[0077] It can be seen from the above that, when constructing the student model to be constructed, no matter what label screening conditions are adopted in the exemplary embodiment of the present disclosure, all target sight labels corresponding to the multiple eye sample images may include real sight labels corresponding to different eye sample images and sight reference information corresponding to different eye sample images. The sight reference information can be used to clean the dirty data in all real sight labels corresponding to the multiple eye sample images, thereby reducing the impact of the dirty data on the accuracy of constructing the student model to be constructed.
[0078] As a possible implementation method, when the loss determined by the gaze prediction information and the corresponding true gaze label does not meet the label filtering condition, it is determined that the target gaze label corresponding to the gaze prediction information also includes corresponding gaze reference information, and the loss determined by the gaze prediction information and the corresponding target gaze label includes a first loss and a second loss. Based on the first loss and the second loss, the loss of the corresponding eye sample image can be determined.
[0079] In actual applications, when the loss determined by the gaze prediction information and the corresponding true gaze label does not meet the label screening condition, it means that there is no dirty data in the current eye sample image, and the true gaze label corresponding to the eye sample image can be directly used to calculate the loss. However, in order to improve the prediction accuracy of the student model to be constructed, the gaze reference information and the true gaze label output by the teacher model with higher accuracy can be used to supervise the student model to be constructed at the same time. At this time, the target gaze label of the gaze prediction information corresponding to the same eye sample image can be selected from the corresponding gaze reference information and the corresponding true gaze label respectively, and the first loss is determined by the gaze prediction information and the corresponding true gaze label, and the second loss is determined by the gaze prediction information and the corresponding gaze reference information, and then the loss of the corresponding eye sample image is determined based on the first loss and the second loss. It should be understood that the loss of the corresponding eye sample image determined based on the first loss and the second loss can be obtained by weighted summing the first loss and the second loss, or by using other methods, which are not limited here. In some examples, the weight of the first loss is greater than the weight of the second loss.
[0080] In other examples, the loss function involved in the sight line prediction model of the exemplary embodiments of the present disclosure can also be expressed as:
[0081] Where, L soft It is the loss obtained by the sight reference information output by the teacher model constructed at temperature = T and the sight prediction information output by the student model to be constructed at temperature = T. This includes the loss determined by the gaze prediction information and the true gaze label when the eye sample image matches the true gaze label. This includes the loss determined by the gaze prediction information and the gaze reference information when the eye sample image does not match the true gaze label.
[0082] in, Represents the sight reference information of the rth class by the softmax output of the constructed teacher model at temperature = T; represents the gaze prediction information of the rth class of the softmax output of the student model to be constructed at temperature = T, N represents the total number of eye sample images input to the student model to be constructed, and r represents an integer greater than or equal to 1 and less than or equal to N.
[0083] c j Indicates the true sight label of the jth class that does not meet the label filtering conditions; It represents the line of sight prediction information of the jth class under the softmax output of the student model to be constructed at temperature = 1.
[0084] Indicates the sight reference information of the kth class under the softmax output of the constructed teacher model at temperature = 1; It represents the gaze prediction information of the kth class under the softmax output of the student model to be constructed at temperature = 1.
[0085] The exemplary embodiments of the present disclosure further provide a gaze prediction method for predicting gaze direction in an eye image. FIG2 illustrates a flow chart of the gaze prediction method provided by the exemplary embodiments of the present disclosure. As shown in FIG2 , the gaze prediction method includes:
[0086] Step 201: Obtain an eye image. The eye sample image is an eye sample image containing a gaze direction. It should be understood that the eye sample image here can be a picture or a video image.
[0087] In practical applications, the eye image can be acquired by an image acquisition device, which can be a camera or a video camera. It can be a separate device, or a camera of a computing device with a display screen, or a camera installed on a bicycle, motorcycle, car or other vehicle. The camera can be a monocular camera or a binocular camera, etc.
[0088] Step 202: Input the eye image into the gaze prediction model to obtain gaze information. The gaze prediction model can be constructed using a gaze prediction model construction method. It should be noted that the gaze information can be the user's "gaze direction" captured by the image acquisition device. For example, when the gaze information is the user's "gaze direction," when the gaze prediction method is used in offline retail, the user's current "gaze direction" can be used to determine the user's interest in products on the shelf, thereby enabling timely product selection and placement adjustments.
[0089] The above mainly introduces the solutions provided by the embodiments of the present disclosure. It is understandable that, in order to achieve the above functions, the device of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.
[0090] The embodiments of the present disclosure can divide the functional units of the device of the present disclosure according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0091] In the case of dividing each functional module according to each function, the exemplary embodiment of the present disclosure provides a construction device, which can be an electronic device or a chip used in an electronic device. Figure 3 shows a schematic block diagram of the functional modules of the construction device according to the exemplary embodiment of the present disclosure. As shown in Figure 3, the construction device 300 includes:
[0092] Prediction module 301 is configured to input multiple eye sample images into the constructed teacher model to obtain multiple gaze reference information. It then inputs multiple eye sample images into the student model to be constructed to obtain multiple gaze prediction information. Each eye sample image and each gaze reference information corresponds to a piece of gaze prediction information.
[0093] Determination module 302 is configured to determine the target gaze label of the eye sample image as the corresponding gaze reference information when the loss determined between the gaze prediction information and the corresponding true gaze label meets the label screening condition; otherwise, determine the target gaze label of the eye sample image as the corresponding true gaze label. If the loss of the student model to be constructed based on the multiple gaze prediction information and the corresponding target gaze labels meets the convergence condition, the student model to be constructed is determined to be the gaze prediction model.
[0094] The updating module 303 is used to update the model parameters of the student model to be constructed based on the multiple gaze prediction information and the corresponding target gaze labels if it is determined that the loss of the student model to be constructed does not meet the convergence condition based on the multiple gaze prediction information and the corresponding target gaze labels.
[0095] In a possible implementation, the label screening condition includes: a loss determined between the gaze prediction information and the corresponding true gaze label is greater than a loss threshold.
[0096] In one possible implementation, the label screening condition includes: sorting multiple line of sight prediction information in descending order according to the loss determined based on the line of sight prediction information and the corresponding target line of sight label, the line of sight prediction information is the kth line of sight prediction information, k is greater than 0 and less than or equal to M, and M represents an integer less than the total number of line of sight prediction information.
[0097] In a possible implementation, the above M is positively correlated with the cleaning rate of the eye sample images, and M is positively correlated with the total number of the eye sample images.
[0098] In one possible implementation, when the loss determined by the line of sight prediction information and the corresponding real line of sight label does not meet the label filtering condition, it is determined that the target line of sight label corresponding to the line of sight prediction information also includes corresponding line of sight reference information, and the loss determined by the line of sight prediction information and the corresponding target line of sight label includes a first loss and a second loss.
[0099] In a possible implementation, the first loss is determined by the gaze prediction information and the corresponding true gaze label, and the second loss is determined by the gaze prediction information and the corresponding gaze reference information.
[0100] In the case of dividing each functional module according to each function, the exemplary embodiment of the present disclosure further provides a sight line prediction device, which can be an electronic device or a chip applied to an electronic device. Figure 4 shows a schematic block diagram of the functional modules of the sight line prediction device according to the exemplary embodiment of the present disclosure. As shown in Figure 4, the sight line prediction device 400 includes:
[0101] The acquisition module 401 is used to acquire an eye image.
[0102] The prediction module 402 is used to input the eye image into the sight line prediction model to obtain sight line information. The sight line prediction model is constructed by the above-mentioned sight line prediction model construction method.
[0103] Figure 5 shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure. As shown in Figure 5, the chip 500 includes one or more (including two) processors 501 and a communication interface 502. The communication interface 502 can support the electronic device to perform the data transmission and reception steps in the above method, and the processor 501 can support the electronic device to perform the data processing steps in the above method.
[0104] Optionally, as shown in FIG5 , the chip 500 further includes a memory 503 , which may include a read-only memory and a random access memory, and provides operating instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).
[0105] In some embodiments, as shown in FIG5 , the processor 501 performs corresponding operations by calling operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 501 controls the processing operations of any one of the terminal devices, and the processor may also be referred to as a central processing unit (CPU). The memory 503 may include a read-only memory and a random access memory, and provides instructions and data to the processor 501. A portion of the memory 503 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include, in addition to the data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are labeled as bus system 504 in FIG5 .
[0106] The methods disclosed in the above embodiments of the present disclosure can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.
[0107] The exemplary embodiments of the present disclosure further provide an electronic device including: a processor; and a memory communicatively connected to the processor and storing a program. The program includes instructions that, when executed by the processor, cause the processor to perform the method according to the exemplary embodiments of the present disclosure.
[0108] Exemplary embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer instructions are used to cause the computer to execute the method according to the embodiment of the present disclosure.
[0109] With reference to Figure 6, a block diagram of an electronic device 600 that can serve as a server or client of an exemplary embodiment of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0110] As shown in Figure 6, electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 606 into a random access memory (RAM) 603. Various programs and data required for the operation of device 600 can also be stored in RAM 603. Computing unit 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.
[0111] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 604 can include but is not limited to a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TMdevices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.
[0112] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method may be implemented as a computer software program that is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the aforementioned method in any other appropriate manner (e.g., by means of firmware).
[0113] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0114] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0115] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0117] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0118] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.
[0119] In the above embodiments, they can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present disclosure are performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).
[0120] Although the present disclosure has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely illustrative of the present disclosure as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, the present disclosure is intended to include such modifications and variations if they fall within the scope of the claims of the present disclosure and their equivalents.
Claims
1. A method for constructing a sight line prediction model, the method comprising: Input multiple eye sample images into the constructed teacher model to obtain multiple sight reference information; Inputting a plurality of the eye sample images into the student model to be constructed to obtain a plurality of sight line prediction information, wherein the eye sample images, the sight line reference information and the sight line prediction information correspond to each other; When the loss determined by the sight prediction information and the corresponding real sight label meets the label screening condition, determining that the target sight label of the eye sample image includes the corresponding sight reference information, otherwise, determining that the target sight label of the eye sample image includes the corresponding real sight label; If it is determined based on the multiple line of sight prediction information and the corresponding target line of sight labels that the loss of the student model to be constructed does not meet the convergence condition, the model parameters of the student model to be constructed are updated based on the multiple line of sight prediction information and the corresponding target line of sight labels; otherwise, the student model to be constructed is determined to be the line of sight prediction model.
2. The method for constructing a sight line prediction model according to claim 1, wherein: The label screening condition includes: a loss determined based on the line of sight prediction information and a corresponding real line of sight label is greater than a loss threshold.
3. The method for constructing a sight line prediction model according to claim 1, wherein: The construction method also includes: sorting in descending order according to the losses determined based on the line of sight prediction information and the corresponding real line of sight labels, and the label screening condition includes: the order of the losses determined based on the line of sight prediction information and the corresponding real line of sight labels in the sorting is less than or equal to M, where M represents an integer less than the total number of line of sight prediction information.
4. The method for constructing a sight line prediction model according to claim 3, wherein: The M is positively correlated with the cleaning rate of the eye sample images, and the M is positively correlated with the total number of the eye sample images.
5. The method for constructing a sight line prediction model according to any one of claims 1 to 4, wherein: The construction method also includes: When the loss determined by the line of sight prediction information and the corresponding real line of sight label does not meet the label filtering condition, it is determined that the target line of sight label corresponding to the line of sight prediction information also includes the corresponding line of sight reference information, and the loss determined by the line of sight prediction information and the corresponding target line of sight label includes a first loss and a second loss, the first loss is determined by the line of sight prediction information and the corresponding real line of sight label, and the second loss is determined by the line of sight prediction information and the corresponding line of sight reference information.
6. The method for constructing a sight line prediction model according to claim 5, wherein: The weight of the first loss is greater than the weight of the second loss.
7. A sight line prediction method, comprising: Acquiring an eye image; An eye image is input into a sight line prediction model to obtain sight line information, wherein the sight line prediction model is constructed by the sight line prediction model construction method according to any one of claims 1 to 6.
8. A construction device, comprising: The prediction module is configured to input a plurality of eye sample images into the constructed teacher model to obtain a plurality of sight line reference information, input a plurality of the eye sample images into the student model to be constructed to obtain a plurality of sight line prediction information, each of the eye sample images and each of the sight line reference information corresponds to each of the sight line prediction information, A determination module is configured to determine that the target sight line label of the eye sample image is the corresponding sight line reference information when the loss determined by the sight line prediction information and the corresponding real sight line label meets the label screening condition; otherwise, the target sight line label of the eye sample image is determined to be the corresponding real sight line label; if the loss of the student model to be constructed is determined based on the plurality of sight line prediction information and the corresponding target sight line labels to meet the convergence condition, the student model to be constructed is determined to be a sight line prediction model; The updating module is configured to update the model parameters of the student model to be constructed based on the multiple line of sight prediction information and the corresponding target line of sight labels if it is determined based on the multiple line of sight prediction information and the corresponding target line of sight labels that the loss of the student model to be constructed does not meet the convergence condition.
9. A sight line prediction device, comprising: An acquisition module, configured to acquire an eye image; The prediction module is configured to input an eye image into a sight line prediction model to obtain sight line information, wherein the sight line prediction model is constructed by the method according to any one of claims 1 to 6.
10. An electronic device, comprising: processor; as well as a memory storing a program in communication with the processor, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.
11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.
12. A computer program product comprising one or more computer programs, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7. 13 . A computer program which, when executed by a computer, causes the computer to execute the method according to claim 1 .
Citation Information
Patent Citations
Product defect classification method based on artificial intelligence, device, equipment and medium
CN111192262A
Image processing method and device, electronic equipment and readable storage medium
CN112307815A
Image segmentation method and device, electronic equipment and medium
CN115797373A
Eye movement tracking method and device, electronic equipment, storage medium and program product
CN116091541A
Decentralized artificial intelligence (AI) / machine learning training system
US20220344049A1
