Service estimation model training method, service estimation method and related equipment
By integrating deep learning models that combine multi-dimensional human eye and environmental features, the problem of insufficient accuracy of human eye analysis technology in complex environments has been solved, enabling personalized service demand judgment in multi-user scenarios and improving the user experience of display devices.
Patent Information
- Application Number
- CN202410939155.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
Existing human eye analysis technology lacks accuracy in complex external environments, making it difficult to accurately determine user needs, especially in multi-user scenarios where personalized services are insufficient.
By integrating static features of the human eye, motion features of the human eye, features of the external environment, and user viewing preferences, a Transformer-based deep learning model is used for feature fusion and training. Combined with a non-dominated sorting genetic algorithm to optimize personalized settings, multi-dimensional analysis is achieved.
It improves the comprehensiveness and accuracy of human eye movement cause analysis, reduces the false judgment rate, and can more accurately judge user needs and provide personalized display device services.
Smart Images

Figure CN118918413A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and more specifically, to a training method of a service prediction model, a service prediction method, and related devices. Background Art
[0002] Human eye analysis technology has been widely used in various fields, including pupil analysis, gaze tracking, emotion recognition, etc. Various services can be provided to users based on human eye analysis technology. Specifically, product optimization can be pushed based on the user's pupil response; or the brightness of the display device can be adjusted based on the change in the user's pupil diameter, etc.
[0003] However, the human eye features used in the human eye analysis technology in the related art are relatively single, and there is a lack of accuracy in judging user needs based on human eye features, especially in complex external environments, where the state of the human eye is easily affected by changes in external light, emotional fluctuations, and external foreign body stimulation. At this time, it is easy to misjudge user needs based only on a single human eye feature, which may make it difficult to accurately provide users with services that meet their actual needs. Summary of the invention
[0004] The present invention provides a training method for a service prediction model, a service prediction method and related equipment to at least solve the problem in the above-mentioned related technologies that it is difficult to accurately provide users with services that meet their actual needs because service prediction is performed based solely on a single human eye feature.
[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a service prediction model is provided, comprising: obtaining a fused feature sample set, wherein each fused feature sample comprises a human eye static feature sub-sample, a human eye motion feature sub-sample, an external environment feature sub-sample, a user viewing preference sub-sample corresponding to the user, and a real identity label of the corresponding user and a real reason label for the user's eye movement, wherein the human eye static feature sub-sample and the human eye motion feature sub-sample are samples obtained by collecting eye features of the user in the process of the user using the display device, the external environment feature sub-sample is a sample obtained by collecting external environment features in the process of the user using the display device, and the user viewing preference sub-sample is a sample obtained by collecting features of content displayed by the display device in the process of the user using the display device. The obtained sample; inputting the fused feature sample into the service prediction model to obtain the predicted identity of the corresponding user and the predicted reason of the user's eye movement, wherein the service prediction model obtains the predicted identity of the corresponding user based on the human eye static feature sub-sample, and the service prediction model obtains the predicted reason of the user's eye movement of the corresponding user based on the human eye movement feature sub-sample, the external environment feature sub-sample and the user viewing preference sub-sample, wherein the obtained predicted identity of the corresponding user and the predicted reason of the user's eye movement are used to predict and execute the display device service; based on the predicted identity, the real identity label, the predicted reason of the user's eye movement, and the real reason label of the user's eye movement, calculate the loss; by adjusting the parameters of the service prediction model according to the loss, the service prediction model is trained.
[0006] Optionally, the calculating of loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement includes: calculating identity loss based on the predicted identity and the real identity label; calculating eye movement loss based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement; calculating the loss based on the identity loss and the eye movement loss.
[0007] Optionally, the training method also includes: pre-training the original service prediction model to obtain a pre-trained service prediction model, replacing part of the original layers contained in the pre-trained service prediction model with new layers to obtain a new service prediction model; inputting the fused feature sample into the service prediction model includes: inputting the fused feature sample into the new service prediction model; adjusting the parameters of the service prediction model according to the loss includes: while freezing the parameters of at least part of the pre-trained layers in the new service prediction model, adjusting the parameters of other layers in the new service prediction model according to the loss.
[0008] Optionally, the new layer corresponds to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
[0009] Optionally, the loss is calculated based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the label of the real reason for the user's eye movement, including: calculating identity loss based on the predicted identity and the real identity label; calculating eye movement loss based on the predicted reason for the user's eye movement and the label of the real reason for the user's eye movement; and calculating the weighted sum of the identity loss and the eye movement loss as the loss by using the first weight parameter as the weight of the identity loss and the second weight parameter as the weight of the eye movement loss.
[0010] Optionally, the static features of the human eye include at least one of the following items: features of the iris of the human eye, features of the pupil of the human eye, and features of the edge of the eyeball of the human eye; the motion features of the human eye include at least one of the following items: gaze point position, features of changes in pupil diameter of the human eye, features of the movement rate of the human eye, and features of the frequency of blinking of the human eye; the user viewing preferences include at least one of the following items: content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; the external environment features include light intensity features and / or environmental noise features.
[0011] According to a second aspect of an embodiment of the present disclosure, a service prediction method is provided, which is applied to a display device, and includes: obtaining a user's fused eye features and external environment features, wherein the fused eye features include human eye static features and human eye motion features, wherein the human eye static features and the human eye motion features are obtained by extracting features of the user's eye area during the user's use of the display device; inputting the fused eye features and the external environment features into a service prediction model trained according to the training method of the present disclosure; using the trained service prediction model to output the identified target identity of the user based on the human eye static features included in the fused eye features; obtaining target viewing preference information corresponding to the target identity; inputting the target viewing preference information into the trained service prediction model; using the trained service prediction model to output the inferred cause of the user's eye movement based on the human eye motion features, the target viewing preference information and the external environment features included in the fused eye features; and predicting and executing display device services based on the cause of the user's eye movement.
[0012] Optionally, the static features of the human eye include at least one of the following items: features of the iris of the human eye, features of the pupil of the human eye, and features of the edge of the eyeball of the human eye; the motion features of the human eye include at least one of the following items: gaze point position, features of changes in pupil diameter of the human eye, features of the movement rate of the human eye, and features of the blinking frequency of the human eye; the target viewing preference information includes at least one of the following items: content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; the external environment features include light intensity features and / or environmental noise features.
[0013] Optionally, obtaining the target viewing preference information corresponding to the target identity includes: obtaining the pre-stored target viewing preference information corresponding to the target identity; or obtaining the target viewing preference information based on a historical operation record of a user with the target identity on the display device.
[0014] According to a third aspect of an embodiment of the present disclosure, a training device for a service prediction model is provided, comprising: a sample acquisition module, configured to acquire a fused feature sample set, wherein each fused feature sample comprises a human eye static feature sub-sample, a human eye motion feature sub-sample, an external environment feature sub-sample, a user viewing preference sub-sample corresponding to the user, and a real identity label of the corresponding user and a real reason label for the user's eye movement, wherein the human eye static feature sub-sample and the human eye motion feature sub-sample are samples obtained by collecting eye features of the user in the process of the user using the display device, the external environment feature sub-sample is a sample obtained by collecting external environment features in the process of the user using the display device, and the user viewing preference sub-sample is a sample obtained by collecting features of content displayed on the display device in the process of the user using the display device; sample input A module is configured to input the fused feature sample into the service prediction model to obtain the predicted identity of the corresponding user and the predicted reason for the user's eye movement, wherein the service prediction model obtains the predicted identity of the corresponding user based on the human eye static feature sub-sample, and the service prediction model obtains the predicted reason for the user's eye movement of the corresponding user based on the human eye movement feature sub-sample, the external environment feature sub-sample and the user viewing preference sub-sample, wherein the obtained predicted identity of the corresponding user and the predicted reason for the user's eye movement are used to predict and execute the display device service; a calculation module is configured to calculate the loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement; a parameter adjustment module is configured to train the service prediction model by adjusting the parameters of the service prediction model according to the loss.
[0015] Optionally, the calculation module is configured to: calculate identity loss based on the predicted identity and the real identity label; calculate eye movement loss based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement; calculate the loss based on the identity loss and the eye movement loss.
[0016] Optionally, the training device also includes: a pre-training module, configured to pre-train the original service prediction model to obtain a pre-trained service prediction model, replace part of the original layers contained in the pre-trained service prediction model with new layers to obtain a new service prediction model; the sample input module is configured to input the fused feature sample into the new service prediction model; the parameter adjustment module is configured to adjust the parameters of other layers in the new service prediction model according to the loss while freezing the parameters of at least part of the pre-trained layers in the new service prediction model.
[0017] Optionally, the new layer corresponds to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
[0018] Optionally, the calculation module is configured to: calculate identity loss based on the predicted identity and the real identity label; calculate eye movement loss based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement; and calculate the weighted sum of the identity loss and the eye movement loss as the loss by using the first weight parameter as the weight of the identity loss and the second weight parameter as the weight of the eye movement loss.
[0019] Optionally, the static features of the human eye include at least one of the following items: features of the iris of the human eye, features of the pupil of the human eye, and features of the edge of the eyeball of the human eye; the motion features of the human eye include at least one of the following items: gaze point position, features of changes in pupil diameter of the human eye, features of the movement rate of the human eye, and features of the frequency of blinking of the human eye; the user viewing preferences include at least one of the following items: content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; the external environment features include light intensity features and / or environmental noise features.
[0020] According to a fourth aspect of an embodiment of the present disclosure, there is provided a service prediction apparatus, which is applied to a display device, comprising: a feature acquisition module, configured to acquire a user's fused eye features and external environment features, wherein the fused eye features include human eye static features and human eye motion features, wherein the human eye static features and the human eye motion features are obtained by extracting features from the user's eye area during the user's use of the display device; a feature input module, configured to input the fused eye features and the external environment features into a trained service prediction model according to the present disclosure; an identity recognition module, configured to utilize the trained service prediction model to obtain a user's fused eye features and external environment features based on the user's The fused eye feature contains the static features of the human eye, and outputs the identified target identity of the user; a preference information acquisition module is configured to obtain target viewing preference information corresponding to the target identity; a preference information input module is configured to input the target viewing preference information into the trained service prediction model; a cause inference module is configured to use the trained service prediction model to output the inferred cause of the user's eye movement based on the human eye movement features contained in the fused eye feature, the target viewing preference information and the external environment features; an estimation module is configured to estimate and execute display device services based on the cause of the user's eye movement.
[0021] Optionally, the static features of the human eye include at least one of the following items: features of the iris of the human eye, features of the pupil of the human eye, and features of the edge of the eyeball of the human eye; the motion features of the human eye include at least one of the following items: gaze point position, features of changes in pupil diameter of the human eye, features of the movement rate of the human eye, and features of the blinking frequency of the human eye; the target viewing preference information includes at least one of the following items: content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; the external environment features include light intensity features and / or environmental noise features.
[0022] Optionally, the preference information acquisition module is configured to: acquire the pre-stored target viewing preference information corresponding to the target identity; or acquire the target viewing preference information based on a historical operation record of a user with the target identity on the display device.
[0023] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a training method for a service prediction model according to the present disclosure, or to implement a service prediction method according to the present disclosure.
[0024] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute a training method for a service prediction model according to the present disclosure, or to execute a service prediction method according to the present disclosure.
[0025] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the training method of the service prediction model according to the present disclosure, or implements the service prediction method according to the present disclosure.
[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0027] In the present disclosure, when estimating services, the characteristics of multiple dimensions such as human eye static characteristics, human eye movement characteristics, external environment characteristics, and user viewing preferences are comprehensively considered, which can ensure a more comprehensive and accurate analysis of the causes of human eye movement, and thus ensure a more accurate judgment of user needs. Therefore, the service estimation method disclosed in the present disclosure can effectively reduce the probability of misjudgment of user needs based only on a single human eye feature, and can ensure that users are provided with services that truly meet their actual needs.
[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0030] Figure 1 is a flowchart illustrating a training method of a service prediction model according to an exemplary embodiment of the present disclosure;
[0031] Figure 2 is a schematic diagram showing a process of performing feature fusion according to an exemplary embodiment of the present disclosure;
[0032] Figure 3 is a schematic diagram showing the overall process of training a service estimation model according to an exemplary embodiment of the present disclosure;
[0033] Figure 4 is a schematic diagram showing a flow chart of supervised fine-tuning of a service estimation model according to an exemplary embodiment of the present disclosure;
[0034] Figure 5 is a flowchart illustrating a service estimation method according to an exemplary embodiment of the present disclosure;
[0035] Figure 6 is a schematic diagram showing a result of inferring a cause of a change in human eyes according to an exemplary embodiment of the present disclosure;
[0036] Figure 7 is a schematic diagram showing estimation and execution of a "smart brightness adjustment" service according to an exemplary embodiment of the present disclosure;
[0037] Figure 8 is a schematic diagram showing estimation and execution of a “smart content recommendation” service according to an exemplary embodiment of the present disclosure;
[0038] Fig. 9 is a schematic diagram showing estimation and execution of a “smart scaling display size” service according to an exemplary embodiment of the present disclosure;
[0039] Fig.10 is a schematic diagram showing an estimation and execution of a "smart reminder suggestion" service according to an exemplary embodiment of the present disclosure;
[0040] Fig.11 is a schematic diagram showing estimation and execution of a “smart focus tracking” service according to an exemplary embodiment of the present disclosure;
[0041] Fig.12 is a schematic diagram showing estimation and execution of a “smart adjustment of 3D display effect” service according to an exemplary embodiment of the present disclosure;
[0042] Fig.13 is a schematic diagram showing personalized settings of a display device based on single user identity recognition according to an exemplary embodiment of the present disclosure;
[0043] Fig.14 is a schematic diagram showing personalized settings of a display device based on multi-user identity recognition according to an exemplary embodiment of the present disclosure;
[0044] Fig.15 is a schematic diagram showing an architecture for estimating and executing services using a service estimation model according to an exemplary embodiment of the present disclosure;
[0045] Fig.16 is a block diagram illustrating a training apparatus for a service prediction model according to an exemplary embodiment of the present disclosure;
[0046] Fig.17 is a block diagram showing a service estimation apparatus according to an exemplary embodiment of the present disclosure;
[0047] Fig.18 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0048] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0050] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0051] In the fields of smart home and smart office, users have increasing requirements for personalized and intelligent display devices. Specifically, users expect display devices to actively sense their attention, emotional state and visual needs, and implement functions such as automatic brightness adjustment and intelligent content recommendation, thereby optimizing the viewing experience.
[0052] With the continuous development of the digitalization of the biological world, the research on human senses is also gradually deepening. Among the many senses of the human body, the research on the human eye is particularly important. This is because the human eye can provide rich biofeedback information, including light adaptation, attention, emotions, cognitive status, etc. Using this information, the server can understand the needs of users more comprehensively, and then provide users with more satisfactory services.
[0053] At present, deep learning technology has been widely used in computer vision, natural language processing and other fields. Among them, large deep learning models, such as the Transformer model, perform particularly well in processing complex data. Its self-attention mechanism allows the model to assign different attention weights to different parts of the input, which helps to process multi-dimensional eye information.
[0054] In the related technologies, the human eye features used in human eye analysis technology are relatively single, and there is a lack of accuracy in judging user needs based on human eye features, especially in complex external environments, where the state of the human eye is easily affected by changes in external light, emotional fluctuations, and external foreign body stimulation. At this time, it is easy to misjudge user needs based solely on a single human eye feature, which may make it difficult to accurately provide users with services that meet their actual needs.
[0055] In addition, current human eye analysis technology faces limitations in multi-user environments, that is, most current methods are only applicable to single-user scenarios and are difficult to meet the needs of personalized settings in multi-user scenarios. Specifically, different users' subjective adaptability to display device brightness varies due to age, gender, and personal habits; different users also have different preferences for content, etc. When multiple users operate the same device, simply recording the operations of all users is difficult to accurately map the needs of different users.
[0056] In order to solve the above problems existing in the related technologies, the training method of the service prediction model, the service prediction method and the related equipment provided by the present disclosure comprehensively consider the characteristics of multiple dimensions such as the static characteristics of the human eye, the movement characteristics of the human eye, the characteristics of the external environment, and the viewing preferences of the user when estimating the service, which can ensure a more comprehensive and accurate analysis of the reasons for the movement of the human eye, and thus ensure a more accurate judgment of the needs of the user. Therefore, the service prediction method disclosed in the present disclosure can effectively reduce the probability of misjudgment of user needs based only on a single human eye feature, and can ensure that the user is provided with services that can truly meet their actual needs.
[0057] Figure 1 is a flowchart illustrating a training method of a service prediction model according to an exemplary embodiment of the present disclosure.
[0058] Reference Figure 1 In step 101, a fused feature sample set may be obtained, wherein each fused feature sample may include a human eye static feature subsample, a human eye motion feature subsample, an external environment feature subsample, a user viewing preference subsample, etc. corresponding to the user. In addition, the human eye static feature subsample may correspond to the user's real identity label; the human eye motion feature subsample, the external environment feature subsample, and the user viewing preference subsample may correspond to the real reason label for the user's eye movement. The above-mentioned human eye static feature subsample and human eye motion feature subsample may be samples obtained by collecting the user's eye features during the user's use of the display device; the above-mentioned external environment feature subsample is a sample obtained by collecting the external environment features during the user's use of the display device; and the above-mentioned user viewing preference subsample is a sample obtained by collecting the features of the content displayed by the display device during the user's use of the display device.
[0059] It should be noted that in the present disclosure, crowdsourcing (an efficient data collection mode) can be used to collect eye annotation data. In order to build a robust eye analysis model, the diversity and variability of the data set are crucial. The constructed data set should be based on the human eye data on the smart display device, and the constructed data set should also include eye information records under different lighting conditions, different display content, different scaling sizes, different emotional states, and unconstrained postures. Specifically, it is necessary to collect human eye images and annotate identity information. Furthermore, it is necessary to collect videos of human eye changes under different conditions and annotate the reasons for the changes in human eyes.
[0060] For example, when collecting eye annotation data, you need to upload a clear photo of the human eye in a normal state (ground state, without external interference), and the length and width of the eye area in the photo must be no less than 500 pixels. In addition, you also need to upload videos of human eye changes under different light intensities (light intensity range 0.001lux~1000lux), different display contents, and different zoom sizes (the user must be no less than 1.5m away from the display device), annotating the light intensity value, interest in the displayed content, screen zoom size, and emotional state, etc. In addition, it should be noted that multiple users need to be labeled separately.
[0061] Further, in the present disclosure, global features of the eye in human eye images can be extracted through convolutional neural networks (CNNs), for example, but not limited to: pupil features, iris texture features, etc.; long short-term memory networks (LSTMs) can also be used to obtain temporal features of eye movements, for example, but not limited to: eye movement trajectory features, pupil diameter change features, etc. Then, all features can be fused through the attention mechanism and the fully connected layer to generate fused features.
[0062] Figure 2 FIG. 1 is a schematic diagram showing a process of performing feature fusion according to an exemplary embodiment of the present disclosure. Figure 2 , Figure 2 The “original image” in the example may be a static image containing the human eye area, which is mainly used to extract the user’s human eye static feature subsample; Figure 2 The "input frame" in the above may be a key frame in an eye video obtained by shooting the user's eyes when the user uses the display device. These key frames are mainly used to extract the user's eye movement feature subsample, the user's viewing preference subsample, etc. It should be noted that the above "original image" and the above "eye video obtained by shooting the user's eyes" should be obtained by shooting the eye area of the same user.
[0063] According to an exemplary embodiment of the present disclosure, the above-mentioned human eye static feature may include at least one of the following items:
[0064] Features of the human iris, pupil, and eyeball edge;
[0065] The above-mentioned human eye movement characteristics may include at least one of the following items:
[0066] The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes;
[0067] The user viewing preference may include at least one of the following items:
[0068] Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information;
[0069] The above-mentioned external environment characteristics may include light intensity characteristics and / or environmental noise characteristics.
[0070] In step 102, the fused feature sample can be input into the service prediction model to obtain the predicted identity of the corresponding user and the predicted reason for the user's eye movement. The service prediction model can obtain the predicted identity of the corresponding user based on the human eye static feature subsample; the service prediction model can obtain the predicted reason for the user's eye movement of the corresponding user based on the human eye movement feature subsample, the external environment feature subsample and the user viewing preference subsample. The obtained predicted identity of the corresponding user and the predicted reason for the user's eye movement can be used to predict and execute the display device service.
[0071] It should be noted that the above-mentioned "prediction identity" can represent the possibility of each classification predicted by the model, which can be a vector or array containing the prediction probabilities of each category after the model predicts the classification of the sample; the above-mentioned "prediction cause" can represent the possibility of each cause predicted by the model, which can be a vector or array containing the prediction probabilities of each cause after the model predicts the cause of the sample.
[0072] In the present disclosure, ViT can be selected as the basic model. ViT is a Transformer-based deep learning model that is widely used in computer vision tasks. Unlike traditional convolutional neural networks, ViT can divide the image into a fixed number of patches and encode these patches as linear inputs, which are processed by the Transformer's multi-headed self-attention mechanism. ViT can capture global relationships across the entire image, which helps the model consider the global structure of the head and face when recognizing identities, and capture the state of the entire eye area when judging eye changes. In addition, ViT has good scalability, can adapt to images of different sizes and resolutions, has good adaptability to temporal features, can process the fusion features of image and temporal information, and effectively learn multimodal tasks.
[0073] In step 103, the loss may be calculated based on the predicted identity, the true identity label, the predicted reason for the user's eye movement, and the true reason label for the user's eye movement.
[0074] According to an exemplary embodiment of the present disclosure, the identity loss can be calculated based on the predicted identity and the real identity label; and the eye movement loss can be calculated based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement. Then, the above losses can be calculated based on the identity loss and the eye movement loss.
[0075] In step 104, the service prediction model may be trained by adjusting parameters of the service prediction model according to the loss.
[0076] It should be noted that a visual model that performs well in image classification tasks can be selected as the basic model, for example, Vision Transformer (ViT), ResNet, EfficientNet, etc. In this disclosure, Vision Transformer (ViT) is mainly selected as the basic model. Then, the pre-trained basic model can be fine-tuned on a specific labeled data set. Next, the trained model can be integrated into the smart display device to realize the functions of user identity recognition and inference of the cause of human eye changes.
[0077] According to an exemplary embodiment of the present disclosure, pre-training can also be performed on the original service estimation model to obtain a pre-trained service estimation model, and part of the original layers included in the pre-trained service estimation model can be replaced with new layers to obtain a new service estimation model. Next, the fused feature samples can be input into the new service estimation model. Then, while freezing the parameters of at least a part of the pre-trained layers in the new service estimation model, the parameters of other layers in the new service estimation model can be adjusted according to the loss.
[0078] Specifically, in the present disclosure, ViT can be pre-trained on the celebrity dataset CelebA first, so that it can learn universal feature representations and then be used for downstream tasks, where "universal feature representations" can mean that the pre-trained model can identify the specific location of the eye area in the face image, etc. Then, part of the network structure in the pre-trained model can be frozen to retain the universal features learned during the pre-training process. Next, supervised fine-tuning (Supervised Fine-tuning, which refers to the use of labeled training datasets when fine-tuning) can be performed on the fused annotated dataset using transfer learning, so that the ViT model has the functions of identity recognition and judgment of the cause of human eye movement. Figure 3 FIG. 1 is a schematic diagram showing the overall process of training a service prediction model according to an exemplary embodiment of the present disclosure. Figure 3 ,The training process mainly includes the early pre-training process and the later ,process of supervised fine-tuning on a specific labeled dataset.
[0079] In this way, through pre-training and supervised fine-tuning, it can be ensured that the model is optimized and improved in the direction expected by the user, and then it can be ensured that the trained model can achieve the specific tasks specified by the user, and the autonomy and flexibility of model training are better.
[0080] According to an exemplary embodiment of the present disclosure, the new layer may correspond to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
[0081] According to an exemplary embodiment of the present disclosure, identity loss can be calculated based on the predicted identity and the real identity label; and eye movement loss can be calculated based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement. Then, the weighted sum of the identity loss and the eye movement loss can be calculated as the loss by taking the first weight parameter as the weight of the identity loss and the second weight parameter as the weight of the eye movement loss. Next, while freezing the parameters of at least a portion of the pre-trained layers in the new service estimation model, the first weight parameter and the second weight parameter corresponding to the new layer can be adjusted according to the above-mentioned loss, so as to train the new service estimation model.
[0082] Figure 4 FIG. 1 is a flow chart showing a process of performing supervised fine-tuning on a service prediction model according to an exemplary embodiment of the present disclosure. Figure 4 As mentioned above, appropriate pre-training parameters can be selected to pre-train the ViT model on CelebA so that the model can learn the differences between individuals, recognize and understand different eye information. Then, part of the structure of the pre-trained ViT model can be frozen to retain the common features learned during the pre-training process. Exemplarily, in the present disclosure, the remaining layers except the classification head layer can be frozen, including: input embedding layer, position encoding layer, and multiple Transformer encoding blocks, etc.
[0083] Next, a new task output layer can be defined. That is, for identity recognition tasks and human eye change cause inference tasks, a new fully connected layer (Fully Connected Layer) needs to be defined to adapt to the label space. Among them, the number of output features can be defined as 512, the number of output categories can be defined as 1000, and the initial weights and biases can be set using Gaussian distribution random initialization. It should be noted that in the present disclosure, the "new service estimation model" can be obtained by replacing the classification head layer of the ViT model with a new layer.
[0084] Furthermore, hyperparameters and loss functions may be defined for the new layer. In the present disclosure, the loss function of the newly defined task output layer may be a combined loss function:
[0085] L total =w identity L identiyy +w eye L eye
[0086] Among them, the combined loss L total It can be composed of the weighted sum of the loss function of the identity recognition task and the loss function of the human eye change cause judgment task. identity" can be the aforementioned "first weight parameter for the identity recognition task corresponding to the new layer", "w eye " can be the aforementioned "second weight parameter for the eye movement cause inference task corresponding to the new layer". And, w identity and w eye It can be a priori hyperparameters, such as learning rate, batch size, number of training rounds, etc.
[0087] L identity It can be the loss function of the identity recognition task, L eye The loss function for the task of determining the cause of eye changes can be defined using cross loss:
[0088] L identiry =-∑ i t i logy i
[0089] L eye =-∑ k t k logy k
[0090] Among them, t i and t k is the true value, y i and k is the predicted value. Specifically, t i can represent the actual classification of each sample, that is, the true identity label, and y i can represent the possibility of each classification predicted by the model, and t i and i can be expressed in the form of vectors or arrays. That is, for each sample, t i It can be a vector or array containing the actual classification information, for example, it can be represented by one-hot encoding; i It can be a vector or array containing the predicted probabilities of each category obtained after the model classifies and predicts the sample. k and k , which has the same meaning as t i and i Similar, no further description is given here.
[0091] In this way, after calculating the value of the loss function during each training process, the model parameters can be gradually adjusted based on the value of the loss function using back propagation and gradient descent until the loss function is minimized. That is, the model fine-tuning process is terminated when the calculated value of the loss function no longer decreases.
[0092] It should be noted that after supervised fine-tuning of the model, the fine-tuned model can also be evaluated and tuned. For example, the model can be evaluated using a validation set, and hyperparameters or fine-tuning strategies can be adjusted based on the evaluation results to improve the accuracy and generalization ability of the model; and the fine-tuned model can also be evaluated and optimized using a test set.
[0093] In the present disclosure, the basic model can be pre-trained first so that it can learn a common feature representation for downstream tasks. Furthermore, part of the network structure in the pre-trained model can be frozen to retain the common features learned. Next, supervised fine-tuning can be performed on the pre-trained model on a specific labeled data set, that is, the model can be fine-tuned using a fusion feature sample composed of feature samples of multiple dimensions, so that the model can perform accurate identity recognition and infer the cause of eye changes based on the user's eye feature information and surrounding environment information in actual application scenarios, thereby enabling the display device to better understand user needs, thereby optimizing the personalized settings of the display device and enhancing the user's viewing experience.
[0094] Figure 5 is a flowchart illustrating a service estimation method according to an exemplary embodiment of the present disclosure, which is applied to a display device.
[0095] Reference Figure 5 In step 501, the user's fused eye features and external environment features can be obtained, wherein the fused eye features can include human eye static features and human eye motion features, and the human eye static features and human eye motion features can be obtained by extracting features from the user's eye area when the user uses the display device.
[0096] Specifically, the user image sequence can be captured by a camera, and the captured image sequence can be subjected to basic preprocessing. Next, the static features of the human eye can be extracted through a convolutional neural network, and the motion features of the human eye, that is, the temporal features of the human eye, can be extracted through a long short-term memory network. Then, the attention mechanism and the fully connected layer can be used to fuse the above-mentioned static features of the human eye and the motion features of the human eye to generate fused features. It should be noted that in the present disclosure, the convolutional neural network, the long short-term memory network and the service estimation model can be set in the same software, and the software can be installed in a display device to provide services to users.
[0097] According to an exemplary embodiment of the present disclosure, the above-mentioned static features of the human eye may include at least one of the following items: human eye iris features, human eye pupil features, and human eyeball edge features; the human eye motion features may include at least one of the following items: gaze point position, human eye pupil diameter change features, human eye movement rate features, and human eye blinking frequency features.
[0098] In step 502, the fused eye features and external environment features may be input into a service prediction model trained by the training method of the present disclosure.
[0099] In step 503, the trained service prediction model can be used to output the target identity of the identified user based on the static features of the human eye included in the fused eye features. That is, the trained service prediction model can be used to output which user the identified user is based on the iris features, pupil features, and eyeball edge features included in the fused eye features.
[0100] In step 504, target viewing preference information corresponding to the target identity may be obtained.
[0101] According to an exemplary embodiment of the present disclosure, the target viewing preference information corresponding to the target identity that is pre-stored may be obtained, that is, the target viewing preference information may be pre-set in the display device for each user based on their actual situation; or, the target viewing preference information may be obtained based on the historical operation records of the user with the target identity on the display device. That is, the target viewing preference information may be obtained based on the historical operation data, historical browsing data, etc. of the user with the target identity on the display device.
[0102] In this way, the user can set his / her own viewing preference information according to his / her actual situation, which has good autonomy and flexibility. Alternatively, the user's historical operation record records the user's operation behavior and operation track at the historical moment before the current moment, which can reflect the user's viewing preference to a certain extent. Therefore, by determining the user's viewing preference through the user's historical operation record, the accuracy of the determined viewing preference can be guaranteed.
[0103] According to an exemplary embodiment of the present disclosure, the target viewing preference information may include at least one of the following items:
[0104] Content viewing preference information, screen size preference information, screen brightness preference information, language preference information.
[0105] In step 505, the target viewing preference information may be input into a trained service prediction model.
[0106] In step 506, the trained service prediction model may be used to output the inferred cause of the user's eye movement based on the human eye movement features, the target viewing preference information, and the external environment features contained in the fused eye features.
[0107] Among them, the "external environment characteristics" can be obtained by the environment perception sensor, and the "external environment characteristics" can be but not limited to: external light intensity characteristics and / or environmental noise characteristics. The "target viewing preference information" can also be called "context information". As mentioned above, the "target viewing preference information" can be pre-set by the user in the display device manually, or can be obtained by the display device analyzing the user's historical operation records.
[0108] Furthermore, the Apache Kafka stream processing framework can be used to process real-time data to ensure efficient information integration. In this way, the model can infer the causes of human eye movement after real-time data preprocessing, and then obtain the classification results of the inferred causes of human eye changes. Figure 6 is a schematic diagram showing the inference result of the cause of the change of the human eye according to an exemplary embodiment of the present disclosure. Figure 6 , showing the inference results of the causes of eye changes of three users. Among them, the inference result of the cause of eye change of user 001 is: "display brightness is low"; the inference result of the cause of eye change of user 002 is: "the user is interested in the displayed content"; the inference result of the cause of eye change of user 003 is: "the user has rested or left".
[0109] In step 507, display device services may be estimated and executed based on the cause of the user's eye movement.
[0110] Exemplarily, the service prediction model provided by the present disclosure can predict and execute the following services: "intelligent brightness adjustment", "intelligent content recommendation", "intelligent display size scaling", "intelligent reminder suggestions", "intelligent focus tracking", "intelligent adjustment of 3D display effects", etc.
[0111] Figure 7 2 is a schematic diagram illustrating estimating and executing a “smart brightness adjustment” service according to an exemplary embodiment of the present disclosure.
[0112] Reference Figure 7 ,The estimation service model can adjust the brightness by detecting the user’s eye information and the external environment information.
[0113] Figure 8 is a schematic diagram illustrating estimating and executing a “smart content recommendation” service according to an exemplary embodiment of the present disclosure.
[0114] Reference Figure 8 ,The estimation service model can infer the user’s interest in the currently displayed content based on the user’s eye information and contextual information, and capture the content that the user is interested in real time.
[0115] Fig. 9is a schematic diagram illustrating estimating and executing a “smart scaling display size” service according to an exemplary embodiment of the present disclosure.
[0116] Reference Fig. 9 ,The estimation service model can judge whether the projection scaling size meets the user ,requirements based on the user’s eye information, and thus realize ,adaptive adjustment of the projection size.
[0117] Fig.10 is a schematic diagram showing the estimation and execution of a “smart reminder suggestion” service according to an exemplary embodiment of the present disclosure.
[0118] Reference Fig.10 ,The prediction service model can infer the user state by detecting the user’s eye information and contextual information, and then provide suggestions to the user.
[0119] Fig.11 is a schematic diagram illustrating estimating and executing a “smart focus tracking” service according to an exemplary embodiment of the present disclosure.
[0120] Reference Fig.11 ,The estimation service model can obtain the user’s gaze point by detecting the user’s eye information, and then can enhance the user’s gaze area in combination with the contextual information.
[0121] Fig.12 is a schematic diagram illustrating estimating and executing a “smart adjustment of 3D display effect” service according to an exemplary embodiment of the present disclosure.
[0122] Reference Fig.12 , the estimation service model can adjust the 3D display effect of the 3D naked-eye display screen by detecting the user's eye information.
[0123] It should be noted that the estimation service model provided by the present disclosure can also perform personalized settings of the display device according to different user quantity scenarios. Among them, "different user quantity scenarios" can be specifically divided into "single user scenario" and "multi-user scenario".
[0124] For the "single-user scenario", the display device only needs to load the user's personalized settings based on the model's identity recognition results.
[0125] For the "multi-user scenario", the personalized setting problem in the multi-user scenario can be solved by Non-dominated Sorting Genetic Algorithm II (NSGA-II, a multi-objective optimization algorithm), that is, the NSGA-II algorithm can be used to find the optimal solution for personalized settings.
[0126] Specifically, for a multi-user scenario, the personalized settings of each user may be different. Therefore, in the present disclosure, a non-dominated sorting genetic algorithm 2 (NSGA-II) can be used to solve the multi-user personalized setting problem, and its objective function can be defined as: minf(x) = ∑ i λ i f i (x).
[0127] Among them, λ i is the weight coefficient for weighing each task (e.g., brightness adjustment, screen zoom, content preference, rest reminder), and f i (x) is the objective function of each task, which can be defined as follows:
[0128] Brightness adjustment:
[0129] Screen zoom:
[0130] Recommended content:
[0131] Status reminder:
[0132] Among them, p 1j It can represent the screen brightness to be solved, q 1j It can represent the expected screen brightness of the jth user. By solving the sum of squares of the difference between the two, we can get f 1 (x); p 2j It can represent the scaled size to be solved, q 2j It can represent the expected scaling size of the jth user. By solving the sum of squares of the difference between the two, we can get f 2 (x); q 2j It can represent the expected scaling size of the jth user. By solving the sum of squares of the difference between the two, we can get f 2 (x); p 3j It can represent the preference content to be solved, q 3j It can represent the expected preference content of the jth user, and f can be obtained by adding the cosine similarity scores. 3 (x); p 4j It can indicate the state reminder time to be solved, q 4j It can represent the expected state reminder time of the jth user, and f can be obtained by summing the absolute differences. 4 (x).
[0133] Next, NSGA-II can optimize the objective function through fast non-dominated sorting and crowding distance calculation to minimize the objective function, and then obtain a set of optimal solutions for personalized settings.
[0134] It should be noted that the service prediction model provided by the present disclosure can also be used in identity recognition scenarios. When the user identity is successfully recognized, the display device can automatically load the user's personalized settings. For multi-user scenarios, the NSGA-II algorithm can also be used to find the optimal solution for personalized settings.
[0135] Fig.13 is a schematic diagram showing a personalized setting of a display device based on single user identity recognition according to an exemplary embodiment of the present disclosure. Fig.13 ,The results of the personalized settings for a single user are: "Adjust the display brightness to 70%", "Recommend sports programs".
[0136] Fig.14 is a schematic diagram showing personalized settings of a display device based on multi-user identity recognition according to an exemplary embodiment of the present disclosure. Fig.14 ,The results of multi-user personalized settings are: "Display brightness adjusted to 68%", "Recommend sports events programs".
[0137] Fig.15 FIG. 1 is a schematic diagram showing an architecture for estimating and executing a service using a service estimation model according to an exemplary embodiment of the present disclosure. Fig.15 When estimating services, it is necessary to obtain the user's eye information, external environment information, and user viewing preference information, namely context information. Then, the service estimation model can infer the cause of eye movement based on the above information, and then estimate and execute services that meet the user's current needs based on the inference results.
[0138] Specifically, as mentioned above, the service estimation model disclosed in the present invention mainly has the following functions:
[0139] Comprehensively process multi-dimensional eye information, including pupil size, blinking frequency, gaze point location, etc.; interpret eye information, analyze and infer the reasons for changes in user eye information, and understand user behavior; learn user eye features to realize user identity recognition, suitable for multi-user scenarios; integrate external environment light intensity information and eye information to realize intelligent brightness adjustment; adjust the screen zoom ratio according to eye information to realize adaptive adjustment of screen size; infer user interests and needs based on eye information to realize personalized content recommendation; analyze user emotional state based on eye information to realize status reminder function.
[0140] The service prediction model provided by the present disclosure is trained by using multi-dimensional feature samples, so that it can judge the reasons for changes in the user's eyes by combining features of different dimensions, and then accurately provide users with services that meet their actual needs. Exemplarily, the services provided can be intelligent adjustment of display device brightness and intelligent adjustment of zoom size to meet the personalized needs of different users; the services provided can also be intelligent recommendation of content to avoid the phenomenon of mis-push caused by content recommendation based on a single feature only; the services provided can also be targeted at multi-user scenarios to make up for the defects of inaccurate personalized services in multi-user scenarios in related technologies.
[0141] It should be noted that the service estimation method provided by the present invention can be applied to display devices, which may include but are not limited to: smart display devices such as televisions, projectors, 3D naked-eye displays; or other devices other than smart display devices, for example, may be but are not limited to: smart home control devices, smart teaching systems, etc.
[0142] According to exemplary embodiments of the present disclosure, the model and method provided by the present disclosure are also applicable to other biometric features, such as, but not limited to, facial expressions, gestures, heart rate, etc. Specifically, these features can be captured by sensors or cameras, and their changes can be analyzed and identified by the model. Alternatively, multimodal analysis can be performed in combination with other biometric features other than the human eye to provide a more comprehensive and accurate user portrait, etc.
[0143] Fig.16 16 is a block diagram illustrating a training device 1600 for a service prediction model according to an exemplary embodiment of the present disclosure.
[0144] Reference Fig.16 The training device 1600 of the service prediction model may include a sample acquisition module 1601, a sample input module 1602, a calculation module 1603 and a parameter adjustment module 1604.
[0145] The sample acquisition module 1601 can obtain a fusion feature sample set, wherein each fusion feature sample can include a human eye static feature subsample, a human eye motion feature subsample, an external environment feature subsample, a user viewing preference subsample, etc. corresponding to the user. In addition, the human eye static feature subsample can correspond to the user's real identity label; the human eye motion feature subsample, the external environment feature subsample and the user viewing preference subsample can correspond to the real reason label of the user's eye movement. The above-mentioned human eye static feature subsample and human eye motion feature subsample can be samples obtained by collecting the user's eye features in the process of the user using the display device; the above-mentioned external environment feature subsample is a sample obtained by collecting the external environment features in the process of the user using the display device; the above-mentioned user viewing preference subsample is a sample obtained by collecting the features of the content displayed by the display device in the process of the user using the display device.
[0146] According to an exemplary embodiment of the present disclosure, the above-mentioned human eye static feature may include at least one of the following items:
[0147] Features of the human iris, pupil, and eyeball edge;
[0148] The above-mentioned human eye movement characteristics may include at least one of the following items:
[0149] The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes;
[0150] The user viewing preference may include at least one of the following items:
[0151] Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information;
[0152] The above-mentioned external environment characteristics may include light intensity characteristics and / or environmental noise characteristics.
[0153] The sample input module 1602 can input the fused feature sample into the service prediction model to obtain the predicted identity of the corresponding user and the predicted reason for the user's eye movement. The service prediction model can obtain the predicted identity of the corresponding user based on the human eye static feature subsample; the service prediction model can obtain the predicted reason for the user's eye movement of the corresponding user based on the human eye movement feature subsample, the external environment feature subsample and the user viewing preference subsample. The obtained predicted identity of the corresponding user and the predicted reason for the user's eye movement can be used to predict and execute the display device service.
[0154] It should be noted that the above-mentioned "prediction identity" can represent the possibility of each classification predicted by the model, which can be a vector or array containing the prediction probabilities of each category after the model predicts the classification of the sample; the above-mentioned "prediction cause" can represent the possibility of each cause predicted by the model, which can be a vector or array containing the prediction probabilities of each cause after the model predicts the cause of the sample.
[0155] In the present disclosure, ViT can be selected as the basic model. ViT is a Transformer-based deep learning model that is widely used in computer vision tasks. Unlike traditional convolutional neural networks, ViT can divide the image into a fixed number of patches and encode these patches as linear inputs, which are processed by the Transformer's multi-headed self-attention mechanism. ViT can capture global relationships across the entire image, which helps the model consider the global structure of the head and face when recognizing identities, and capture the state of the entire eye area when judging eye changes. In addition, ViT has good scalability, can adapt to images of different sizes and resolutions, has good adaptability to temporal features, can process the fusion features of image and temporal information, and effectively learn multimodal tasks.
[0156] The calculation module 1603 can calculate the loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement.
[0157] According to an exemplary embodiment of the present disclosure, the calculation module 1603 may calculate the identity loss based on the predicted identity and the real identity label; and the calculation module 1603 may calculate the eye movement loss based on the predicted cause of the user's eye movement and the real cause label of the user's eye movement. Then, the calculation module 1603 may calculate the above losses based on the identity loss and the eye movement loss.
[0158] The parameter adjustment module 1604 may train the service prediction model by adjusting the parameters of the service prediction model according to the loss.
[0159] It should be noted that a visual model that performs well in image classification tasks can be selected as the basic model, for example, Vision Transformer (ViT), ResNet, EfficientNet, etc. In this disclosure, Vision Transformer (ViT) is mainly selected as the basic model. Then, the pre-trained basic model can be fine-tuned on a specific labeled data set. Next, the trained model can be integrated into the smart display device to realize the functions of user identity recognition and inference of the cause of human eye changes.
[0160] According to an exemplary embodiment of the present disclosure, the training device 1600 may further include a pre-training module.
[0161] The pre-training module can also perform pre-training on the original service estimation model to obtain a pre-trained service estimation model, and can replace part of the original layers contained in the pre-trained service estimation model with new layers to obtain a new service estimation model. Next, the sample input module 1602 can input the fused feature sample into the new service estimation model. Then, while freezing the parameters of at least a part of the pre-trained layers in the new service estimation model, the parameter adjustment module 1604 can adjust the parameters of other layers in the new service estimation model according to the loss.
[0162] In this way, through pre-training and supervised fine-tuning, it can be ensured that the model is optimized and improved in the direction expected by the user, and then it can be ensured that the trained model can achieve the specific tasks specified by the user, and the autonomy and flexibility of model training are better.
[0163] According to an exemplary embodiment of the present disclosure, the new layer may correspond to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
[0164] According to an exemplary embodiment of the present disclosure, the calculation module 1603 can calculate the identity loss based on the predicted identity and the real identity label; and the calculation module 1603 can calculate the eye movement loss based on the predicted reason for the user's eye movement and the real reason label for the user's eye movement. Then, the calculation module 1603 can calculate the weighted sum of the identity loss and the eye movement loss as the above-mentioned loss by using the first weight parameter as the weight of the identity loss and the second weight parameter as the weight of the eye movement loss. Next, the parameter adjustment module 1604 can adjust the first weight parameter and the second weight parameter corresponding to the new layer according to the above-mentioned loss while freezing the parameters of at least a part of the pre-trained layers in the new service estimation model, thereby training the new service estimation model.
[0165] In the present disclosure, the basic model can be pre-trained first so that it can learn a common feature representation for downstream tasks. Furthermore, part of the network structure in the pre-trained model can be frozen to retain the common features learned. Next, supervised fine-tuning can be performed on the pre-trained model on a specific labeled data set, that is, the model can be fine-tuned using a fusion feature sample composed of feature samples of multiple dimensions, so that the model can perform accurate identity recognition and infer the cause of eye changes based on the user's eye feature information and surrounding environment information in actual application scenarios, thereby enabling the display device to better understand user needs, thereby optimizing the personalized settings of the display device and enhancing the user's viewing experience.
[0166] Fig.17 is a block diagram illustrating a service estimation apparatus according to an exemplary embodiment of the present disclosure.
[0167] Reference Fig.17 The service estimation device 1700 may include a feature acquisition module 1701, a feature input module 1702, an identity recognition module 1703, a preference information acquisition module 1704, a preference information input module 1705, a cause inference module 1706 and an estimation module 1707.
[0168] The feature acquisition module 1701 can acquire the user's fused eye features and external environment features, wherein the fused eye features may include static features of the human eye and motion features of the human eye, and the static features of the human eye and the motion features of the human eye may be obtained by extracting features from the user's eye area during the user's use of the display device.
[0169] Specifically, the user image sequence can be captured by a camera, and the captured image sequence can be subjected to basic preprocessing. Next, the static features of the human eye can be extracted through a convolutional neural network, and the motion features of the human eye, that is, the temporal features of the human eye, can be extracted through a long short-term memory network. Then, the attention mechanism and the fully connected layer can be used to fuse the above-mentioned static features of the human eye and the motion features of the human eye to generate fused features. It should be noted that in the present disclosure, the convolutional neural network, the long short-term memory network and the service estimation model can be set in the same software, and the software can be installed in a display device to provide services to users.
[0170] According to an exemplary embodiment of the present disclosure, the above-mentioned static features of the human eye may include at least one of the following items: human eye iris features, human eye pupil features, and human eyeball edge features; the human eye motion features may include at least one of the following items: gaze point position, human eye pupil diameter change features, human eye movement rate features, and human eye blinking frequency features.
[0171] The feature input module 1702 can input the fused eye features and external environment features into the service prediction model trained by the training method of the present disclosure.
[0172] The identity recognition module 1703 can use the trained service prediction model to output the target identity of the identified user based on the static features of the human eye included in the fusion eye features. That is, the trained service prediction model can be used to output which user the identified user is based on the iris features, pupil features, and eyeball edge features included in the fusion eye features.
[0173] The preference information acquisition module 1704 may acquire target viewing preference information corresponding to the target identity.
[0174] According to an exemplary embodiment of the present disclosure, the preference information acquisition module 1704 may acquire pre-stored target viewing preference information corresponding to the target identity, that is, the target viewing preference information may be pre-set in the display device for each user based on their actual situation; or, the target viewing preference information may be acquired based on the historical operation records of the user with the target identity on the display device. That is, the target viewing preference information may be acquired based on the historical operation data, historical browsing data, etc. of the user with the target identity on the display device.
[0175] In this way, the user can set his / her own viewing preference information according to his / her actual situation, which has good autonomy and flexibility. Alternatively, the user's historical operation record records the user's operation behavior and operation track at the historical moment before the current moment, which can reflect the user's viewing preference to a certain extent. Therefore, by determining the user's viewing preference through the user's historical operation record, the accuracy of the determined viewing preference can be guaranteed.
[0176] According to an exemplary embodiment of the present disclosure, the target viewing preference information may include at least one of the following items:
[0177] Content viewing preference information, screen size preference information, screen brightness preference information, language preference information.
[0178] The preference information input module 1705 may input the target viewing preference information into the trained service prediction model.
[0179] The cause inference module 1706 can use the trained service prediction model to output the inferred cause of the user's eye movement based on the human eye movement characteristics, target viewing preference information and external environment characteristics contained in the fusion eye features.
[0180] Among them, the "external environment characteristics" can be obtained by the environment perception sensor, and the "external environment characteristics" can be but not limited to: external light intensity characteristics and / or environmental noise characteristics. The "target viewing preference information" can also be called "context information". As mentioned above, the "target viewing preference information" can be pre-set by the user in the display device manually, or can be obtained by the display device analyzing the user's historical operation records.
[0181] Furthermore, the Apache Kafka stream processing framework can be used to process real-time data to ensure efficient information integration. In this way, the model can infer the causes of human eye movement after real-time data preprocessing, and then obtain the classification results of the inferred causes of human eye changes.
[0182] The estimation module 1707 can estimate and execute display device services based on the reasons for the user's eye movement. Exemplarily, the service estimation model provided by the present disclosure can estimate and execute the following services: "intelligent brightness adjustment", "intelligent content recommendation", "intelligent display size scaling", "intelligent reminder suggestions", "intelligent focus tracking", "intelligent adjustment of 3D display effects", etc.
[0183] The service prediction model provided by the present disclosure is trained by using multi-dimensional feature samples, so that it can judge the reasons for changes in the user's eyes by combining features of different dimensions, and then accurately provide users with services that meet their actual needs. Exemplarily, the services provided can be intelligent adjustment of display device brightness and intelligent adjustment of zoom size to meet the personalized needs of different users; the services provided can also be intelligent recommendation of content to avoid the phenomenon of mis-push caused by content recommendation based on a single feature only; the services provided can also be targeted at multi-user scenarios to make up for the defects of inaccurate personalized services in multi-user scenarios in related technologies.
[0184] Fig.18 is a block diagram illustrating an electronic device 1800 according to an exemplary embodiment of the present disclosure.
[0185] Reference Fig.18 The electronic device 1800 includes at least one memory 1801 and at least one processor 1802, wherein the at least one memory 1801 stores instructions, and when the instructions are executed by the at least one processor 1802, a training method for a service prediction model or a service prediction method according to an exemplary embodiment of the present disclosure is executed.
[0186] As an example, the electronic device 1800 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 1800 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1800 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0187] In the electronic device 1800, the processor 1802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0188] The processor 1802 may execute instructions or codes stored in the memory 1801, wherein the memory 1801 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0189] The memory 1801 may be integrated with the processor 1802, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1801 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1801 and the processor 1802 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1802 can read files stored in the memory.
[0190] In addition, the electronic device 1800 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1800 may be connected to each other via a bus and / or a network.
[0191] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the above-mentioned service estimation model or the service estimation method. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0192] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, which, when executed by a processor, implements the training method of the service prediction model or the service prediction method according to the present disclosure.
[0193] According to the training method, service prediction method and related equipment of the service prediction model disclosed in the present invention, when estimating services, the characteristics of multiple dimensions such as human eye static characteristics, human eye movement characteristics, external environment characteristics, user viewing preferences, etc. are comprehensively considered, which can ensure a more comprehensive and accurate analysis of the reasons for human eye movement, and thus ensure a more accurate judgment of user needs. Therefore, the service prediction method disclosed in the present invention can effectively reduce the probability of misjudgment of user needs based only on a single human eye feature, and can ensure that users are provided with services that truly meet their actual needs.
[0194] According to the exemplary embodiments of the present disclosure, through pre-training and supervised fine-tuning, it can be ensured that the model is optimized and improved in the direction expected by the user, and then it can be ensured that the trained model can achieve the specific tasks specified by the user, and the autonomy and flexibility of model training are better.
[0195] According to the exemplary embodiments of the present disclosure, users can arbitrarily set their own viewing preference information according to their actual situation, which has good autonomy and flexibility. Alternatively, the user's historical operation record records the user's operation behavior and operation trajectory at historical moments before the current moment, which can reflect the user's viewing preference to a certain extent. Therefore, by determining the user's viewing preference through the user's historical operation record, the accuracy of the determined viewing preference can be guaranteed.
[0196] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0197] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A training method for a service estimation model, It is characterized in that include: Obtain a fused feature sample set, wherein each fused feature sample comprises a human eye static feature subsample, a human eye motion feature subsample, an external environment feature subsample, a user viewing preference subsample of the corresponding user, and a real identity label of the corresponding user and a real reason label of the user's eye movement, wherein the human eye static feature subsample and the human eye motion feature subsample are samples obtained by collecting eye features of the user in the process of the user using the display device, the external environment feature subsample is a sample obtained by collecting external environment features in the process of the user using the display device, and the user viewing preference subsample is a sample obtained by collecting features of content displayed by the display device in the process of the user using the display device; Inputting the fused feature sample into the service prediction model to obtain the predicted identity of the corresponding user and the predicted reason for the user's eye movement, wherein the service prediction model obtains the predicted identity of the corresponding user based on the human eye static feature subsample, and the service prediction model obtains the predicted reason for the user's eye movement of the corresponding user based on the human eye movement feature subsample, the external environment feature subsample and the user viewing preference subsample, wherein the obtained predicted identity of the corresponding user and the predicted reason for the user's eye movement are used to predict and execute the display device service; Calculating a loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement; The service prediction model is trained by adjusting parameters of the service prediction model according to the loss.
2. The training method according to claim 1, It is characterized in that The calculating the loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement includes: Calculating identity loss based on the predicted identity and the true identity label; Calculating an eye movement loss based on the predicted cause of the user's eye movement and the true cause label of the user's eye movement; Based on the identity loss and the eye movement loss, the loss is calculated.
3. The training method according to claim 1, It is characterized in that The training method further comprises: Pre-training the original service prediction model to obtain a pre-trained service prediction model, and replacing part of the original layers included in the pre-trained service prediction model with new layers to obtain a new service prediction model; The step of inputting the fused feature sample into the service prediction model comprises: Inputting the fused feature sample into the new service prediction model; The adjusting the parameters of the service prediction model according to the loss includes: While freezing parameters of at least a portion of pre-trained layers in the new service estimation model, parameters of other layers in the new service estimation model are adjusted according to the loss.
4. The training method according to claim 3, It is characterized in that The new layer corresponds to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
5. The training method according to claim 4, It is characterized in that The calculating the loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement includes: Calculating identity loss based on the predicted identity and the true identity label; Calculating an eye movement loss based on the predicted cause of the user's eye movement and the true cause label of the user's eye movement; By taking the first weight parameter as the weight of the identity loss and taking the second weight parameter as the weight of the eye movement loss, a weighted sum of the identity loss and the eye movement loss is calculated as the loss.
6. The training method according to claim 1, It is characterized in that The human eye static feature includes at least one of the following items: Features of the human iris, pupil, and eyeball edge; The human eye movement feature includes at least one of the following items: The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes; The user viewing preference includes at least one of the following items: Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; The external environment characteristics include light intensity characteristics and / or environmental noise characteristics.
7. A service estimation method, applied to a display device, It is characterized in that include: Acquire fused eye features of the user and external environment features, wherein the fused eye features include static features of human eyes and motion features of human eyes, wherein the static features of human eyes and the motion features of human eyes are obtained by extracting features from the eye region of the user when the user uses the display device; Inputting the fused eye feature and the external environment feature into a service prediction model trained by the training method according to any one of claims 1 to 6; Outputting the identified target identity of the user based on the human eye static features contained in the fused eye features using the trained service prediction model; Acquire target viewing preference information corresponding to the target identity; inputting the target viewing preference information into the trained service prediction model; Outputting the inferred cause of the user's eye movement using the trained service prediction model based on the human eye movement feature contained in the fused eye feature, the target viewing preference information and the external environment feature; Based on the cause of the user's eye movement, a display device service is estimated and executed.
8. The service estimation method according to claim 7, It is characterized in that The human eye static feature includes at least one of the following items: Features of the human iris, pupil, and eyeball edge; The human eye movement feature includes at least one of the following items: The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes; The target viewing preference information includes at least one of the following items: Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; The external environment characteristics include light intensity characteristics and / or environmental noise characteristics.
9. The service estimation method according to claim 7, It is characterized in that The acquiring target viewing preference information corresponding to the target identity includes: Acquire the target viewing preference information corresponding to the target identity that is pre-stored; Alternatively, the target viewing preference information is acquired based on a historical operation record of a user with the target identity on the display device.
10. A training device for a service prediction model, It is characterized in that include: The sample acquisition module is configured to acquire a fused feature sample set, wherein each fused feature sample comprises a human eye static feature subsample, a human eye motion feature subsample, an external environment feature subsample, a user viewing preference subsample of the corresponding user, and a real identity label of the corresponding user and a real reason label for the user's eye movement, wherein the human eye static feature subsample and the human eye motion feature subsample are samples obtained by collecting eye features of the user in the process of the user using the display device, the external environment feature subsample is a sample obtained by collecting external environment features in the process of the user using the display device, and the user viewing preference subsample is a sample obtained by collecting features of content displayed by the display device in the process of the user using the display device; A sample input module is configured to input the fused feature sample into the service prediction model to obtain a predicted identity of the corresponding user and a predicted reason for the user's eye movement, wherein the service prediction model obtains the predicted identity of the corresponding user based on the human eye static feature subsample, and the service prediction model obtains the predicted reason for the user's eye movement of the corresponding user based on the human eye movement feature subsample, the external environment feature subsample and the user viewing preference subsample, wherein the obtained predicted identity of the corresponding user and the predicted reason for the user's eye movement are used to predict and execute the display device service; a calculation module configured to calculate a loss based on the predicted identity, the real identity label, the predicted reason for the user's eye movement, and the real reason label for the user's eye movement; The parameter adjustment module is configured to train the service prediction model by adjusting the parameters of the service prediction model according to the loss.
11. The training device according to claim 10, It is characterized in that The computing module is configured as follows: Calculating identity loss based on the predicted identity and the true identity label; Calculating an eye movement loss based on the predicted cause of the user's eye movement and the true cause label of the user's eye movement; Based on the identity loss and the eye movement loss, the loss is calculated.
12. The training device according to claim 10, It is characterized in that The training device also includes: A pre-training module is configured to pre-train the original service estimation model to obtain a pre-trained service estimation model, and replace part of the original layers included in the pre-trained service estimation model with new layers to obtain a new service estimation model; The sample input module is configured to: Inputting the fused feature sample into the new service prediction model; The parameter adjustment module is configured as follows: While freezing parameters of at least a portion of pre-trained layers in the new service estimation model, parameters of other layers in the new service estimation model are adjusted according to the loss.
13. The training device according to claim 12, It is characterized in that The new layer corresponds to a first weight parameter for an identity recognition task and a second weight parameter for an eye movement cause inference task.
14. The training device according to claim 13, It is characterized in that The computing module is configured as follows: Calculating identity loss based on the predicted identity and the true identity label; Calculating an eye movement loss based on the predicted cause of the user's eye movement and the true cause label of the user's eye movement; By taking the first weight parameter as the weight of the identity loss and taking the second weight parameter as the weight of the eye movement loss, a weighted sum of the identity loss and the eye movement loss is calculated as the loss.
15. The training device according to claim 10, It is characterized in that The human eye static feature includes at least one of the following items: Features of the human iris, pupil, and eyeball edge; The human eye movement feature includes at least one of the following items: The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes; The user viewing preference includes at least one of the following items: Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; The external environment characteristics include light intensity characteristics and / or environmental noise characteristics.
16. A service estimation device, applied to a display device, It is characterized in that include: A feature acquisition module, configured to acquire fused eye features of a user and external environment features, wherein the fused eye features include static features of human eyes and motion features of human eyes, wherein the static features of human eyes and the motion features of human eyes are obtained by extracting features from the eye region of the user when the user uses the display device; A feature input module, configured to input the fused eye feature and the external environment feature into a trained service prediction model as claimed in any one of claims 10 to 15; an identity recognition module configured to output the identified target identity of the user based on the human eye static features contained in the fused eye features by using the trained service prediction model; A preference information acquisition module, configured to acquire target viewing preference information corresponding to the target identity; a preference information input module, configured to input the target viewing preference information into the trained service prediction model; a cause inference module configured to output the inferred cause of the user's eye movement based on the human eye movement feature, the target viewing preference information and the external environment feature contained in the fused eye feature by using the trained service prediction model; The estimation module is configured to estimate and execute display device services based on the cause of the user's eye movement.
17. The service estimation device according to claim 16, It is characterized in that The human eye static feature includes at least one of the following items: Features of the human iris, pupil, and eyeball edge; The human eye movement feature includes at least one of the following items: The location of the gaze point, the change characteristics of the pupil diameter, the movement rate characteristics of the eyes, and the blinking frequency characteristics of the eyes; The target viewing preference information includes at least one of the following items: Content viewing preference information, screen size preference information, screen brightness preference information, and language preference information; The external environment characteristics include light intensity characteristics and / or environmental noise characteristics.
18. The service estimation device according to claim 16, It is characterized in that The preference information acquisition module is configured to: Acquire the target viewing preference information corresponding to the target identity that is pre-stored; Alternatively, the target viewing preference information is acquired based on a historical operation record of a user with the target identity on the display device.
19. An electronic device, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the service prediction model as described in any one of claims 1 to 6, or to implement the service prediction method as described in any one of claims 7 to 9.
20. A computer-readable storage medium, It is characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the service prediction model as described in any one of claims 1 to 6, or to execute the service prediction method as described in any one of claims 7 to 9.
21. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, it implements the training method of the service prediction model according to any one of claims 1 to 6, or implements the service prediction method according to any one of claims 7 to 9.
Citation Information
Patent Citations
Head mounted display with iris scan profiling
CN103033936A
Model training method and device, data storage method and device and electronic equipment
CN114090535A
Eye gaze control for magnifying user interface
CN114556270A
Expression recognition model training method, device and equipment
CN114882566A
Screen control method and control device of head-mounted display equipment
CN115826243A
