Operation interaction method and device, equipment, medium and program product

Abstract

The invention provides an operation interaction method which can be applied to the technical field of artificial intelligence. The method comprises the following steps: determining pupil trajectory information of a target user; converting the determined pupil trajectory information into a quantum state coding form, and determining a user operation intention predicted for the pupil trajectory information in the quantum state coding form based on a pre-trained intention recognition model; and executing an operation process for realizing the determined user operation intention, and displaying an execution result of the operation process. The invention further provides an operation interaction device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular, to an operation interaction method, apparatus, device, medium, and program product. Background Art

[0002] With the widespread adoption of smart devices, users are able to access a wide range of functions. For example, users can play music, browse web pages, fill out information, and more. However, currently, the only form of user interaction with smart devices is screen tapping, which is relatively simple. Summary of the Invention

[0003] In view of the above problems, the present application provides an operation interaction method, apparatus, device, medium and program product.

[0004] According to a first aspect of the present application, an operation interaction method is provided, comprising:

[0005] Determine pupil trajectory information of the target user;

[0006] Converting the determined pupil trajectory information into a quantum state encoding form, and determining the user operation intention predicted by the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model;

[0007] An operation process for realizing the determined user operation intention is executed, and an execution result of the operation process is displayed.

[0008] Optionally, determining pupil trajectory information of the target user includes:

[0009] Determine the target user's authorization to monitor human eye characteristics;

[0010] monitoring environmental information and the eye characteristics of the target user with authorization from the target user for monitoring the eye characteristics;

[0011] Determine the matching pupil tracking method based on the currently monitored environmental information and human eye characteristics;

[0012] Based on the determined pupil tracking method, pupil trajectory information of the target user is determined.

[0013] Optionally, determining a matching pupil tracking method based on currently monitored environmental information and human eye features includes:

[0014] When it is determined that the degree of change of the monitored environmental information is greater than a first preset degree, and / or the degree of change of the monitored human eye characteristics is greater than a second preset degree, the tracking effects are determined for different pupil tracking methods based on the currently monitored environmental information and human eye characteristics, and the pupil tracking method whose tracking effect meets the preset effect conditions is determined as the matching pupil tracking method.

[0015] Optionally, executing the operation process for realizing the determined user operation intention includes:

[0016] In a case where the operation flow for realizing the determined user operation intention includes at least two parallel operations, during the execution of the operation flow, different parallel operations included in the operation flow are executed in parallel.

[0017] Optionally, executing an operation process for realizing the determined user operation intention and displaying an execution result of the operation process includes:

[0018] Execute the operation process for realizing the determined user operation intention in the background;

[0019] When it is determined that the operation process is completed, the execution result of the operation process is displayed on the front end.

[0020] Optionally, the training method of the intent recognition model includes:

[0021] Determine a training sample set; any sample in the training sample set includes: a sample feature for characterizing pupil trajectory information of the target user, and a sample label for characterizing user operation intention;

[0022] Convert the sample features in the training sample set into quantum state encoding form;

[0023] The intent recognition model is trained based on the sample features in quantum state encoding form and the corresponding sample labels.

[0024] Optionally, determining the user operation intention predicted based on pupil trajectory information in quantum state encoding form based on a pre-trained intention recognition model includes:

[0025] The determined pupil trajectory information is converted into a quantum state encoding form, and based on a pre-trained intention recognition model, the pupil trajectory information for the quantum state encoding form and the user operation intention predicted by at least one of the following are determined: currently displayed content, historical operation records, and historical pupil trajectory information.

[0026] A second aspect of the present application provides an operation interaction device, comprising:

[0027] A trajectory determination unit, configured to determine pupil trajectory information of a target user;

[0028] a prediction unit, configured to convert the determined pupil trajectory information into a quantum state encoding form, and determine the user operation intention predicted based on the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model;

[0029] The execution unit is used to execute the operation process for realizing the determined user operation intention and display the execution result of the operation process.

[0030] Optionally, the trajectory determination unit is configured to:

[0031] Determine the target user's authorization to monitor human eye characteristics;

[0032] monitoring environmental information and the eye characteristics of the target user with authorization from the target user for monitoring the eye characteristics;

[0033] Determine the matching pupil tracking method based on the currently monitored environmental information and human eye characteristics;

[0034] Based on the determined pupil tracking method, pupil trajectory information of the target user is determined.

[0035] Optionally, the trajectory determination unit is configured to:

[0036] When it is determined that the degree of change of the monitored environmental information is greater than a first preset degree, and / or the degree of change of the monitored human eye characteristics is greater than a second preset degree, the tracking effects are determined for different pupil tracking methods based on the currently monitored environmental information and human eye characteristics, and the pupil tracking method whose tracking effect meets the preset effect conditions is determined as the matching pupil tracking method.

[0037] Optionally, the execution unit is configured to:

[0038] In a case where the operation flow for realizing the determined user operation intention includes at least two parallel operations, during the execution of the operation flow, different parallel operations included in the operation flow are executed in parallel.

[0039] Optionally, the execution unit is configured to:

[0040] Execute the operation process for realizing the determined user operation intention in the background;

[0041] When it is determined that the operation process is completed, the execution result of the operation process is displayed on the front end.

[0042] Optionally, the training method of the intent recognition model includes:

[0043] Determine a training sample set; any sample in the training sample set includes: a sample feature for characterizing pupil trajectory information of the target user, and a sample label for characterizing user operation intention;

[0044] Convert the sample features in the training sample set into quantum state encoding form;

[0045] The intent recognition model is trained based on the sample features in quantum state encoding form and the corresponding sample labels.

[0046] Optionally, the prediction unit is configured to:

[0047] The determined pupil trajectory information is converted into a quantum state encoding form, and based on a pre-trained intention recognition model, the pupil trajectory information for the quantum state encoding form and the user operation intention predicted by at least one of the following are determined: currently displayed content, historical operation records, and historical pupil trajectory information.

[0048] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0049] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0050] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0051] One or more of the above embodiments have the following beneficial effects: by performing operation interactions based on the user's pupil trajectory information, a new operation interaction method that is different from screen clicking is provided, thereby improving the diversity of operation interaction forms.

[0052] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0054] Figure 1 A diagram schematically illustrates an application scenario of an operation interaction method according to an embodiment of the present application;

[0055] Figure 2A flowchart of an operation interaction method according to an embodiment of the present application is schematically shown;

[0056] Figure 3 A block diagram schematically illustrates a structure of an operation interaction device according to an embodiment of the present application; and

[0057] Figure 4 A block diagram of an electronic device suitable for implementing an operation interaction method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0058] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0059] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0061] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0062] With the widespread adoption of smart devices, users are able to access a wide range of functions. For example, users can play music, browse web pages, fill out information, and more. However, currently, the only form of user interaction with smart devices is screen tapping, which is relatively simple.

[0063] The embodiments of the present application can provide an operation interaction method, which can specifically determine the user's operation intention based on the user's pupil trajectory information, thereby automatically performing operations to realize the determined user operation intention, realize eye movement operation interaction between the user and the device, and provide a new operation interaction method that is different from screen clicks, thereby improving the diversity of operation interaction forms.

[0064] In this method, machine learning can be used to train an intention recognition model, which can predict and determine the user's operation intention based on the user's pupil trajectory information, thereby improving the accuracy and efficiency of identifying the user's operation intention.

[0065] In this method, considering the continuity of pupil trajectory, the collected user pupil trajectory information can be converted into quantum state coding form, which can make it easier to extract complex features in the pupil trajectory information, improve the training effect of the intention recognition model, and improve the accuracy and efficiency of identifying user operation intentions.

[0066] It should be noted that the methods and devices disclosed in the embodiments of this application can be used in the field of financial technology. For example, for financial-related services or applications, the user's operation intention can be determined based on the user's pupil trajectory information. They can also be applied to any field other than financial technology, such as audio and video processing or virtual reality. The application fields of the methods and devices disclosed in the embodiments of this application are not limited.

[0067] Figure 1 The following schematically illustrates an application scenario diagram of an operation interaction method according to an embodiment of the present application.

[0068] like Figure 1 As shown, the application scenario 100 according to this embodiment may include: a first terminal device 101 , a second terminal device 102 , a third terminal device 103 , a collection device 104 , a network 105 and a server 106 .

[0069] The network 105 may be used to provide a medium for a communication link between the second terminal device 102 and the acquisition device 104, and may also be used to provide a medium for a communication link between the third terminal device 103 and the server 106. The network 105 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0070] The user can perform eye movement operation interactions on the first terminal device 101 , the second terminal device 102 , and the third terminal device 103 respectively.

[0071] Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples), which can automatically perform corresponding operations based on the user's operation intention to be implemented to achieve the operation intention.

[0072] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0073] Among them, the first terminal device 101 can directly collect or obtain the user's pupil trajectory information, determine the user's operation intention locally based on the pupil trajectory information, and implement the user's operation intention to complete the eye movement operation interaction.

[0074] The second terminal device 102 can obtain the user pupil trajectory information collected by the collection device 104 through the network 105, determine the user's operation intention locally based on the pupil trajectory information, and implement the user's operation intention to complete the eye movement operation interaction.

[0075] The third terminal device 103 can directly collect or obtain the user's pupil trajectory information and send the pupil trajectory information to the server 106 via the network 105. The server 106 determines the user's operation intention based on the pupil trajectory information and returns the determined user operation intention to the third terminal device 103 via the network 105. The third terminal device 103 can implement the received user operation intention and complete the eye movement operation interaction.

[0076] The server 106 may be a server that provides various services, for example, determining the user's operation intention based on the user's pupil trajectory information.

[0077] It is understandable that the first terminal device 101, the second terminal device 102, and the third terminal device 103 are three different examples of eye movement operation interaction, and different examples can be combined with each other. This application does not limit the specific application scenarios of the disclosed operation interaction method.

[0078] It should be noted that the operation interaction method provided in the embodiment of the present application can generally be performed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the operation interaction device provided in the embodiment of the present application can generally be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0079] It should be understood that Figure 1The number of terminal devices, networks, and servers in the embodiment is merely illustrative. Any number of terminal devices, networks, and servers may be provided as required.

[0080] Figure 2 The following schematically shows a flow chart of an operation interaction method according to an embodiment of the present application.

[0081] like Figure 2 As shown, the operation interaction method of this embodiment may include operations S210 to S230.

[0082] The method flow does not limit the specific execution subject. The execution subject can be any electronic device, such as a terminal device or a server device.

[0083] Operation S210: determining pupil trajectory information of a target user.

[0084] Operation S220 : Convert the determined pupil trajectory information into a quantum state encoding form, and determine the user operation intention predicted for the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model.

[0085] In operation S230 , an operation process for realizing the determined user operation intention is executed, and an execution result of the operation process is displayed.

[0086] This method can perform operation interaction based on the user's pupil trajectory information, provide a new operation interaction mode that is different from screen clicking, and improve the diversity of operation interaction forms.

[0087] This method can also convert pupil trajectory information into quantum state coding form to facilitate the extraction of complex features in pupil trajectory information, improve the training effect of the intention recognition model, improve the accuracy and efficiency of identifying user operation intentions, and improve the accuracy and efficiency of intention recognition of eye movement operation interaction.

[0088] In the technical solution of this application, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, pupil trajectory information, user eye information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0089] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided in the embodiments of the present application all provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0090] In an embodiment of the present application, prior to obtaining the user's information, the user's consent or authorization may be required. For example, prior to operation S210, a request may be issued to the user to determine the user's pupil trajectory information. If the user consents or authorizes the determination of the user's pupil trajectory information, operation S210 is performed.

[0091] In an embodiment of the present application, a corresponding operation entry can be provided for the user to choose to agree or reject the automated decision result. That is, before determining the operation intention based on the user's pupil trajectory information, the user can obtain an instruction input through the corresponding operation entry to agree or reject "determine the operation intention based on the user's pupil trajectory information". If the user agrees to determine the operation intention based on the pupil trajectory information, the operation intention is determined based on the pupil trajectory information, that is, operation S220 is performed. If the user refuses to determine the operation intention based on the pupil trajectory information, the expert decision-making process can be entered. Similarly, operation S230 can also be performed according to the same process. For example, if the user agrees to implement the determined operation intention, operation S230 is performed.

[0092] The embodiments of the present application do not limit the target user, and the target user may be any user. Specifically, any user who needs to perform an operation interaction may be referred to as the target user.

[0093] The following explains various aspects of the above method flow.

[0094] 1. Operation S210: Determine pupil trajectory information of the target user.

[0095] The embodiments of this application do not limit the specific form of pupil trajectory information. Optionally, the pupil trajectory information can be in the form of multiple sets of correspondences between pupil positions and time points, or in the form of a trajectory without time information, or in the form of a trajectory on a two-dimensional plane or in a three-dimensional space.

[0096] The embodiments of the present application do not limit the method of determining the pupil trajectory information of the target user. Optionally, the pupil trajectory information of the target user can be directly collected, or the pupil trajectory information of the target user collected by other devices can be obtained.

[0097] In a specific example, the execution subject of the method process can be a terminal device, which can collect the pupil trajectory information of the target user through a camera. The terminal device can specifically be smart glasses; the terminal device can also obtain the pupil trajectory information of the target user collected by the eye information collection device by communicating with the eye information collection device (such as smart glasses).

[0098] In an optional embodiment, the target user's eye information may be continuously monitored, and the target user's pupil position may be determined. The target user's pupil may be tracked to obtain the target user's pupil trajectory information.

[0099] The embodiments of this application do not limit the method for tracking the user's pupils. Alternatively, a bright pupil tracking method or a dark pupil tracking method may be used. The bright pupil tracking method primarily tracks the pupil by identifying a bright pupil within a dark iris, while the dark pupil tracking method primarily tracks the pupil by identifying a dark pupil within a bright iris.

[0100] The embodiments of this application do not limit the specific method for selecting the pupil tracking method. Optionally, the pupil trajectory information of the target user can always be determined based on the same pupil tracking method; or, based on the actual needs, a matching pupil tracking method can be selected to determine the pupil trajectory information of the target user, thereby improving the pupil tracking accuracy. Among them, environmental information and human eye characteristics may affect the tracking effect of the pupil tracking method.

[0101] In an optional embodiment, a matching pupil tracking method may be selected based on current environmental information and / or eye characteristics of the target user.

[0102] The aforementioned bright pupil tracking and dark pupil tracking methods match different environmental information (e.g., lighting conditions) and eye characteristics (e.g., pupil color). Bright pupil tracking achieves higher tracking accuracy in low-light environments, while dark pupil tracking achieves higher accuracy in bright environments. Bright pupil tracking is less affected by eyelashes, while dark pupil tracking is more affected. Bright pupil tracking is suitable for light-colored pupils, while dark pupil tracking is suitable for dark-colored pupils.

[0103] Therefore, a matching pupil tracking method can be selected based on the current environmental information and / or the current target user's eye characteristics to determine the target user's pupil trajectory information, thereby improving pupil tracking accuracy. The embodiments of this application do not limit the specific matching relationship between "environmental information and / or eye characteristics" and pupil tracking methods, nor do they limit the specific matching method.

[0104] Optionally, determining the pupil trajectory information of the target user can specifically include: monitoring environmental information and human eye characteristics of the target user; determining a matching pupil tracking method based on the currently monitored environmental information and human eye characteristics; and determining the pupil trajectory information of the target user based on the determined pupil tracking method.

[0105] This embodiment does not limit the specific form and content of environmental information, nor does it limit the specific form and content of human eye characteristics. In one example, environmental information may include information such as lighting that may affect pupil tracking, and human eye characteristics may include information such as pupil information, eyelash shading, pupil color, and other information that may affect pupil tracking.

[0106] This embodiment does not limit the specific process of determining the pupil trajectory information of the target user based on the determined pupil tracking method. Optionally, the pupil trajectory information of the target user can be determined based on the determined pupil tracking method and the monitored human eye features. Specifically, the pupil position can be determined based on the determined pupil tracking method and the monitored human eye features for tracking, thereby determining the pupil trajectory information of the target user. Optionally, the pupil position of the target user can be determined based on the determined pupil tracking method and the tracking can be performed, thereby determining the pupil trajectory information of the target user.

[0107] In addition, monitoring of environmental information and / or eye features of the target user may be performed with authorization from the target user.

[0108] Optionally, determining the pupil trajectory information of the target user may specifically include: determining the target user's authorization to monitor eye characteristics; monitoring environmental information and the target user's eye characteristics upon obtaining the target user's authorization to monitor eye characteristics; determining a matching pupil tracking method based on the currently monitored environmental information and eye characteristics; and determining the target user's pupil trajectory information based on the determined pupil tracking method. It is understood that it is also possible to determine the target user's authorization to monitor environmental information, and upon obtaining the target user's authorization to monitor both environmental information and eye characteristics, monitor the environmental information and the target user's eye characteristics.

[0109] This embodiment can improve the tracking precision and accuracy of pupil tracking by selecting and determining a matching pupil tracking method based on environmental information and human eye characteristics.

[0110] This embodiment does not limit the specific timing for selecting and determining a matching pupil tracking method. Alternatively, a matching pupil tracking method may be determined in real time based on the current monitored situation, and if the determined pupil tracking method changes, the new pupil tracking method may be used to determine the pupil trajectory information of the target user. Alternatively, a matching pupil tracking method may be reselected and determined if there is a significant change in the monitored information.

[0111] Optionally, a matching pupil tracking method is determined based on currently monitored environmental information and eye characteristics. Specifically, if it is determined that the degree of change in the monitored environmental information is greater than a first preset degree and / or the degree of change in the monitored eye characteristics is greater than a second preset degree, the matching pupil tracking method is re-determined based on the currently monitored environmental information and eye characteristics. This embodiment can improve the tracking precision and accuracy of pupil tracking by re-determining the matching pupil tracking method when the monitored information undergoes significant changes.

[0112] This embodiment does not limit the specific form and content of the first preset degree and the second preset degree. Optionally, the illumination change in the environmental information may be greater than a preset illumination change threshold, or the pupil color change in the human eye feature may be greater than a preset color change threshold.

[0113] The embodiments of this application do not limit the specific method for determining a matching pupil tracking method. Alternatively, a matching pupil tracking method can be determined based on analysis of the currently monitored environmental information and human eye characteristics. For example, if the lighting characteristics in the monitored environmental information are bright light, a matching bright pupil tracking method can be determined. Alternatively, the pupil can be tracked using different pupil tracking methods in practice, the tracking effect (e.g., tracking accuracy) can be determined, and the pupil tracking method can be determined based on the tracking effect.

[0114] Optionally, based on the currently monitored environmental information and human eye characteristics, a matching pupil tracking method is determined. Specifically, when it is determined that the degree of change of the monitored environmental information is greater than a first preset degree, and / or the degree of change of the monitored human eye characteristics is greater than a second preset degree, the tracking effects are determined for different pupil tracking methods based on the currently monitored environmental information and human eye characteristics, and the pupil tracking method whose tracking effect meets the preset effect conditions is determined as the matching pupil tracking method.

[0115] This embodiment can improve the tracking effect, tracking precision and accuracy of pupil tracking and improve the degree of matching between the pupil tracking method and the current situation (environmental information and human eye characteristics) by re-determining the matching pupil tracking method based on the tracking effects of different pupil tracking methods when the monitored information undergoes significant changes.

[0116] This embodiment does not limit the form of the tracking effect, which may specifically be the tracking accuracy. This embodiment also does not limit the preset effect condition, which may specifically be a pupil tracking method with a tracking effect that is better than other pupil tracking methods.

[0117] This embodiment does not limit the specific method for determining the tracking effect. Optionally, different pupil tracking methods may be used within a preset time period to obtain pupil trajectory information, and the tracking effect (e.g., tracking accuracy) may be determined based on the pupil trajectory information. For example, the tracking effect may be determined based on the completeness or accuracy of the pupil trajectory information. Alternatively, different pupil tracking methods may be used within a preset time period to obtain pupil information, and the tracking effect may be determined based on the pupil information. For example, the tracking effect may be determined based on the completeness or accuracy of the pupil information.

[0118] Optionally, using bright pupil tracking and dark pupil tracking as examples, a matching pupil tracking method may be determined based on currently monitored environmental information and human eye characteristics. Specifically, upon determining that the degree of change in the monitored environmental information is greater than a first preset degree, and / or the degree of change in the monitored human eye characteristics is greater than a second preset degree, the tracking effect is determined for each of the bright pupil tracking method and the dark pupil tracking method based on the currently monitored environmental information and human eye characteristics, and the pupil tracking method with a better tracking effect than the other pupil tracking method is determined as the matching tracking method. For other multiple pupil tracking methods, a matching tracking method may also be determined by separately determining the tracking effect (tracking accuracy) and comparing them.

[0119] 2. Operation S220: convert the determined pupil trajectory information into a quantum state encoding form, and determine the user operation intention predicted by the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model.

[0120] The embodiments of the present application do not limit the specific method for converting pupil trajectory information into a quantum state encoding form. Alternatively, the pupil trajectory information can be converted into a quantum state encoding form locally, or the pupil trajectory information can be sent to an external device, which then converts the pupil trajectory information into a quantum state encoding form, and the pupil trajectory information converted into a quantum state encoding form can be determined. Optionally, the pupil trajectory information can be specifically represented by quantum bits.

[0121] Accordingly, the intention recognition model can process pupil trajectory information in the form of quantum state encoding to predict the user's operational intention. The embodiments of the present application are not limited to the intention recognition model. Optionally, the parameters in the intention recognition model can be in the form of quantum states, for example, based on a deep learning network in the complex domain. The parameters in the intention recognition model (such as model weights) can be represented in the form of quantum states or quantum bits, and the pupil trajectory information in the form of quantum state encoding of the input can be processed. Specifically, the processing can be performed using quantum computing.

[0122] This embodiment can convert pupil trajectory information into quantum state coding form, which can facilitate the extraction of complex features in pupil trajectory information, improve the training effect of the intention recognition model, improve the accuracy and efficiency of identifying user operation intentions, and improve the accuracy and efficiency of intention recognition of eye movement operation interaction.

[0123] The embodiments of the present application do not limit the specific structure and specific algorithm of the intent recognition model. Optionally, the intent recognition model can be a convolutional neural network structure or a decision tree structure.

[0124] The embodiments of this application do not limit the training method of the intent recognition model. Optionally, the intent recognition model can be trained based on training samples from different users. Alternatively, personalized training can be performed for the target user, using training samples related to the target user to improve the accuracy of the intent recognition model in recognizing the target user's operational intent.

[0125] Optionally, the training method of the intent recognition model may include: determining a training sample set; any sample in the training sample set includes: sample features for characterizing the pupil trajectory information of the target user, and sample labels for characterizing the user's operation intention; converting the sample features in the training sample set into quantum state coding form; training the intent recognition model based on the sample features in the quantum state coding form and the corresponding sample labels.

[0126] This embodiment can train the intention recognition model based on training samples related to the target user, thereby improving the personalization of the intention recognition model for the target user and improving the recognition accuracy of the intention recognition model for the target user's operation intention.

[0127] Optionally, the intent recognition model targeted by the above training method can be an untrained intent recognition model; or it can be an intent recognition model pre-trained based on training samples of different users, which already has a certain ability to predict and recognize user operation intentions.

[0128] In a specific example, as the target user's operational interactions gradually increase, the correlation between the target user's operational intentions and the target user's pupil trajectory information can be collected, so that relevant training samples of the target user can be gradually accumulated, and the training intention recognition model can be continuously optimized, so that the intention recognition model can learn the target user's operational interaction habits and improve the personalization of the intention recognition model for the target user.

[0129] For example, when the target user performs the "fill in the verification code" operation intention through screen interaction, the corresponding pupil trajectory information can be collected; when the target user performs the "drag down" operation intention through screen interaction, the corresponding pupil trajectory information can be collected; or the target user can pre-set specific pupil trajectory information for the "check the weather" operation intention, and the corresponding association relationship can be determined for constructing training samples.

[0130] In a specific example, the target user can set the association between "operation intention" and "pupil trajectory information" by himself. Specifically, the target user can provide a specific pupil trajectory information for binding to a selected operation intention, so that the selected operation intention can be determined when it is determined that the pupil trajectory information of the target user conforms to the specific pupil trajectory information, or the similarity with the specific pupil trajectory information is greater than the preset similarity. The set of associations set by the target user can also be constructed as training samples to optimize the training intention recognition model. Specifically, the weight of the constructed training samples can be increased for the associations set by the target user to better learn the operation habits of the target user.

[0131] The embodiments of the present application do not limit the specific method of converting the sample features into a quantum state encoding form. For details, please refer to the explanation of converting pupil trajectory information into a quantum state encoding form.

[0132] The embodiments of the present application do not limit the specific device for training the intent recognition model. Optionally, the intent recognition model can be pre-trained by the execution entity of the method process, or by other external devices.

[0133] The embodiments of the present application do not limit the deployment device of the intention recognition model. Optionally, the execution subject of the present method process may be deployed with a pre-trained intention recognition model, so that the execution subject can determine the user operation intention predicted by the pupil trajectory information in the form of quantum state encoding based on the pre-trained intention recognition model. It is also possible that other external devices are deployed with a pre-trained intention recognition model, and the pupil trajectory information in the form of quantum state encoding can be sent to the external device, and the external device determines the user operation intention predicted by the pupil trajectory information in the form of quantum state encoding based on the pre-trained intention recognition model, and then the external device returns the determined user operation intention to the execution subject; it is also possible that the pupil trajectory information is sent to an external device, and the external device converts the pupil trajectory information into a quantum state encoding form and predicts the user operation intention.

[0134] Accordingly, determining the user operation intention may specifically be determining the user operation intention output by a local intention recognition model, or determining the user operation intention predicted by an external device based on the intention recognition model.

[0135] The embodiments of this application do not limit the specific form and content of the user's operation intention. Optionally, the user's operation intention may include at least one of the following: moving up a page, moving down a page, zooming in on a page, clicking a certain location, double-clicking a certain location, executing a certain function, playing a certain video, checking the weather, filling in a verification code, etc.

[0136] Optionally, the intent recognition model can convert user operation intent into quantum state encoding. Specifically, the user operation intent can be represented by quantum bits, which can improve the training effect and intent recognition accuracy of the intent recognition model. The intent recognition model can output the user operation intent in quantum state encoding form, which can further determine the matched user operation intent in text form.

[0137] The embodiments of this application do not limit the input and output of the intent recognition model. Optionally, pupil trajectory information in the form of quantum state encoding, as well as other information, can be used as input to the intent recognition model to predict the user's operation intention, thereby improving the recognition accuracy of the user's operation intention.

[0138] Optionally, the currently displayed content may be associated with the user's operation intention. For example, for a currently displayed verification page, the user's operation intention may be to fill in the verification information; for a currently displayed video page, the user's operation intention may be to play the video.

[0139] Optionally, the historical operation record may be associated with the user's operation intention. For example, after the "move page down" operation, the user's operation intention may be to click a button at the bottom of the page.

[0140] Optionally, historical pupil trajectory information can also be used as input to the intention recognition model, so that the intention recognition model can predict the user's operation intention based on more comprehensive pupil trajectory information.

[0141] Therefore, in an optional embodiment, based on a pre-trained intention recognition model, the user operation intention predicted by the pupil trajectory information in quantum state encoding form is determined. Specifically, it can be: the determined pupil trajectory information is converted into a quantum state encoding form, and based on the pre-trained intention recognition model, the user operation intention predicted by the pupil trajectory information in quantum state encoding form and at least one of the following is determined: currently displayed content, historical operation records, and historical pupil trajectory information.

[0142] This embodiment can be combined with other input information to use the intention recognition model to predict the user's operation intention, which can improve the accuracy and comprehensiveness of the intention recognition model for operation intention.

[0143] This embodiment does not limit the input form of the intent recognition model. Optionally, the input information can be converted into a quantum state encoding form, and then the input information in the quantum state encoding form can be input into the intent recognition model. For example, the current display content can be converted into a quantum state encoding form, or historical pupil trajectory information can be converted into a quantum state encoding form. Alternatively, the input information can be directly input into the intent recognition model.

[0144] Optionally, the user operation intention predicted by the intention recognition model can specifically be an operation intention for the currently displayed content. The intention recognition model can make predictions based on the operation intentions supported by the currently displayed content to improve the efficiency and accuracy of intention recognition.

[0145] The embodiments of the present application do not limit the output form of the intent recognition model. Optionally, the intent recognition model can output one or more predicted user operation intentions. For a user operation intention output by the intent recognition model, the subsequent steps can be directly executed. For multiple user operation intentions output by the intent recognition model, further screening can be performed to determine the user operation intention that needs to be implemented.

[0146] Therefore, optionally, based on a pre-trained intent recognition model, the user operation intention predicted by the pupil trajectory information in quantum state encoding is determined. Specifically, the user operation intention predicted by the pupil trajectory information in quantum state encoding is determined based on the pre-trained intent recognition model, and the determined user operation intention is screened to determine the user operation intention that needs to be implemented. In other words, the user operation intention predicted by the intent recognition model can be screened to determine the user operation intention that needs to be implemented.

[0147] The embodiments of the present application do not limit the number and manner of specifically screening the user operation intentions that need to be implemented. Optionally, one or more user operation intentions that need to be implemented can be determined. For example, although the user's pupil trajectory information indicates that an operation intention needs to be implemented, another predicted operation intention is also what the user wants to implement, so multiple user operation intentions that need to be implemented can be determined.

[0148] Optionally, multiple user operation intentions output by the intention recognition model can be displayed to the target user, and the user operation intention that needs to be implemented can be determined based on the target user's selection operation. This embodiment does not limit the specific form of the selection operation, which can be in the form of screen interaction, eye movement interaction, or voice interaction. For example, the target user gazes at a displayed user operation intention to determine that the intention needs to be implemented, or receives the target user's voice "select the second and third operation intentions" to determine the operation intention that needs to be implemented.

[0149] Optionally, new training samples can be constructed based on the target user's selected actions to optimize the training intent recognition model. For example, the user action intention selected by the target user can be used to construct a training sample with the corresponding pupil trajectory information. User action intentions not selected by the target user can be used to construct negative samples with the corresponding pupil trajectory information, or no training samples can be constructed, or similar training samples can be deleted.

[0150] Optionally, for multiple user operation intentions output by the intent recognition model, the operation intentions that meet preset statistical conditions can be determined as the operation intentions to be implemented based on the target user's historical operation intention statistics. This embodiment does not limit the preset statistical conditions, and can specifically be used to represent recent frequent implementations. For example, the preset statistical condition can be that the number of implementations of the operation intention within a preset time period before the current moment exceeds a preset number threshold.

[0151] In addition, optionally, the intention recognition model can also output no operation intention. For example, for each user operation intention, the probability predicted by the intention recognition model is lower than the preset probability threshold, so that it can output that the target user currently has no operation intention, or the target user's current operation intention is uncertain. Optionally, when determining the user operation intention that needs to be implemented, it can also be determined that there is no user operation intention that needs to be implemented. For example, when the target user's voice "no operation intention" is received, it is determined that there is currently no user operation intention that needs to be implemented.

[0152] After determining the user operation intention that needs to be implemented based on the intention recognition model, subsequent steps can be performed to implement the determined user operation intention.

[0153] 3. Operation S230: executing an operation process for realizing the determined user operation intention, and displaying the execution result of the operation process.

[0154] The embodiments of the present application do not limit the specific conditions or specific timing for executing the operation process.

[0155] Optionally, when the user's operation intention is determined, the operation process for realizing the determined user's operation intention can be directly executed; or when the user's operation intention is determined, the target user can be asked whether he is sure to perform the operation to realize the determined user's operation intention. If the target user confirms that the determined user's operation intention needs to be realized, the operation process for realizing the determined user's operation intention can be executed.

[0156] This embodiment does not limit the specific method of asking the target user. In a specific example, a query window can be displayed for the target user so that the target user can choose to "confirm to implement the determined user operation intention" or "reject to implement the determined user operation intention".

[0157] This embodiment can improve the convenience of user operation and user experience by asking the target user whether the determined user operation intention needs to be implemented.

[0158] In addition, new training samples can be constructed based on the target user's selection to optimize the training intention recognition model. Optionally, if the target user selects "Confirm to implement the determined user operation intention," the determined user operation intention and the corresponding pupil trajectory information can be constructed as training samples. If the target user selects "Reject to implement the determined user operation intention," the determined user operation intention and the corresponding pupil trajectory information can be constructed as negative samples, or no training samples can be constructed, or similar training samples can be deleted.

[0159] The embodiments of the present application do not limit the correspondence between "user operation intention" and "operation process for realizing user operation intention". Optionally, the operation process corresponding to the user operation intention can be used to realize the corresponding user operation intention. Among them, a single user operation intention can correspond to one operation process or multiple different operation processes. For example, the same operation intention "contact a certain user" can be realized through different operation processes.

[0160] The embodiment of the present application does not limit the construction method of the correspondence between the user operation intention and the operation process. Optionally, an operation process for realizing the user operation intention can be constructed for any user operation intention.

[0161] The embodiments of the present application do not limit the specific method of constructing the "operation process for realizing the user's operation intention." Optionally, the operation steps for realizing the user's operation intention can be extracted from the operation log and combined into the operation process.

[0162] The operation process can include one or more operation steps. For example, the operation intention of "dragging the page down" can be achieved through a single operation of "dragging down"; the operation intention of "filling in the verification code" can be achieved through multiple operations of "sending a verification code acquisition request", "obtaining the verification code", and "filling in the verification code in the designated area".

[0163] For an operation process that includes multiple steps, further analysis can be performed to determine the relationship between the different steps and determine the execution order of the different steps to improve the execution efficiency of the operation process. For example, two different operation steps in the operation process that do not affect each other can be executed in parallel to improve the execution efficiency of the operation process.

[0164] Therefore, optionally, executing the operation process for realizing the determined user operation intention may specifically be: when the operation process for realizing the determined user operation intention includes at least two parallel operations, in the process of executing the operation process, executing different parallel operations included in the operation process in parallel.

[0165] This embodiment can improve the execution efficiency of the operation process by executing different operations in the operation process in parallel.

[0166] Optionally, the at least two parallel operations may be different operations that do not affect each other, or may be different operations that are allowed to be executed in parallel. For example, the two operations of "sending a video playback request" and "reducing the page" do not affect each other and are allowed to be executed in parallel. Thus, they can be executed in parallel in the operation process to improve the execution efficiency of the operation process.

[0167] Therefore, optionally, in the process of constructing the operation process, the relationship between different operation steps can be analyzed to determine the execution order between different operation steps, thereby improving the execution efficiency of the operation process.

[0168] The embodiments of the present application do not limit the specific execution method of the operation process, nor do they limit the content displayed during the execution of the operation process.

[0169] Optionally, the operation process can be executed, and the results of the intermediate operations can be displayed to the target user; the operation process can also be executed in the background, and the results of the intermediate operations are not displayed to the target user, but the overall results of the operation process are displayed to the target user when the operation process is completed.

[0170] Optionally, the operation process for realizing the determined user operation intention is executed, and the execution result of the operation process is displayed. Specifically, it can be: executing the operation process for realizing the determined user operation intention in the background; when it is determined that the operation process is completed, the execution result of the operation process is displayed on the front end.

[0171] This embodiment executes the operation process in the background and displays the execution results of the operation process on the front end, which can reduce the operations during the execution of the operation process, such as interactive operations with the front end, and can improve the execution efficiency of the operation process and improve the efficiency of realizing the user's operation intentions.

[0172] In this embodiment, the operation process can optionally be executed in the background, and some operations in the operation process may not be displayed on the front end. For example, "Find a specific component on a page" can directly search for page information in the background, without displaying the page drag effect on the front end. Therefore, the interaction between the front end and the operation process can be reduced during the execution of the operation process, thereby improving the execution efficiency of the operation process.

[0173] The embodiments of the present application do not limit the specific content and form of the execution results of the operation process.

[0174] Optionally, the execution result of the operation process can be used to indicate whether the operation process was successfully executed, or whether the user's operation intention was successfully achieved. The execution result of the operation process can also be used to indicate the business execution result of the operation process. For example, for the operation intention of "video play", the execution result of the operation process can be displayed as a played video.

[0175] The embodiments of the present application do not limit the display method and display device of the execution results of the operation process.

[0176] Optionally, the execution result of the operation process can be displayed locally on the execution entity; or the execution entity can display the execution result of the operation process through other display devices, which can be a display device that is communicatively coupled to the execution entity, such as a display screen, or a virtual screen display device.

[0177] For ease of understanding, the embodiments of the present application also provide an application embodiment for illustrative purposes.

[0178] This embodiment can apply quantum computing and deep learning to perform complex judgments on pupil movement and control the display and movement of various functions of a screen (such as a mobile phone screen).

[0179] There is an urgent need for automated and intelligent mobile phone screen control. More and more users want to control the functions of their mobile phones without touching them. This embodiment can be applied to close-up viewing of a mobile phone. Based on pupil trajectory information such as the distance, route, and dwell time of pupil movement, quantum computing and deep learning are performed to predict the user's operating intentions.

[0180] Combining quantum computing and deep learning can leverage the strengths of both to solve more complex problems. Quantum computing can be used to accelerate the training process of deep learning models, especially those involving large numbers of parameters and complex computations. Optimizing the model's weights and biases through quantum algorithms can significantly improve model training speed and accuracy. Quantum computing can serve as a new feature extraction method, extracting more complex and useful features from input data. These features can be input into deep neural networks for classification or prediction. Traditional computing methods may encounter performance bottlenecks when processing high-dimensional data. Quantum computing, on the other hand, can efficiently process high-dimensional data, enabling more accurate classification and prediction in high-dimensional spaces. Quantum computing can accelerate the execution of many machine learning algorithms, including support vector machines, decision trees, and random forests.

[0181] This embodiment can use quantum computing to process the multi-dimensional information of the pupil for prediction, and use deep learning to perform classification to determine the user's operation intention.

[0182] The method steps of this embodiment are explained in detail below.

[0183] This embodiment may include a tracking module, a processing module, an analysis module and a triggering module.

[0184] The tracking module can be used to determine pupil trajectory information.

[0185] Pupil capture. Close-range light makes pupils easier to capture. The camera captures pupil position and size information and sends it to the processing module. Normal pupil capture can be affected by factors such as lighting and pupil physiological characteristics. To optimize accuracy, this embodiment utilizes bright pupil / dark pupil tracking technology to reduce the impact of physiological characteristics and determine ambient light conditions.

[0186] Bright pupil tracking works by identifying a bright pupil within a dark iris. It's less affected by eyelashes and works better in dim environments, suitable for light-colored pupils. Dark pupil tracking, on the other hand, works by identifying a dark pupil within a bright iris. It's more affected by eyelashes and works better in bright environments, suitable for dark-colored pupils.

[0187] Pupil movement tracking and recording. After selecting a tracking method through the built-in eye tracker, information such as the distance between the pupil and the camera, angle, and movement trajectory is obtained. Under normal circumstances, the tracking method will not be changed unless there is an environmental change that significantly changes the tracking ability. At this time, the eye tracker will perform a one-time test of both tracking methods to determine the matching tracking method and send the obtained information to the processing module.

[0188] The system can obtain eye characteristics and ambient light conditions to select a matching tracking method. Eye characteristics can determine whether eyelashes are obstructed and pupil color. Pupil movement information is measured. If significant changes in tracking capabilities are detected, such as changes in eye characteristics or ambient light, an internal test can be initiated to re-determine the tracking method by comparing the results of the bright and dark pupils. The measured pupil movement information (pupil trajectory information) can then be transmitted to the processing module.

[0189] The processing module integrates the pupil trajectory information passed by the tracking module and inputs it into the calculation module.

[0190] The computing module first records data through deep learning, predicts multi-dimensional data results through quantum computing, and matches the input information with the predicted results.

[0191] In previous calculations, there were generally only two possible predictions, 0 and 1. However, the movement trajectory of the eyeball changes in real time and is continuous. The superposition state characteristics of quantum computing are very consistent with the movement of the eyeball, and can truly reflect the correspondence between eyeball movement and results. In addition, superposition calculations of multiple states at the same time also improve computing efficiency.

[0192] The screen control method of this embodiment is not limited to physical screens, but also covers the control of various virtual screens and images. The control device is generally worn on the user's eyes and is not limited to glasses, headphones or small devices. It can be independently connected to the screen to be controlled through the network and supports extended connections such as Bluetooth or USB, which means that it can switch to connect various devices.

[0193] In its initial state, this embodiment requires the user to perform directional training, including simple movements of up, down, left, and right, as well as some simulated control functions. After capturing the user's pupil movement trajectory, the user's actions are matched. When implementing complex, continuous functions (those not displayed on the surface), deep learning can continuously record and correct matching results, moving from simple step-by-step execution to direct background execution, ultimately displaying the desired results.

[0194] Examples of quantum computing applications:

[0195] Required operations: Swipe down the screen to display an input box, a Send Verification Code button, and a Jump button. Click Send Verification Code on this page and check the text message. Copy and paste the information into the input box and click the Jump button. If the information is correct, the web page will be redirected.

[0196] Process implementation:

[0197] According to the normal processing logic, first slide down the screen and click the Send Verification Code button to trigger the SMS sending, and then wait for the SMS reminder. After receiving the SMS, the message prompt will generally display part of the content floating above. At this time, the pupils will move up to browse the information content, and stay to trigger the copy and paste function (if the thumbnail display is enabled, the full display function will also be triggered). After the content is pasted, move to the jump button to trigger the click.

[0198] This embodiment, through the fusion of deep learning and quantum computing, can detect that a button needs to be clicked on a page and retrieve the content to be filled in, while the pupil's movement trajectory remains unchanged. The processing logic is parallel, with the button click and the waiting for information to be copied triggered simultaneously. When the user's pupil moves upward, the waiting for information to be copied has already completed data processing and the content has been copied. In the seconds between the upward and downward movement of the gaze, the content is filled in, and the jump is completed by simply moving the gaze downward to the jump button. This is the contribution of quantum computing.

[0199] Application Description: First, the pupil's trajectory over time is three-dimensional and can be represented by xyz coordinates, where x and y represent two adjacent directions. Generally, x represents left, y represents down, and z represents time. Therefore, the pupil's trajectory forms a three-dimensional trajectory in three-dimensional space. Quantum superposition is ideally suited for this application scenario. The combination of countless possibilities and a single reality (Schrödinger's equation) allows for the prediction of complex movement trajectories, while any actual path still influences the outcome. Deep learning can already understand the user's operating habits, and subtle changes in the pupil's path can distinguish between a single operation and the simultaneous execution of multiple functions. Quantum computing can combine various operations to determine the appropriate control combination in a short period of time.

[0200] The processing module can perform preliminary calculations based on pupil trajectory information and send matching results to the analysis module. The analysis module can use deep learning predictions to determine the sustainability of functional implementation.

[0201] If it is a continuous operation (multiple operations need to be performed), the analysis module can perform quantum calculations on the pupil trajectory information, match the results of each operation step in real time, and the trigger module actively monitors the operation of the analysis module to achieve continuous execution of the operation.

[0202] If it is not a continuous operation, a simple function can be executed by the trigger module.

[0203] The beneficial effects of this embodiment include at least:

[0204] 1. This embodiment can collect and count the user's pupil movements through deep learning, which can not only optimize the relationship between the information after pupil recognition and the prediction results, but also predict the user's needs that can be realized simultaneously.

[0205] 2. This embodiment can use the superposition state of quantum computing to extract the characteristics of the pupil trajectory, which is very suitable for the prediction of three-dimensional trajectories such as pupil movement, breaking through the prediction bottleneck of multi-dimensional data, improving the effectiveness of feature extraction, and improving the accuracy and efficiency of operation implementation.

[0206] Corresponding to the above method embodiment, the present application also provides an operation interaction device. Figure 3 The device is described in detail.

[0207] Figure 3 The following schematically shows a structural block diagram of an operation interaction device according to an embodiment of the present application.

[0208] like Figure 3 As shown, the operation interaction device 300 of this embodiment includes:

[0209] The trajectory determination unit 301 is configured to determine pupil trajectory information of the target user. In one embodiment, the trajectory determination unit 301 may be configured to perform the operation S210 described above, which will not be described in detail herein.

[0210] Prediction unit 302 is configured to convert the determined pupil trajectory information into a quantum state encoding format and, based on a pre-trained intent recognition model, determine the user's predicted operation intention based on the pupil trajectory information in the quantum state encoding format. In one embodiment, prediction unit 302 may be configured to perform operation S220 described above, which will not be further described here.

[0211] The execution unit 303 is configured to execute the operation process for realizing the determined user operation intention and display the execution result of the operation process. In one embodiment, the execution unit 303 can be configured to execute the operation S230 described above, which will not be described in detail here.

[0212] Optionally, the trajectory determination unit 301 is used to: determine the target user's authorization for monitoring human eye characteristics; monitor environmental information and the target user's human eye characteristics when the target user has authorized monitoring of human eye characteristics; determine a matching pupil tracking method based on the currently monitored environmental information and human eye characteristics; and determine the target user's pupil trajectory information based on the determined pupil tracking method.

[0213] Optionally, the trajectory determination unit 301 is used to: when it is determined that the degree of change of the monitored environmental information is greater than a first preset degree, and / or the degree of change of the monitored human eye characteristics is greater than a second preset degree, determine the tracking effect for different pupil tracking methods based on the currently monitored environmental information and human eye characteristics, and determine the pupil tracking method whose tracking effect meets the preset effect conditions as the matching pupil tracking method.

[0214] Optionally, the execution unit 303 is configured to: when an operation flow for realizing the determined user operation intention includes at least two parallel operations, execute different parallel operations included in the operation flow in parallel during the execution of the operation flow.

[0215] Optionally, the execution unit 303 is configured to: execute an operation process for realizing the determined user operation intention in the background; and display an execution result of the operation process in the front end when it is determined that the operation process is completed.

[0216] Optionally, the training method of the intent recognition model includes: determining a training sample set; any sample in the training sample set includes: sample features for characterizing the pupil trajectory information of the target user, and sample labels for characterizing the user's operation intention; converting the sample features in the training sample set into quantum state coding form; training the intent recognition model based on the sample features in the quantum state coding form and the corresponding sample labels.

[0217] Optionally, the prediction unit 302 is used to: convert the determined pupil trajectory information into a quantum state coding form, and based on a pre-trained intention recognition model, determine the pupil trajectory information for the quantum state coding form and the predicted user operation intention for at least one of the following: currently displayed content, historical operation records, and historical pupil trajectory information.

[0218] The explanation of the above device embodiment can refer to the explanation of the method embodiment.

[0219] According to an embodiment of the present application, any multiple units among the trajectory determination unit 301, the prediction unit 302, and the execution unit 303 can be combined into a single unit, or any one of these units can be split into multiple units. Alternatively, at least part of the functionality of one or more of these units can be combined with at least part of the functionality of other units and implemented in a single unit.

[0220] According to an embodiment of the present application, at least one of the trajectory determination unit 301, the prediction unit 302, and the execution unit 303 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the trajectory determination unit 301, the prediction unit 302, and the execution unit 303 can be at least partially implemented as a computer program unit, which can perform the corresponding function when executed.

[0221] Figure 4 A block diagram of an electronic device suitable for implementing an operation interaction method according to an embodiment of the present application is schematically shown.

[0222] like Figure 4 As shown, an electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage unit 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0223] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0224] According to an embodiment of the present application, electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to bus 904. Electronic device 900 may also include one or more of the following components connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 908 including a hard disk; and a communication section 909 including a network interface card such as a LAN card or modem. Communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 910 as needed, so that computer programs read from the removable media can be installed into storage section 908 as needed.

[0225] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, implements any method embodiment according to the embodiments of this application.

[0226] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0227] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to cause the computer system to implement any method embodiment provided in the embodiments of the present application.

[0228] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 901 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0229] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0230] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0231] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0232] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in connection can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0233] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

Claims

1. An operation interaction method, characterized in that: The method comprises: Determine pupil trajectory information of the target user; Converting the determined pupil trajectory information into a quantum state encoding form, and determining the user operation intention predicted by the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model; An operation process for realizing the determined user operation intention is executed, and an execution result of the operation process is displayed.

2. The method according to claim 1, characterized in that Determining pupil trajectory information of the target user includes: Determine the target user's authorization to monitor human eye characteristics; monitoring environmental information and the eye characteristics of the target user with authorization from the target user for monitoring the eye characteristics; Determine the matching pupil tracking method based on the currently monitored environmental information and human eye characteristics; Based on the determined pupil tracking method, pupil trajectory information of the target user is determined.

3. The method according to claim 2, characterized in that The method of determining a matching pupil tracking method based on the currently monitored environmental information and human eye characteristics includes: When it is determined that the degree of change of the monitored environmental information is greater than a first preset degree, and / or the degree of change of the monitored human eye characteristics is greater than a second preset degree, the tracking effects are determined for different pupil tracking methods based on the currently monitored environmental information and human eye characteristics, and the pupil tracking method whose tracking effect meets the preset effect conditions is determined as the matching pupil tracking method.

4. The method according to claim 1, wherein The execution of the operation process for realizing the determined user operation intention includes: In a case where the operation flow for realizing the determined user operation intention includes at least two parallel operations, during the execution of the operation flow, different parallel operations included in the operation flow are executed in parallel.

5. The method according to claim 1, wherein The executing the operation process for realizing the determined user operation intention and displaying the execution result of the operation process includes: Execute the operation process for realizing the determined user operation intention in the background; When it is determined that the operation process is completed, the execution result of the operation process is displayed on the front end.

6. The method according to claim 1, wherein The training method of the intent recognition model includes: Determine a training sample set; any sample in the training sample set includes: a sample feature for characterizing pupil trajectory information of the target user, and a sample label for characterizing user operation intention; Convert the sample features in the training sample set into quantum state encoding form; The intent recognition model is trained based on the sample features in quantum state encoding form and the corresponding sample labels.

7. The method according to claim 1, characterized in that The method of determining the user operation intention predicted by pupil trajectory information in quantum state encoding based on the pre-trained intention recognition model includes: The determined pupil trajectory information is converted into a quantum state encoding form, and based on a pre-trained intention recognition model, the pupil trajectory information for the quantum state encoding form and the user operation intention predicted by at least one of the following are determined: currently displayed content, historical operation records, and historical pupil trajectory information.

8. An operation interaction device, characterized in that: The device comprises: A trajectory determination unit, configured to determine pupil trajectory information of a target user; a prediction unit, configured to convert the determined pupil trajectory information into a quantum state encoding form, and determine the user operation intention predicted based on the pupil trajectory information in the quantum state encoding form based on a pre-trained intention recognition model; The execution unit is used to execute the operation process for realizing the determined user operation intention and display the execution result of the operation process.

9. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method further comprises the step of executing the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images