Information processing device, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program configured to accurately detect a false operation of a user and restore the state to a former state.SOLUTION: An information processing apparatus includes: a display control unit which causes a user terminal to display multiple target contents which are to be operated; an acquisition unit which acquires operation information indicating an operation of a user with respect to the target contents; an estimation unit which estimates whether or not the operation of the user is false on the basis of line-of-sight information and expression information of the user operating the target contents; and a generation unit which generates, when the estimation unit estimates that the operation of the user is false, suggestion information indicating a suggestion to the user for restoring the state of the target contents to a former state.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Techniques for analyzing a user's eye movement to grasp the line-of-sight direction and recognizing the position of a fixation point at the tip of the line of sight are known. For example, Patent Document 1 below discloses a visual function detection device that appropriately examines the visual function of a subject. The device displays a judgment image on a display unit, moves the judgment image within a display area, detects the position of the fixation point of the subject observing the display unit, and detects the visual function of the subject based on the relationship between the movement direction of the judgment image and the movement direction of the fixation point.

[0003] Techniques for applying such a line-of-sight recognition technology to HMI (Human Machine Interface) and accepting the designation of an object displayed on a display as a GUI (Graphical User Interface) are known. This makes it possible to grasp the operation intention of a person with a severe disability who has difficulty using means for accepting user operations by recognizing physical movements such as a keyboard or a mouse, and to accept operations on devices such as personal computers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in user interfaces (UIs) using eye-tracking technology, the user may not have intended to select or manipulate the object they are focusing their gaze on. The computer may select or manipulate an object that is contrary to the user's intent. Furthermore, if an object is selected or manipulated against the user's intent, it may not be possible to revert to the original state.

[0006] In light of the above issues, this disclosure aims to provide an information processing device, an information processing method, and an information processing program that can accurately detect user errors and restore the system to its original state. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the information processing device according to this disclosure comprises: a display control unit that displays a plurality of target contents that are the subject of operation processing on a user terminal; an acquisition unit that acquires operation information indicating the user's operation on the target contents; an estimation unit that estimates whether or not the user's operation was incorrect based on the user's gaze information and facial expression information at the time of the user's operation on the target contents; and a generation unit that generates suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state when the estimation unit estimates that the user's operation was incorrect. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide an information processing device, an information processing method, and an information processing program that can accurately detect user errors and restore the system to its original state. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3]Figure 3 shows an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of information stored in the operation information storage unit of the information processing apparatus according to the embodiment. [Figure 5] Figure 5 shows an example of information stored in the model data storage unit of the information processing device according to the embodiment. [Figure 6] Figure 6 shows an example of information stored in the content storage unit of the information processing device according to this embodiment. [Figure 7] Figure 7 shows an example of the configuration of a user terminal according to this embodiment. [Figure 8] Figure 8 is a flowchart showing an example of information processing according to the embodiment. [Figure 9] Figure 9 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0010] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). However, these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application.

[0011] (Embodiment) [1-1. An example of information processing according to the embodiment] First, an example of information processing according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiment. In Figure 1, an example is shown in which the information processing according to the embodiment is executed by the information processing device 100 and the user terminal 200. Hereinafter, the information processing according to the embodiment will be described step by step.

[0012] First, the information processing device 100 displays multiple pieces of content to be operated on the user terminal 200 (step S1). For example, the information processing device 100 displays multiple pieces of content to be operated on the user terminal 200, including e-book pages, text boxes, video scroll bars, browser web page scroll bars and links, and CAD (Computer-Aided Design) objects, and receives operation information from the user. The state in which multiple pieces of content to be operated are displayed refers to, for example, a state in which multiple scroll bars that accept swipe operations are displayed, a state in which scroll bars and page screens that accept swipe operations are displayed on top of each other, a state in which multiple links to transition to other pages are displayed side by side, or a state in which a keyboard screen for text input is displayed with multiple input keys lined up.

[0013] Next, the information processing device 100 acquires user operation information, gaze information during operation, and facial expression information from the user terminal 200 (step S2). Here, operation information is information indicating the user's operation on the user terminal 200, and may be, for example, information on touch operations on a touch panel or information on the gaze point of an eye-tracking device. Gaze information is information about the user's gaze direction as measured by the eye-tracking device. Facial expression information is image information of the user's facial expression captured by the imaging unit 250 of the user terminal 200. For example, when user U performs an operation on the user terminal 200, the information processing device 100 acquires user U's operation information, user U's gaze information during operation, and facial expression information all at once.

[0014] Next, the information processing apparatus 100 estimates whether the user's operation is incorrect based on the user's gaze information and expression information during the operation (step S3). For example, the information processing apparatus 100 may use a learned model that has learned the relationship between the user's gaze information and expression information during the operation and the correctness of the user's operation to estimate whether the user's operation is incorrect based on the user's gaze information and expression information during the operation. The learned model in this case may be realized by a CNN (Convolutional Neural Network) or the like. For example, it may be realized by learning the relationship between features such as the gaze movement speed and pupil diameter of each of the left and right eyeballs as gaze information, and the mouth corners, eyebrow angles, etc. as expression information, and the error of the user's operation.

[0015] Next, when it is estimated that the user's operation is incorrect, the information processing apparatus 100 generates proposal information indicating a proposal to the user to return to the original state (step S4). For example, the information processing apparatus 100 determines an incorrect operation mode indicating the mode of the user's incorrect operation, and generates proposal information indicating a proposal to the user to return to the original state according to the determined incorrect operation mode.

[0016] Next, the information processing apparatus 100 causes the user terminal 200 to display the generated proposal information (step S5). For example, the information processing apparatus 100 causes the user terminal 200 to display proposal information indicating a proposal to the user to return to the original state, and accepts the user's intention regarding the process of returning to the original state. In this case, the information processing apparatus 100 may display the proposal information superimposed on the plurality of contents originally displayed on the user terminal 200.

[0017] Thereby, the information processing apparatus 100 can estimate the error of the user's operation based on the user's gaze information and expression information, and provide the user terminal 200 with proposal information for returning to the original state. Therefore, it is possible to provide the information processing apparatus 100 that can accurately detect the error of the user's operation and return to the original state.

[0018] 〔1-2. Other Example 1 of Information Processing According to the Embodiment〕 When the information processing apparatus 100 estimates that the user's operation is an error, it determines an incorrect operation mode indicating the mode of the user's operation error, and when it is determined that the user has transitioned to an incorrect page with respect to the user's incorrect operation mode, it generates proposal information including a browser back button for returning to the original page.

[0019] The following is an explanation of this information processing in sequence. First, the information processing apparatus 100 executes the same processing as that from steps S1 to S3 shown in FIG. 1. The processing from steps S1 to S3 is the same as the above-described processing, so the explanation is omitted.

[0020] Next, when the information processing apparatus 100 estimates that the user's operation is an error, it determines an incorrect operation mode indicating the mode of the user's operation error, and when it is determined that the user has transitioned to an incorrect page with respect to the user's incorrect operation mode, it generates proposal information including a browser back button for returning to the original page (step S4-1). For example, when the state of the content at the time of receiving the user's operation information is a state in which the browser is displayed, and the user's operation information indicates the selection of a link to another page different from the page displayed on the browser, and even though the user terminal 200 displays another page according to the operation information, it is assumed that the estimation unit 132 estimates that the user's operation is an error. In this case, the information processing apparatus 100 determines that the user has transitioned to an incorrect page with respect to the user's incorrect operation mode. Then, the information processing apparatus 100 generates proposal information including a browser back button for returning to the original page.

[0021] Next, the information processing apparatus 100 executes the same processing as step S5 shown in FIG. 1. The processing of step S5 is the same as the above-described processing, so the explanation is omitted.

[0022] Thereby, when the browser is displayed and the user has transitioned to an incorrect page different from the user's intention, the back button for returning to the original page can be displayed on the user terminal 200.

[0023] [1-3. Another example of information processing according to the embodiment 2] If the information processing device 100 estimates that the user's operation was incorrect, it determines the type of error that indicates the nature of the user's operation error. If it determines that the user selected the wrong link among several links based on the type of error, it generates suggestion information that includes the link displayed next to the link selected by the user.

[0024] This information processing will be explained step by step. First, the information processing device 100 performs the same processing as steps S1 to S3 shown in Figure 1. Since the processing from steps S1 to S3 is the same as the processing described above, the explanation will be omitted.

[0025] Next, if the information processing device 100 estimates that the user's operation is incorrect, it determines the type of error that indicates the nature of the user's operation error. If it determines that the user selected the wrong link from among multiple links, it generates suggestion information that includes the link displayed next to the link selected by the user (step S4-2). For example, suppose the content state at the time of receiving the user's operation information is a state in which multiple links to a different page from the page displayed in the browser are densely displayed, and the user's operation information indicates the selection of one of the multiple links, and the user terminal 200 displays the other page corresponding to the selected link according to the operation information, but in step S3 the information processing device 100 estimates that the user's operation is incorrect. In this case, the information processing device 100 determines that the user selected the wrong link from among multiple links. The information processing device 100 then generates suggestion information that includes the link displayed closest to the link selected by the user.

[0026] Next, the information processing device 100 performs the same process as in step S5 shown in Figure 1. Since the process in step S5 is the same as the process described above, its explanation will be omitted.

[0027] This allows suggestion information, including links displayed next to the selected link, to be displayed on the user terminal 200 if the user selects the wrong link from among several links.

[0028] [1-4. Other Examples of Information Processing According to the Embodiment 3] When the information processing device 100 estimates that the user's operation is incorrect, it determines the type of error that indicates the nature of the user's operation error. Based on the type of error, if it is estimated that the user entered the wrong key when inputting characters, it generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key.

[0029] This information processing will be explained step by step. First, the information processing device 100 performs the same processing as steps S1 to S3 shown in Figure 1. Since the processing from steps S1 to S3 is the same as the processing described above, the explanation will be omitted.

[0030] Next, if the information processing device 100 estimates that the user's operation is incorrect, it determines the type of error that indicates the nature of the user's operation error. Based on the user's operation error, if it estimates that the user entered the wrong key when inputting characters, it generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key (step S4-3). For example, suppose the user is inputting the characters "gaze" and, when selecting the last character "n" in "gaze", makes the mistake of selecting "ru". Then, in step S3, the information processing device 100 estimates that the user's operation is incorrect, and the information processing device 100 determines that the user entered the wrong key when inputting characters. In this case, the information processing device 100 generates suggestion information that includes predictive conversion candidates for "gaze" that are inferred from the characters input up to "shise".

[0031] Next, the information processing device 100 performs the same process as in step S5 shown in Figure 1. Since the process in step S5 is the same as the process described above, its explanation will be omitted.

[0032] This allows the system to suggest predictive text suggestions that do not take into account the most recent key pressed when a user enters an incorrect key during text input.

[0033] [2. Configuration of the Information Processing System] Next, the configuration of the information processing system according to the embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the information processing system according to the embodiment. As shown in Figure 2, the information processing system 1 includes an information processing device 100, a user terminal 200, and a network N. Note that the information processing system 1 shown in Figure 2 may be configured to include multiple user terminals 200.

[0034] The information processing device 100 may be, for example, a personal computer (PC), a workstation (WS), or a computer equipped with server functions. The information processing device 100 processes information transmitted from the user terminal 200 via the network N.

[0035] The user terminal 200 is an information processing device used by the user. The user terminal 200 may be, for example, a smartphone, tablet, desktop PC, notebook PC, mobile phone, or PDA (Personal Digital Assistant). In the example shown in Figure 1, the user terminal 200 is a smartphone.

[0036] Network N connects the information processing device 100 and the user terminal 200 so that they can communicate with each other via wired or wireless connection. If Network N is wired, it may be implemented using Ethernet (registered trademark) as defined in IEEE 802.3. If Network N is wireless, it may be implemented using a wireless LAN (Local Area Network) as defined in IEEE 802.11.

[0037] [3. Configuration of the Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described using Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing device according to the embodiment. As shown in Figure 3, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Although not shown in Figure 3, the information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.

[0038] (Regarding Communications Unit 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with the user terminal 200.

[0039] (Regarding memory unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks, SSDs (Solid State Drives), and optical discs. As shown in Figure 3, the storage unit 120 includes an operation information storage unit 121, a model storage unit 122, and a content storage unit 123.

[0040] (Regarding the operation information storage unit 121) The operation information storage unit 121 stores user operation information obtained from the user terminal 200. Here, an example of the information stored in the operation information storage unit 121 will be explained using Figure 4. Figure 4 is a diagram showing an example of information stored in the operation information storage unit of the information processing device according to this embodiment.

[0041] In the example shown in Figure 4, the operation information storage unit 121 stores information related to the items "User ID" and "Operation Information" in an associated manner.

[0042] The "User ID" is an identifier that identifies a user and is represented by a string of characters or a number. The "Operation Information" is the user's operation information linked to the "User ID". When operation information is received via a touch panel, the operation information may include the touch duration for each finger of the user, touch movement speed (X axis, Y axis), touchdown position (X axis, Y axis), touchup position (X axis, Y axis), contact area, and the difference between touchdown and touchup positions. Furthermore, when operation information is received via an eye-tracking device, the operation information may include the gaze direction of the left and right eyeballs, and the position of the gaze points of the left and right eyeballs.

[0043] In other words, Figure 4 shows that the user operation information identified by the user ID "UID#1" is operation information "operation information #1".

[0044] Furthermore, the information stored in the operation information storage unit 121 is not limited to information related to the items "User ID" and "Operation Information," but may also store any other information related to the user's operations.

[0045] (Regarding the model memory unit 122) The model storage unit 122 stores a trained model that has learned the relationship between the correctness of the user's actions and the relationship between the user's gaze information and facial expression information during user operation. Figure 5 shows an example of the information stored in the model storage unit of the information processing device according to the embodiment.

[0046] In the example shown in Figure 5, the model storage unit 122 stores information related to the items "Model ID" and "Model Data" in an associated manner.

[0047] The "Model ID" is an identifier that identifies a machine learning model and is represented by a string, number, or other format. "Model Data" refers to the model data of the machine learning model. For example, the machine learning model may be a neural network.

[0048] In other words, in Figure 5, the model identified by model ID "M#1" represents the machine learning model M#1. Furthermore, the model data "MDT#1" represents the model data for machine learning model M#1.

[0049] In this case, if the machine learning model is a neural network, the model data "MDT#1" includes various information such as connection information, which describes how the nodes in each of the multiple layers that make up the neural network are connected to each other, and connection coefficients, which are multiplied by the numerical values ​​that are input and output between the connected nodes.

[0050] Furthermore, the model storage unit 122 is not limited to information related to the items "model ID" and "model data," but may store any other information related to the machine learning model.

[0051] (Regarding content storage unit 123) The content storage unit 123 stores multiple target contents to be displayed on the user terminal 200. Figure 6 is a diagram showing an example of information stored in the content storage unit of the information processing device according to this embodiment.

[0052] In the example shown in Figure 6, the content storage unit 123 stores information related to the items "Content ID" and "Content Data" in an associated manner.

[0053] The "Content ID" is an identifier that identifies the content and is represented by a string of characters, a number, or other means. The "Content Data" is the data of the content to be displayed on the user terminal 200.

[0054] In other words, Figure 6 shows that the content data indicated by content data "CD#1" is stored as content identified by content ID "CT#1".

[0055] Furthermore, the content storage unit 123 is not limited to information related to the items "Content ID" and "Content Data," but may store any other information related to the content.

[0056] (Regarding the control unit 130) Next, returning to Figure 3, the control unit 130 will be described. The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs stored in the memory of the information processing device 100 using RAM as the working area. Alternatively, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0057] As shown in Figure 3, the control unit 130 includes an acquisition unit 131, an estimation unit 132, a determination unit 133, a generation unit 134, and a display control unit 135.

[0058] (Regarding acquisition unit 131) The acquisition unit 131 acquires various types of information from inside or outside the information processing device 100.

[0059] For example, the acquisition unit 131 acquires operation information indicating the user's actions on the target content. If the user terminal 200 accepts user operations via a touch panel, the acquisition unit 131 acquires operation information including the touch duration, touch movement speed (X axis, Y axis), touch-down position (X axis, Y axis), touch-up position (X axis, Y axis), contact area, and the difference between the touch-down and touch-up positions. Furthermore, if the user terminal 200 accepts user operations via an eye-tracking device, the acquisition unit 131 acquires operation information including the gaze direction of the user's left and right eyes, and the positions of the points of fixation of the left and right eyes.

[0060] Furthermore, the acquisition unit 131 acquires the user's gaze information and facial expression information. Specifically, the acquisition unit 131 acquires the user's gaze information measured by the eye-tracking device of the input unit 230. The gaze information may include information such as the direction of gaze for each eye, the position of the point of fixation for each eye, the speed of gaze movement for each eye, and the pupil diameter for each eye. In addition, the acquisition unit 131 acquires image information of the user's facial expression captured by the imaging unit 250 of the user terminal 200 from the user terminal 200.

[0061] (Regarding Estimation Section 132) The estimation unit 132 estimates whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's operation of the target content. For example, the estimation unit 132 may use a pre-trained model that has learned the relationship between the correctness of the user's operation and the relationship between the user's gaze information and facial expression information during the user's operation to estimate whether the user's operation was incorrect based on the user's gaze information and facial expression information. In this case, the pre-trained model may be implemented using a CNN (Convolutional Neural Network) or the like. For example, the pre-trained model may be implemented by learning the relationship between the user's operation errors and features such as the gaze movement speed and pupil diameter of each eyeball as gaze information, and the corner of the mouth and eyebrow angle as facial expression information.

[0062] (Regarding the determination unit 133) The determination unit 133 determines the type of user error when the estimation unit 132 estimates that the user's operation was incorrect. In other words, the determination unit 133 determines the type of user error according to the state of the content at the time the corresponding user operation information was received when the estimation unit 132 estimates that the user's operation was incorrect.

[0063] For example, suppose the content state at the time of receiving user operation information is that the browser is displayed, and the user operation information indicates the selection of a link to a different page than the page displayed in the browser, and even though the user terminal 200 displays a different page according to the operation information, the estimation unit 132 estimates that the user's operation was incorrect. In this case, the determination unit 133 determines that the user transitioned to the wrong page.

[0064] Furthermore, for example, suppose the content state at the time of receiving user operation information is such that multiple links to a different page from the page displayed in the browser are densely displayed, and the user operation information indicates the selection of one of the multiple links, and the user terminal 200 displays the other page corresponding to the selected link according to the operation information, but the estimation unit 132 estimates that the user's operation was incorrect. In this case, the determination unit 133 determines that the user's incorrect operation was that the user selected the wrong link among the multiple links.

[0065] Furthermore, for example, suppose the content state when receiving user operation information is such that multiple input keys for character input keyboard are displayed, the user operation information indicates the selection of one of the multiple input keys, and the character corresponding to that input key is displayed on the user terminal 200 according to the operation information, but the estimation unit 132 estimates that the user's operation was incorrect. In this case, the determination unit 133 determines that the user entered the wrong key when inputting characters.

[0066] (Regarding the generation unit 134) The generation unit 134 generates suggestion information that indicates a suggestion to the user to restore the target content to its original state when the estimation unit 132 estimates that the user's operation was incorrect. Specifically, the generation unit 134 generates suggestion information according to the type of erroneous operation determined by the determination unit 133. The suggestion information generated by the generation unit 134 will be described below for each type of erroneous operation determined by the determination unit 133.

[0067] The generation unit 134 generates suggestion information, including a browser back button for returning to the original page, when the determination unit 133 determines that the user has navigated to the wrong page based on the user's error. For example, the generation unit 134 generates suggestion information that includes a back button represented by an object with an arrow pointing in the direction of the return, which is programmed to execute a process to return to the original page when selected by the user.

[0068] Furthermore, if the determination unit 133 determines that the user made an error in operation and that the user selected the wrong link from among multiple links, the generation unit 134 generates suggestion information that includes the link displayed closest to the link selected by the user. For example, the generation unit 134 identifies the location of the link selected by the user based on the operation information and identifies the nearest link that is closest to that link. The generation unit 134 then generates suggestion information that includes the nearest link object, along with a program that executes a process to transition to the page associated with the identified nearest link.

[0069] Furthermore, if the determination unit 133 determines that the user has entered the wrong key during character input, the generation unit 134 generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key. For example, suppose the user is entering the characters "gaze" and, when selecting the last character "ン" in "gaze", mistakenly selects "ル". The estimation unit 132 estimates that the user's operation is incorrect, and the determination unit 133 determines that the user has entered the wrong key during character input. In this case, the generation unit 134 cancels the input of the most recent key, "ル", and generates suggestion information that includes predictive conversion candidates for "gaze" that are inferred from the characters entered up to "シセ".

[0070] (Regarding the display control unit 135) The display control unit 135 controls the display on the output unit 240 of the user terminal 200.

[0071] For example, the display control unit 135 displays multiple target content items on the user terminal 200 that are the subject of the operation process. The display control unit 135 displays multiple content items, such as ebook pages, text boxes, video scroll bars, browser web page scroll bars and links, and CAD objects. However, the content displayed by the display control unit 135 is not limited to these, and may also include content utilizing VR (Virtual Reality) or MR (Mixed Reality) technologies.

[0072] Furthermore, the display control unit 135 displays the suggestion information generated by the generation unit 134 on the output unit 240 of the user terminal 200. For example, if the determination unit 133 determines that the user has made an error and selected the wrong link from among several links, the display control unit 135 displays suggestion information on the user terminal 200 that includes the link closest to the link selected by the user. Also, if the determination unit 133 determines that the user has made an error and transitioned to the wrong page, the display control unit 135 displays suggestion information on the user terminal 200 that includes the browser's back button to return to the original page. Furthermore, if the determination unit 133 determines that the user has made an error and entered the wrong key when entering text, the display control unit 135 displays suggestion information on the user terminal 200 that includes predictive conversion candidates that do not take into account the most recent input key.

[0073] [4. User terminal configuration] Next, the configuration of the user terminal 200 according to the embodiment will be described using Figure 7. Figure 7 is a diagram showing an example of the configuration of a user terminal according to the embodiment. As shown in Figure 7, the user terminal 200 has a communication unit 210, a storage unit 220, an input unit 230, an output unit 240, an imaging unit 250, and a control unit 260.

[0074] The communication unit 210 is implemented, for example, by a NIC (Network Interface Card). The communication unit 210 is connected to the network N by wire or wireless connection and transmits and receives various types of information to and from the information processing device 100 via the network N.

[0075] The storage unit 220 is a memory device that stores various types of information. For example, the storage unit 220 stores imaging data captured by the imaging unit 250. The storage unit 220 comprises a main memory and an auxiliary memory. The main memory may be implemented by semiconductor memory elements such as RAM, ROM, or flash memory. The auxiliary memory may be implemented by a hard disk, SSD, or optical disc, for example.

[0076] The input unit 230 receives various operation information from the user. For example, the input unit 230 accepts various operations from the user via a touch panel on a display surface (e.g., the output unit 240). Alternatively, the input unit 230 may be an eye-tracking device. An eye-tracking device analyzes the user's eye movements based on image information of the user's eyeballs captured by a stereo camera, determines the direction of the user's gaze, and accepts the user's point of fixation as operation information.

[0077] The output unit 240 is a display screen for a tablet terminal, for example, which is implemented using a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. In other words, if the input unit 230 of the user terminal 200 is a touch panel, the output unit 240, which is the display screen, accepts user input and also outputs to the user. The output unit 240 may also be a speaker and output sound through the speaker.

[0078] The imaging unit 250 is an imaging device that captures the facial expressions of the user. The imaging unit 250 may be, for example, a camera. A camera includes optical elements and an image sensor. Optical elements are elements that constitute an optical system, such as lenses, mirrors, prisms, and filters. An image sensor is an element that converts light incident through the optical elements into an image signal, which is an electrical signal. Examples of image sensors include CCD (Charge Coupled Device) sensors and CMOS (Complementary Metal Oxide Semiconductor) sensors.

[0079] The control unit 260 is implemented, for example, by a CPU or MPU executing various programs stored in the user terminal 200 using RAM as a working area. Alternatively, the control unit 260 may be implemented by an integrated circuit such as an ASIC or FPGA.

[0080] As shown in Figure 7, the control unit 260 includes a reception unit 261 and a display control unit 262. These components will be described in detail below.

[0081] The reception unit 261 receives operation information for content from the user. Alternatively, for example, the reception unit 261 can display multiple content items to be operated on the output unit 240 and receive operation information for multiple content items from the user. If the output unit 240 is a touch panel, the reception unit 261 may receive operation information for content from the user by sensing information such as the center point coordinates of the touch coordinates for each of the user's fingers, movement speed, movement direction, and contact area. If the input unit 230 is an eye-tracking device, the reception unit 261 may also receive information such as the gaze direction and fixation point of the user's left and right eyes as operation information.

[0082] The display control unit 262 controls the information displayed on the output unit 240. For example, the display control unit 262 controls the output unit 240 to display multiple target contents according to the control commands of the display control unit 135 of the information processing device 100. The display control unit 262 also controls the output unit 240 to display suggestion information generated by the generation unit 134 of the information processing device 100. In this case, the display control unit 262 may display the suggestion information overlaid on the original display.

[0083] [5. Information Processing Flow] Next, the information processing procedure by the information processing device 100 according to the embodiment will be described using Figure 8. Figure 8 is a flowchart of an example of information processing according to the embodiment. The information processing procedure according to the embodiment will be described below in accordance with the flowchart shown in Figure 8.

[0084] For example, the information processing device 100 displays multiple target content items that are the subject of the operation process on the user terminal 200 (step S101). The information processing device 100 then acquires operation information indicating the user's operation on the target content (step S102). The information processing device 100 then estimates whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's operation on the target content (step S103). If the information processing device 100 estimates that the user's operation was incorrect, it generates suggestion information indicating a suggestion to the user for restoring the state of the target content to its original state (step S104).

[0085] According to this, based on the user's gaze information and facial expression information, it is possible to estimate the user's operational errors and provide the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing method and information processing program that can accurately detect user operational errors and restore them to their original state.

[0086] [6. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 9. Figure 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

[0087] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic unit 1030 for various calculations and various databases, and is implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, etc.

[0088] Output IF1060 is an interface for transmitting information to be output to output devices 1010, which output various types of information such as monitors and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners. It is implemented using, for example, USB.

[0089] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may also be an external storage medium such as a USB memory stick.

[0090] Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N.

[0091] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0092] For example, when a computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040, thereby realizing the functions of the control unit 130 of the information processing device 100.

[0093] [7. Structure and Effects] The information processing device 100 according to this disclosure includes a display control unit 135 that displays a plurality of target contents that are the subject of operation processing on a user terminal 200, an acquisition unit 131 that acquires operation information indicating the user's operation on the target contents, an estimation unit 132 that estimates whether or not the user's operation was incorrect based on the user's gaze information and facial expression information when the user operates on the target contents, and a generation unit 134 that generates suggestion information indicating a suggestion to the user to return the state of the target contents to its original state when the estimation unit 132 estimates that the user's operation was incorrect.

[0094] This configuration allows the system to estimate user errors based on the user's gaze and facial expressions, and to provide the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing device 100 that can accurately detect user errors and restore it to its original state.

[0095] The estimation unit 132 of the information processing device 100 according to this disclosure estimates whether the user's operation is correct or incorrect using a trained model that has learned the relationship between the user's gaze information and the correctness of the operation on facial expression information.

[0096] This configuration allows for the estimation of whether a user's actions are correct or incorrect based on their gaze and facial expressions. Therefore, errors in user actions can be accurately detected.

[0097] The information processing device 100 according to this disclosure further comprises a determination unit 133 that determines a type of error indicating the nature of the user's operation error when the estimation unit 132 estimates that the user's operation is incorrect.

[0098] This configuration allows for the determination of the type of user error, and thus enables the execution of appropriate processing according to each type of error.

[0099] The generation unit 134 of the information processing device 100 according to this disclosure generates suggestion information, including a browser back button for returning to the original page, when the determination unit 133 determines that the user has navigated to the wrong page based on the user's error.

[0100] With this configuration, if the user navigates to an incorrect page unintended by the user while the browser is displayed, a back button can be displayed on the user terminal 200 to return to the original page. Therefore, it is possible to provide an information processing device 100 that can accurately detect user errors and restore the user to their original state.

[0101] The generation unit 134 of the information processing device 100 according to this disclosure generates suggestion information, including links displayed next to the link selected by the user, when the determination unit 133 determines that the user has selected the wrong link among multiple links.

[0102] With this configuration, if a user selects the wrong link from among several links, the user terminal 200 can be provided with suggestion information, including links displayed next to the link the user selected. Therefore, it is possible to provide an information processing device 100 that can accurately detect user errors and restore the system to its original state.

[0103] The generation unit 134 of the information processing device 100 according to this disclosure generates suggested information, including predictive conversion candidates that do not take into account the most recent input key, when the determination unit 133 estimates that the user entered the wrong key when inputting characters, based on the manner of the user's error.

[0104] This configuration allows for the suggestion of predictive text candidates that do not take into account the most recent key input when the user enters an incorrect key during character input. Therefore, it is possible to provide an information processing device 100 that can accurately detect user errors and restore the system to its original state.

[0105] Furthermore, the information processing method relating to this disclosure includes the steps of: displaying a plurality of target contents that are the subject of operation processing on the user terminal 200; acquiring operation information indicating the user's operation on the target contents; estimating whether the user's operation was incorrect based on the user's gaze information and facial expression information at the time of the user's operation on the target contents; and, if it is estimated that the user's operation was incorrect, generating suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state.

[0106] This configuration allows for the estimation of user errors based on the user's gaze and facial expressions, and provides the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing method that can accurately detect user errors and restore them to their original state.

[0107] Furthermore, the information processing program relating to this disclosure causes a computer to execute the following steps: display a plurality of target contents that are the subject of operation processing on the user terminal 200; acquire operation information indicating the user's operation on the target contents; estimate whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's operation on the target contents; and, if it is estimated that the user's operation was incorrect, generate suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state.

[0108] This configuration allows for the estimation of user errors based on the user's gaze and facial expressions, and provides the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing program that accurately detects user errors and restores it to its original state.

[0109] Although embodiments of the present application have been described in detail based on the drawings, these are illustrative examples, and the present invention can be implemented in various other forms, including those described in the disclosure section of the invention, based on the knowledge of those skilled in the art.

[0110] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit." For example, the acquisition unit 131 can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0111] 1. Information Processing System 100 Information Processing Devices 110 Communications Department 120 Storage section 121 Operation information storage unit 122 Model Memory Unit 123 Content Storage Unit 130 Control Unit 131 Acquisition Department 132 Estimation Department 133 Judgment section 134 Generation part 135 Display Control Unit 200 user terminals 210 Communications Department 220 Storage section 230 Input section 240 Output section 250 Imaging Unit 260 Control Unit 261 Reception Department 262 Display Control Unit N Network

Claims

1. A display control unit that displays multiple target content items that are the subject of the operation process on the user terminal, An acquisition unit that acquires operation information indicating the user's actions on the target content, An estimation unit that estimates whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's interaction with the target content, If the estimation unit estimates that the user's operation is incorrect, the generation unit generates suggestion information indicating a suggestion to the user to restore the target content to its original state. An information processing device equipped with the following features.

2. The estimation unit estimates the correctness of the user's actions using a pre-trained model that has learned the relationship between the correctness of the user's eye gaze information and facial expression information. The information processing apparatus according to claim 1.

3. The system further includes a determination unit that determines a type of user error when the estimation unit estimates that the user's operation is incorrect, indicating the nature of the user's operation error. The information processing apparatus according to claim 1 or claim 2.

4. The generation unit generates suggestion information, including a browser back button to return to the original page, when the determination unit determines that the user has navigated to the wrong page based on the user's error. The information processing apparatus according to claim 3.

5. The generation unit generates suggestion information, including links displayed next to the link selected by the user, when the determination unit determines that the user has selected the wrong link among multiple links. The information processing apparatus according to claim 3.

6. The generation unit generates suggestion information, including predictive conversion candidates that do not take into account the most recent input key, when the determination unit estimates that the user entered the wrong key when inputting text. The information processing apparatus according to claim 3.

7. The steps include displaying multiple target content items that are the subject of the operation process on the user's terminal, A step to obtain operation information indicating the user's actions on the target content, A step to estimate whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's interaction with the target content, The process includes generating suggestion information that indicates suggestions to the user on how to restore the target content to its original state, when it is estimated that the user's operation was incorrect, and Information processing methods including

8. The steps include displaying multiple target content items that are the subject of the operation process on the user's terminal, A step to obtain operation information indicating the user's actions on the target content, A step to estimate whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's interaction with the target content, The process includes generating suggestion information that indicates suggestions to the user on how to restore the target content to its original state, when it is estimated that the user's operation was incorrect, and An information processing program that causes a computer to execute something.

Images