Information processing device, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program configured to accurately detect a false change in display mode and restore the state to a former state.SOLUTION: An information processing apparatus includes: a display control unit which causes a user terminal to display multiple target contents which are to be operated; an acquisition unit which acquires operation information indicating an operation of a user with respect to the target contents; an estimation unit which estimates whether or not the change in display mode of the target contents is false on the basis of line-of-sight information and expression information of the user operating the target contents; and a generation unit which generates, when the estimation unit estimates that the change in display mode of the target contents is false, suggestion information indicating a suggestion to the user for restoring the state of the target contents to a former state.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Techniques for analyzing a user's eye movement to grasp the line-of-sight direction and recognizing the position of a fixation point at the tip of the line of sight are known. For example, Patent Document 1 below discloses a visual function detection device that appropriately examines the visual function of a subject. The device includes a display unit that displays a judgment image, moves the judgment image within a display area, detects the position of the fixation point of the subject observing the display unit, and detects the visual function of the subject based on the relationship between the movement direction of the judgment image and the movement direction of the fixation point.

[0003] Techniques for applying such a line-of-sight recognition technique to an HMI (Human Machine Interface) and accepting the designation of an object displayed on a display as a GUI (Graphical User Interface) are known. This makes it possible to grasp the operation intention of a person with a severe disability who has difficulty using means for accepting user operations by recognizing physical movements such as a keyboard or a mouse, and to accept operations on devices such as personal computers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in user interfaces (UIs) using eye-tracking technology, the user may not have intended to select or manipulate the object they are focusing their gaze on. The computer may select or manipulate an object that is contrary to the user's intent. Furthermore, if an object is selected or manipulated against the user's intent, it may not be possible to revert to the original state.

[0006] In view of the above issues, this disclosure aims to provide an information processing device, an information processing method, and an information processing program that can accurately detect errors in changes in the display manner of target content and restore it to its original state. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objectives, the information processing device according to this disclosure comprises: a display control unit that displays a plurality of target contents that are the subject of operation processing on a user terminal; an acquisition unit that acquires operation information indicating the user's operation on the target contents; an estimation unit that estimates whether or not a change in the display manner of the target contents was an error based on the user's gaze information and facial expression information during the user's operation on the target contents; and a generation unit that generates suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state when the estimation unit estimates that a change in the display manner of the target contents was an error. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide an information processing device, an information processing method, and an information processing program that can accurately detect errors in changes in the display manner of target content and restore it to its original state. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of information stored in the operation information storage unit of the information processing apparatus according to the embodiment. [Figure 5] Figure 5 shows an example of information stored in the model data storage unit of the information processing device according to the embodiment. [Figure 6] Figure 6 shows an example of information stored in the content storage unit of the information processing device according to this embodiment. [Figure 7] Figure 7 shows an example of the configuration of a user terminal according to this embodiment. [Figure 8] Figure 8 is a flowchart showing an example of information processing according to the embodiment. [Figure 9] Figure 9 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0010] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). However, these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application.

[0011] (Embodiment) [1-1. An example of information processing according to the embodiment] First, an example of information processing according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiment. In Figure 1, an example is shown in which the information processing according to the embodiment is executed by the information processing device 100 and the user terminal 200. Hereinafter, the information processing according to the embodiment will be described step by step.

[0012] First, the information processing device 100 displays multiple pieces of content to be operated on the user terminal 200 (step S1). For example, the information processing device 100 displays multiple pieces of content to be operated on the user terminal 200, including e-book pages, text boxes, video scroll bars, browser web page scroll bars and links, and CAD (Computer-Aided Design) objects, and receives operation information from the user. The state in which multiple pieces of content to be operated are displayed refers to, for example, a state in which multiple scroll bars that accept swipe operations are displayed, a state in which scroll bars and page screens that accept swipe operations are displayed on top of each other, a state in which multiple links to transition to other pages are displayed side by side, or a state in which a keyboard screen for text input is displayed with multiple input keys lined up.

[0013] Next, the information processing device 100 acquires user operation information, gaze information during operation, and facial expression information from the user terminal 200 (step S2). Here, operation information refers to information indicating the user's operation on the user terminal 200, and may be, for example, information on touch operations on a touch panel or information on the gaze point of an eye-tracking device. Gaze information refers to information about the user's gaze direction as measured by the eye-tracking device. Facial expression information refers to image information of the user's facial expression captured by the imaging unit 250 of the user terminal 200. For example, when user U performs an operation on the user terminal 200, the information processing device 100 acquires user U's operation information, user U's gaze information during operation, and facial expression information all at once.

[0014] Next, the information processing apparatus 100 changes the display target of the target content based on the operation information (step S3). For example, when the operation information indicates a rightward movement of the scroll bar of the video on the screen, the information processing apparatus 100 moves the scroll bar rightward and changes the display mode of the target content so that the playback time of the video advances to the time indicated by the scroll bar. Also, when the operation information indicates a click on a link in the browser, the information processing apparatus 100 changes the display mode of the target content to switch to the page associated with the link.

[0015] Next, the information processing apparatus 100 estimates whether the change in the display mode of the target content is an error based on the gaze information and the expression information during the user's operation (step S4). For example, the information processing apparatus 100 may estimate whether the change in the display mode of the target content is an error based on the gaze information and the expression information during the user's operation using a learned model that has learned the relationship between the change in the display mode of the target content and the gaze information and the expression information during the user's operation. The learned model in this case may be realized by a CNN (Convolutional Neural Network) or the like. For example, it may be realized by learning the relationship between features such as the gaze movement speed and pupil diameter for each of the left and right eyeballs as the gaze information, and the mouth corners and eyebrow angles as the expression information, and the error of the user's operation.

[0016] Next, when it is estimated that the change in the display mode of the target content is an error, the information processing apparatus 100 generates proposal information indicating a proposal to the user to return to the original state (step S5). For example, the information processing apparatus 100 determines an incorrect display mode indicating the mode of the change in the display mode of the target content, and generates proposal information indicating a proposal to the user to return to the original state according to the determined incorrect display mode.

[0017] Next, the information processing apparatus 100 causes the user terminal 200 to display the generated proposal information (step S6). For example, the information processing apparatus 100 causes the user terminal 200 to display proposal information indicating a proposal to the user for returning to the original state, and accepts the intention of the user regarding the process of returning to the original state. In this case, the information processing apparatus 100 may display the proposal information over the plurality of contents originally displayed on the user terminal 200.

[0018] Thereby, the information processing apparatus 100 can estimate an error in the change of the display mode of the target content based on the user's gaze information and expression information, and provide the user terminal 200 with proposal information for returning to the original state. Therefore, it is possible to provide the information processing apparatus 100 that can accurately detect an error in the change of the display mode of the target content and return it to the original state.

[0019] 〔1-2. Other Example 1 of Information Processing According to Embodiment〕 When the information processing apparatus 100 estimates that the change in the display mode of the target content is an error, it determines an error display mode indicating the mode of the error in the change of the display mode of the target content. When it is determined that the display mode of the target content has been changed so as to transition to an incorrect page regarding the error display mode of the target content, it generates proposal information including the back button of the browser for returning to the original page.

[0020] This information processing will be described step by step. First, the information processing apparatus 100 executes the same processing as steps S1 to S4 shown in FIG. 1. The processing from steps S1 to S4 is the same as the above-described processing, so the description thereof will be omitted.

[0021] Next, if the information processing device 100 estimates that the change in the display mode of the target content is incorrect, it determines a erroneous display mode indicating the erroneous change in the display mode of the target content. If it determines that the display mode of the target content has been changed to redirect to an incorrect page, it generates suggestion information including a browser back button to return to the original page (step S5-1). For example, suppose the state of the target content when the user's operation information is received is such that the browser is displayed, the user's operation information indicates the selection of a link to a different page than the page displayed in the browser, and the information processing device 100 displays a different page on the user terminal 200 according to the operation information, but in step S4 the information processing device 100 estimates that the change in the display mode of the target content is incorrect. In this case, the information processing device 100 determines that the display mode of the target content has been changed to redirect to an incorrect page. Then, the information processing device 100 generates suggestion information including a browser back button to return to the original page.

[0022] Next, the information processing device 100 performs the same process as in step S6 shown in Figure 1. Since the process in step S6 is the same as the process described above, its explanation will be omitted.

[0023] This allows a back button to be displayed on the user terminal 200 to return to the original page if the display of the target content is changed in a way that redirects the user to an incorrect page that is not what the user intended, while the browser is displayed.

[0024] [1-3. Another example of information processing according to the embodiment 2] If the information processing device 100 estimates that a change in the display manner of the target content is erroneous, it determines the erroneous display manner that indicates the erroneous change in the display manner of the target content. If it determines that the display manner of the target content has been changed to redirect to the page corresponding to the erroneous link among multiple links, it generates suggestion information including the link displayed next to the link selected by the user.

[0025] This information processing will be explained step by step. First, the information processing device 100 performs the same processing as steps S1 to S4 shown in Figure 1. Since the processing from steps S1 to S4 is the same as the processing described above, the explanation will be omitted.

[0026] Next, if the information processing device 100 estimates that the change in the display manner of the target content is incorrect, it determines the incorrect display manner that indicates the incorrect change in the display manner of the target content. If it determines that the display manner of the target content has been changed to transition to the page corresponding to the incorrect link among the multiple links, it generates suggestion information including the link displayed next to the link selected by the user (step S5-2). For example, suppose the state of the target content when the user's operation information is received is a state in which multiple links to a different page from the page displayed in the browser are densely displayed, and the user's operation information indicates the selection of one of the multiple links, and the information processing device 100 displays the other page corresponding to the selected link on the user terminal 200 according to the operation information, but in step S3 the information processing device 100 estimates that the change in the display manner of the target content is incorrect. In this case, the information processing device 100 determines that the display manner of the target content has been changed to transition to the page corresponding to the incorrect link among the multiple links. Then, the information processing device 100 generates suggestion information including the link displayed closest to the link selected by the user.

[0027] Next, the information processing device 100 performs the same process as in step S6 shown in Figure 1. Since the process in step S6 is the same as the process described above, its explanation will be omitted.

[0028] This allows suggestion information, including links displayed next to the selected link, to be displayed on the user terminal 200 if the display of the target content changes and the user is redirected to a page corresponding to an incorrect link among multiple links.

[0029] [1-4. Other Examples of Information Processing According to the Embodiment 3] The information processing device 100, when it is estimated that the display mode of the target content has been changed to display predictive conversion candidates for characters corresponding to the incorrect key pressed during character input, generates proposed information that includes predictive conversion candidates that do not take into account the most recent input key.

[0030] This information processing will be explained step by step. First, the information processing device 100 performs the same processing as steps S1 to S4 shown in Figure 1. Since the processing from steps S1 to S4 is the same as the processing described above, the explanation will be omitted.

[0031] Next, if the information processing device 100 estimates that the display mode of the target content has been changed to display a predictive conversion candidate for a character corresponding to an incorrect key pressed during character input, it generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key (step S5-3). For example, suppose a user is typing the characters "gaze" and, when selecting the last character "n" in "gaze", makes a mistake and selects "ru". Then, in step S4, the information processing device 100 estimates that the change in the display mode of the target content is incorrect, and determines that the display mode of the target content has been changed to display a predictive conversion candidate for a character corresponding to an incorrect key pressed during character input. In this case, the information processing device 100 cancels the display of the predictive conversion candidate corresponding to the most recent input key, "ru", and generates suggestion information that includes a predictive conversion candidate for "gaze" inferred from the characters entered up to "shise".

[0032] Next, the information processing device 100 performs the same process as in step S6 shown in Figure 1. Since the process in step S6 is the same as the process described above, its explanation will be omitted.

[0033] This allows for the suggestion of predictive text candidates that do not take into account the most recent input key when the display of the target content changes and predictive text candidates that consider the incorrect key pressed during text input are displayed.

[0034] [2. Configuration of the Information Processing System] Next, the configuration of the information processing system according to the embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the information processing system according to the embodiment. As shown in Figure 2, the information processing system 1 includes an information processing device 100, a user terminal 200, and a network N. Note that the information processing system 1 shown in Figure 2 may be configured to include multiple user terminals 200.

[0035] The information processing device 100 may be, for example, a personal computer (PC), a workstation (WS), or a computer equipped with server functions. The information processing device 100 processes information transmitted from the user terminal 200 via the network N.

[0036] The user terminal 200 is an information processing device used by the user. The user terminal 200 may be, for example, a smartphone, tablet, desktop PC, notebook PC, mobile phone, or PDA (Personal Digital Assistant). In the example shown in Figure 1, the user terminal 200 is a smartphone.

[0037] Network N connects the information processing device 100 and the user terminal 200 so that they can communicate with each other via wired or wireless connection. If Network N is wired, it may be implemented using Ethernet (registered trademark) as defined in IEEE 802.3. If Network N is wireless, it may be implemented using a wireless LAN (Local Area Network) as defined in IEEE 802.11.

[0038] [3. Configuration of the Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described using Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing device according to the embodiment. As shown in Figure 3, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Although not shown in Figure 3, the information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.

[0039] (Regarding Communications Unit 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with the user terminal 200.

[0040] (Regarding memory unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks, SSDs (Solid State Drives), and optical discs. As shown in Figure 3, the storage unit 120 includes an operation information storage unit 121, a model storage unit 122, and a content storage unit 123.

[0041] (Regarding the operation information storage unit 121) The operation information storage unit 121 stores user operation information obtained from the user terminal 200. Here, an example of the information stored in the operation information storage unit 121 will be explained using Figure 4. Figure 4 is a diagram showing an example of information stored in the operation information storage unit of the information processing device according to this embodiment.

[0042] In the example shown in Figure 4, the operation information storage unit 121 stores information related to the items "User ID" and "Operation Information" in an associated manner.

[0043] The "User ID" is an identifier that identifies a user and is represented by a string of characters or a number. The "Operation Information" is the user's operation information linked to the "User ID". When operation information is received via a touch panel, the operation information may include the touch duration for each finger of the user, touch movement speed (X axis, Y axis), touchdown position (X axis, Y axis), touchup position (X axis, Y axis), contact area, and the difference between touchdown and touchup positions. Furthermore, when operation information is received via an eye-tracking device, the operation information may include the gaze direction of the left and right eyeballs, and the position of the gaze points of the left and right eyeballs.

[0044] In other words, Figure 4 shows that the operation information of the user identified by the user ID "UID#1" is operation information "operation information #1".

[0045] Furthermore, the information stored in the operation information storage unit 121 is not limited to information related to the items "User ID" and "Operation Information," but may also store any other information related to the user's operations.

[0046] (Regarding the model memory unit 122) The model storage unit 122 stores a trained model that has learned the relationship between the changes in the display manner of the target content in relation to the user's gaze information and facial expression information during operation. Figure 5 is a diagram showing an example of the information stored in the model storage unit of the information processing device according to the embodiment.

[0047] In the example shown in Figure 5, the model storage unit 122 stores information related to the items "Model ID" and "Model Data" in an associated manner.

[0048] The "Model ID" is an identifier that identifies a machine learning model and is represented by a string, number, or other format. "Model Data" refers to the model data of the machine learning model. For example, the machine learning model may be a neural network.

[0049] In other words, in Figure 5, the model identified by model ID "M#1" represents the machine learning model M#1. Furthermore, the model data "MDT#1" represents the model data for machine learning model M#1.

[0050] In this case, if the machine learning model is a neural network, the model data "MDT#1" includes various information such as connection information, which describes how the nodes in each of the multiple layers that make up the neural network are connected to each other, and connection coefficients, which are multiplied by the numerical values ​​that are input and output between the connected nodes.

[0051] Furthermore, the model storage unit 122 is not limited to information related to the items "model ID" and "model data," but may store any other information related to the machine learning model.

[0052] (Regarding content storage unit 123) The content storage unit 123 stores multiple pieces of content to be displayed on the user terminal 200. Figure 6 shows an example of information stored in the content storage unit of the information processing device according to this embodiment.

[0053] In the example shown in Figure 6, the content storage unit 123 stores information related to the items "Content ID" and "Content Data" in an associated manner.

[0054] The "Content ID" is an identifier that identifies the content and is represented by a string of characters, a number, or other means. The "Content Data" is the data of the content to be displayed on the user terminal 200.

[0055] In other words, Figure 6 shows that the content data indicated by content data "CD#1" is stored as content identified by content ID "CT#1".

[0056] Furthermore, the content storage unit 123 is not limited to information related to the items "Content ID" and "Content Data," but may store any other information related to the content.

[0057] (Regarding the control unit 130) Next, returning to Figure 3, the control unit 130 will be described. The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs stored in the memory of the information processing device 100 using RAM as the working area. Alternatively, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0058] As shown in Figure 3, the control unit 130 includes an acquisition unit 131, an estimation unit 132, a determination unit 133, a generation unit 134, and a display control unit 135.

[0059] (Regarding acquisition section 131) The acquisition unit 131 acquires various types of information from inside or outside the information processing device 100.

[0060] For example, the acquisition unit 131 acquires operation information indicating the user's actions on the target content. If the user terminal 200 accepts user operations via a touch panel, the acquisition unit 131 acquires operation information including the touch duration, touch movement speed (X axis, Y axis), touch-down position (X axis, Y axis), touch-up position (X axis, Y axis), contact area, and the difference between the touch-down and touch-up positions. Furthermore, if the user terminal 200 accepts user operations via an eye-tracking device, the acquisition unit 131 acquires operation information including the gaze direction of the user's left and right eyes, and the positions of the points of fixation of the left and right eyes.

[0061] Furthermore, the acquisition unit 131 acquires the user's gaze information and facial expression information. Specifically, the acquisition unit 131 acquires the user's gaze information measured by the eye-tracking device of the input unit 230. The gaze information may include information such as the direction of gaze for each eye, the position of the point of fixation for each eye, the speed of gaze movement for each eye, and the pupil diameter for each eye. In addition, the acquisition unit 131 acquires image information of the user's facial expression captured by the imaging unit 250 of the user terminal 200 from the user terminal 200.

[0062] (Regarding Estimation Section 132) The estimation unit 132 estimates whether the user's operation was incorrect based on the user's gaze information and facial expression information during the user's operation of the target content. For example, the estimation unit 132 may use a pre-trained model that has learned the relationship between the user's gaze information, facial expression information, and the correctness of the user's operation to estimate whether the user's operation was incorrect based on the user's gaze information and facial expression information. In this case, the pre-trained model may be implemented using a CNN (Convolutional Neural Network) or the like. For example, the pre-trained model may be implemented by learning the relationship between features such as the gaze movement speed and pupil diameter of each eyeball as gaze information, and the corner of the mouth and eyebrow angle as facial expression information, and errors in the user's operation.

[0063] (Regarding the determination unit 133) The determination unit 133 determines the erroneous display pattern of the target content when the estimation unit 132 estimates that the change in the display pattern of the target content is erroneous. In other words, the determination unit 133 determines the erroneous display pattern of the target content according to the state of the target content at the time the corresponding user operation information is received when the estimation unit 132 estimates that the change in the display pattern of the target content is erroneous.

[0064] For example, suppose the state of the target content when user operation information is received is that the browser is displayed, and the user operation information indicates the selection of a link to a different page than the page displayed in the browser, and even though the user terminal 200 displays a different page according to the operation information, the estimation unit 132 estimates that the change in the display mode of the target content is incorrect. In this case, the determination unit 133 determines that the incorrect display mode of the target content was changed to cause the user to transition to an incorrect page.

[0065] Furthermore, for example, suppose the state of the target content when user operation information is received is such that multiple links to a different page from the page displayed in the browser are densely displayed, and the user operation information indicates the selection of one of the multiple links, and the user terminal 200 displays the other page corresponding to the selected link according to the operation information, but the estimation unit 132 estimates that the change in the display manner of the target content is incorrect. In this case, the determination unit 133 determines that the display manner of the target content has been changed to transition to the page corresponding to the incorrect link among the multiple links.

[0066] Furthermore, for example, suppose the state of the target content when receiving user operation information is such that multiple input keys for character input keyboard are displayed, the user operation information indicates the selection of one of the multiple input keys, and the user terminal 200 displays the character corresponding to that input key according to the operation information, but the estimation unit 132 estimates that the change in the display mode of the target content is incorrect. In this case, the determination unit 133 determines that the display mode of the target content has been changed to display a predicted conversion candidate for the character corresponding to the incorrect key during character input.

[0067] (Regarding the generation unit 134) The generation unit 134 generates suggestion information that indicates suggestions to the user for restoring the state of the target content to its original state when the estimation unit 132 estimates that the change in the display manner of the target content is erroneous. Specifically, the generation unit 134 generates suggestion information according to the erroneous display manner determined by the determination unit 133. The suggestion information generated by the generation unit 134 will be described below for each erroneous display manner determined by the determination unit 133.

[0068] The generation unit 134 generates suggestion information, including a browser back button for returning to the original page, when the determination unit 133 determines that the display mode of the target content has been changed to cause the user to transition to the wrong page. For example, the generation unit 134 generates suggestion information that includes a back button represented by an object with an arrow pointing in the direction of the return, which is programmed to execute a process to return to the original page when selected by the user.

[0069] Furthermore, if the determination unit 133 determines that the display mode of the target content has been changed to redirect the user to the page corresponding to the incorrect link among multiple links, the generation unit 134 generates suggestion information that includes the link displayed next to the link selected by the user. For example, the generation unit 134 identifies the position of the link selected by the user based on the operation information and identifies the nearest link that is closest to that link. The generation unit 134 then generates suggestion information that includes the nearest link object, along with a program that executes a process to redirect to the page associated with the identified nearest link.

[0070] Furthermore, if the determination unit 133 estimates that the display mode of the target content has been changed to display a predictive conversion candidate for a character corresponding to an incorrect key pressed during character input, the generation unit 134 generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key. For example, suppose a user is typing the characters "gaze" and, when selecting the last character "ン" in "gaze", mistakenly selects "ル". Then, the estimation unit 132 estimates that the change in the display mode of the target content is incorrect, and the determination unit 133 determines that the display mode of the target content has been changed to display a predictive conversion candidate for a character corresponding to an incorrect key pressed during character input. In this case, the generation unit 134 cancels the display of the predictive conversion candidate corresponding to the most recent input key, "ル", and generates suggestion information that includes a predictive conversion candidate for "gaze" inferred from the characters entered up to "シセ".

[0071] (Regarding the display control unit 135) The display control unit 135 controls the display on the output unit 240 of the user terminal 200.

[0072] For example, the display control unit 135 displays multiple target content items on the user terminal 200 that are the subject of the operation process. The display control unit 135 displays multiple content items, such as ebook pages, text boxes, video scroll bars, browser web page scroll bars and links, and CAD objects. However, the content displayed by the display control unit 135 is not limited to these, and may also include content utilizing VR (Virtual Reality) or MR (Mixed Reality) technologies.

[0073] Furthermore, the display control unit 135 causes the output unit 240 of the user terminal 200 to display the suggestion information generated by the generation unit 134. For example, if the determination unit 133 determines that the display mode of the target content has been changed to redirect to the wrong page, the display control unit 135 displays suggestion information on the user terminal 200 that includes the browser's back button for returning to the original page. If the determination unit 133 determines that the display mode of the target content has been changed to redirect to the page corresponding to the wrong link among multiple links, the display control unit 135 displays suggestion information on the user terminal 200 that includes the link displayed closest to the link selected by the user. Also, if the determination unit 133 determines that the display mode of the target content has been changed to display predictive conversion candidates for characters corresponding to the wrong key pressed during character input, the display control unit 135 displays suggestion information on the user terminal 200 that includes predictive conversion candidates that do not take into account the most recent input key.

[0074] [4. User terminal configuration] Next, the configuration of the user terminal 200 according to the embodiment will be described using Figure 8. Figure 8 is a diagram showing an example of the configuration of a user terminal according to the embodiment. As shown in Figure 8, the user terminal 200 has a communication unit 210, a storage unit 220, an input unit 230, an output unit 240, an imaging unit 250, and a control unit 260.

[0075] The communication unit 210 is implemented, for example, by a NIC (Network Interface Card). The communication unit 210 is connected to the network N by wire or wireless connection and transmits and receives various types of information to and from the information processing device 100 via the network N.

[0076] The storage unit 220 is a memory device that stores various types of information. For example, the storage unit 220 stores imaging data captured by the imaging unit 250. The storage unit 220 comprises a main memory and an auxiliary memory. The main memory may be implemented by semiconductor memory elements such as RAM, ROM, or flash memory. The auxiliary memory may be implemented by a hard disk, SSD, or optical disc, for example.

[0077] The input unit 230 receives various operation information from the user. For example, the input unit 230 accepts various operations from the user via a touch panel on a display surface (e.g., the output unit 240). Alternatively, the input unit 230 may be an eye-tracking device. An eye-tracking device analyzes the user's eye movements based on image information of the user's eyeballs captured by a stereo camera, determines the direction of the user's gaze, and accepts the user's point of fixation as operation information.

[0078] The output unit 240 is a display screen for a tablet terminal, for example, which is implemented using a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. In other words, if the input unit 230 of the user terminal 200 is a touch panel, the output unit 240, which is the display screen, accepts user input and also outputs to the user. The output unit 240 may also be a speaker and output sound through the speaker.

[0079] The imaging unit 250 is an imaging device that captures the facial expressions of the user. The imaging unit 250 may be, for example, a camera. A camera includes optical elements and an image sensor. Optical elements are elements that constitute an optical system, such as lenses, mirrors, prisms, and filters. An image sensor is an element that converts light incident through the optical elements into an image signal, which is an electrical signal. Examples of image sensors include CCD (Charge Coupled Device) sensors and CMOS (Complementary Metal Oxide Semiconductor) sensors.

[0080] The control unit 260 is implemented, for example, by a CPU or MPU executing various programs stored in the user terminal 200 using RAM as a working area. Alternatively, the control unit 260 may be implemented by an integrated circuit such as an ASIC or FPGA.

[0081] As shown in Figure 8, the control unit 260 includes a reception unit 261 and a display control unit 262. These components will be described in detail below.

[0082] The reception unit 261 receives operation information for target content from the user. Alternatively, for example, the reception unit 261 can display multiple target content items on the output unit 240 and receive operation information for multiple target content items from the user. If the output unit 240 is a touch panel, the reception unit 261 may receive operation information for target content from the user by sensing information such as the center point coordinates of the touch coordinates for each of the user's fingers, movement speed, movement direction, and contact area. If the input unit 230 is an eye-tracking device, the reception unit 261 may also receive information such as the gaze direction and fixation point of the user's left and right eyes as operation information.

[0083] The display control unit 262 controls the information displayed on the output unit 240. For example, the display control unit 262 controls the output unit 240 to display multiple target contents according to the control commands of the display control unit 135 of the information processing device 100. The display control unit 262 also controls the output unit 240 to display suggestion information generated by the generation unit 134 of the information processing device 100. In this case, the display control unit 135 may display the suggestion information overlaid on the original display.

[0084] [5. Information Processing Flow] Next, the information processing procedure by the information processing device 100 according to the embodiment will be described using Figure 8. Figure 8 is a flowchart of an example of information processing according to the embodiment. The information processing procedure according to the embodiment will be described below in accordance with the flowchart shown in Figure 8.

[0085] For example, the information processing device 100 displays multiple target content items that are the subject of the operation process on the user terminal 200 (step S101). The information processing device 100 then acquires operation information indicating the user's operation on the target content (step S102). The information processing device 100 then changes the display mode of the content based on the operation information (step S103). The information processing device 100 then estimates whether the change in the display mode of the target content was erroneous based on the user's gaze information and facial expression information during the user's operation on the target content (step S104). If the information processing device 100 estimates that the change in the display mode of the target content was erroneous, it generates suggestion information indicating a suggestion to the user for returning the state of the target content to its original state (step S105).

[0086] According to this, based on the user's gaze information and facial expression information, it is possible to estimate errors in the display mode of the target content and provide the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing device 100 that can accurately detect errors in the display mode of the target content and restore it to its original state.

[0087] [6. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 9. Figure 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

[0088] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic unit 1030 for various calculations and various databases, and is implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, etc.

[0089] Output IF1060 is an interface for transmitting information to be output to output devices 1010, which output various types of information such as monitors and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners. It is implemented using, for example, USB.

[0090] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may also be an external storage medium such as a USB memory stick.

[0091] Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N.

[0092] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0093] For example, when a computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040, thereby realizing the functions of the control unit 130 of the information processing device 100.

[0094] [7. Structure and Effects] The information processing device 100 according to this disclosure includes a display control unit 135 that displays a plurality of target contents that are the subject of operation processing on a user terminal 200, an acquisition unit 131 that acquires operation information indicating the user's operation on the target contents, an estimation unit 132 that estimates whether or not the change in the display manner of the target contents was an error based on the user's gaze information and facial expression information when the user operates on the target contents, and a generation unit 134 that generates suggestion information indicating a suggestion to the user to return the state of the target contents to its original state when the estimation unit 132 estimates that the change in the display manner of the target contents was an error.

[0095] This configuration allows the system to estimate errors in the display of target content based on the user's gaze and facial expressions, and to provide the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing device 100 that can accurately detect errors in the display of target content and restore it to its original state.

[0096] The estimation unit 132 of the information processing device 100 according to this disclosure estimates the correctness of the user's operation using a trained model that has learned the relationship between the correctness of changes in the display manner of the target content in relation to the user's gaze information and facial expression information.

[0097] This configuration allows for the estimation of whether a user's actions are correct or incorrect based on their gaze and facial expressions. Therefore, errors in user actions can be accurately detected.

[0098] The information processing device 100 according to this disclosure further comprises a determination unit 133 that determines a misdisplay pattern indicating a misdisplay pattern when the estimation unit 132 estimates that the change in the display pattern of the target content is erroneous.

[0099] This configuration allows for the determination of misdisplay patterns, which indicate errors in the display of the target content. Therefore, appropriate processing can be performed according to each misdisplay pattern.

[0100] The generation unit 134 of the information processing device 100 according to this disclosure generates suggestion information, including a browser back button for returning to the original page, when the determination unit 133 determines that the display manner of the target content has been changed to cause the user to transition to an incorrect page.

[0101] With this configuration, if the display of the target content is changed in a way that redirects the user to an incorrect page that does not match their intention while the browser is displayed, a back button can be displayed on the user terminal 200 to return to the original page. Therefore, it is possible to provide an information processing device 100 that can accurately detect errors in the change in the display of the target content and restore it to its original state.

[0102] The generation unit 134 of the information processing device 100 according to this disclosure generates suggestion information, including the link displayed next to the link selected by the user, when the determination unit 133 determines that the display manner of the target content has been changed to transition to the page corresponding to the incorrect link among multiple links.

[0103] With this configuration, if the display of the target content changes and transitions to a page corresponding to an incorrect link among multiple links, the user terminal 200 can be provided with suggestion information, including links displayed next to the link selected by the user. Therefore, it is possible to provide an information processing device 100 that can accurately detect errors in the display of the target content and restore it to its original state.

[0104] The generation unit 134 of the information processing device 100 according to this disclosure generates proposed information, including predictive conversion candidates that do not take into account the most recent input key, when the determination unit 133 determines that the display mode of the target content has been changed to display predictive conversion candidates for characters corresponding to the incorrect key pressed during character input.

[0105] With this configuration, if a change in the display mode of the target content causes the system to display predictive conversion candidates that take into account the incorrect key pressed during character input, it can suggest predictive conversion candidates that do not take into account the most recent input key. Therefore, it is possible to provide an information processing device 100 that can accurately detect errors in changes in the display mode of the target content and restore it to its original state.

[0106] Furthermore, the information processing method relating to this disclosure includes the steps of: displaying a plurality of target contents that are the subject of operation processing on the user terminal 200; acquiring operation information indicating the user's operation on the target contents; estimating whether or not the change in the display manner of the target contents was an error based on the user's gaze information and facial expression information during the user's operation on the target contents; and, if it is estimated that the change in the display manner of the target contents was an error, generating suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state.

[0107] This configuration allows for the estimation of errors in the display mode of the target content based on the user's gaze and facial expression information, and provides the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing method that can accurately detect errors in the display mode of the target content and restore it to its original state.

[0108] Furthermore, the information processing program relating to this disclosure causes the computer to execute the following steps: displaying multiple target contents that are the subject of operation processing on the user terminal 200; acquiring operation information indicating the user's operation on the target contents; estimating whether or not the change in the display manner of the target contents was an error based on the user's gaze information and facial expression information during the user's operation on the target contents; and, if it is estimated that the change in the display manner of the target contents was an error, generating suggestion information indicating a suggestion to the user for restoring the state of the target contents to its original state.

[0109] This configuration allows for the estimation of errors in the display of target content based on the user's gaze and facial expressions, and provides the user terminal 200 with suggested information to restore it to its original state. Therefore, it is possible to provide an information processing program that can accurately detect errors in the display of target content and restore it to its original state.

[0110] Although embodiments of the present application have been described in detail based on the drawings, these are illustrative examples, and the present invention can be implemented in various other forms, including those described in the disclosure section of the invention, based on the knowledge of those skilled in the art.

[0111] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit." For example, the acquisition unit 131 can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0112] 1. Information Processing System 100 Information Processing Devices 110 Communications Department 120 Storage section 121 Operation information storage unit 122 Model Memory Unit 123 Content Storage Unit 130 Control Unit 131 Acquisition Department 132 Estimation Department 133 Judgment section 134 Generation part 135 Display Control Unit 200 user terminals 210 Communications Department 220 Storage section 230 Input section 240 Output section 250 Imaging Unit 260 Control Unit 261 Reception Department 262 Display Control Unit N Network

Claims

1. A display control unit that displays multiple target content items that are the subject of the operation process on the user terminal, An acquisition unit that acquires operation information indicating the user's actions on the target content, An estimation unit that estimates whether or not the change in the display manner of the target content was erroneous, based on the user's gaze information and facial expression information during the user's interaction with the target content. If the estimation unit estimates that the change in the display mode of the target content is incorrect, the generation unit generates suggestion information that indicates a suggestion to the user for restoring the state of the target content to its original state. An information processing device equipped with the following features.

2. The estimation unit estimates the correctness of the user's operation using a pre-trained model that has learned the relationship between the correctness of changes in the display manner of the target content in relation to the user's gaze information and facial expression information. The information processing apparatus according to claim 1.

3. The system further includes a determination unit that determines a misdisplay pattern indicating the erroneous nature of the change in the display pattern of the target content when the estimation unit estimates that the change in the display pattern of the target content is erroneous. The information processing apparatus according to claim 1 or claim 2.

4. The generation unit generates suggestion information, including a browser back button for returning to the original page, when the determination unit determines that the display mode of the target content has been changed to cause the user to transition to an incorrect page. The information processing apparatus according to claim 3.

5. The generation unit generates suggestion information, including links displayed next to the link selected by the user, when the determination unit determines that the display mode of the target content has been changed to redirect to the page corresponding to the incorrect link among multiple links. The information processing apparatus according to claim 3.

6. The generation unit generates suggestion information that includes predictive conversion candidates that do not take into account the most recent input key, when the determination unit estimates that the display mode of the target content has been changed to display predictive conversion candidates for characters corresponding to the incorrect key pressed during character input. The information processing apparatus according to claim 3.

7. The steps include displaying multiple target content items that are the subject of the operation process on the user's terminal, A step to obtain operation information indicating the user's actions on the target content, A step of estimating whether the change in the display of the target content was erroneous based on the user's gaze information and facial expression information during the user's interaction with the target content, The process includes the step of generating suggestion information that indicates suggestions to the user on how to restore the target content to its original state when it is estimated that the change in the display manner of the target content is incorrect, and Information processing methods including

8. The steps include displaying multiple target content items that are the subject of the operation process on the user's terminal, A step to obtain operation information indicating the user's actions on the target content, A step of estimating whether the change in the display of the target content was erroneous based on the user's gaze information and facial expression information during the user's interaction with the target content, The process includes the step of generating suggestion information that indicates suggestions to the user on how to restore the target content to its original state when it is estimated that the change in the display manner of the target content is incorrect, and An information processing program that causes a computer to execute something.

Images