Information processing device, information processing method, and program

By using an estimation model trained through deep reinforcement learning, combined with changes in user physical state and information patterns, the problem of inaccurate attention estimation in existing technologies is solved, achieving accurate estimation of attention intensity and spatial range, thereby improving the relevance of information display and user experience.

CN118901055APending Publication Date: 2024-11-05SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029202.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-29
Filing Date
2023-02-14
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately estimate a user's attention level, particularly the intensity and spatial range of that attention, and cannot effectively distinguish whether a user is truly paying attention to the information on the screen.

Method used

By training an estimation model using deep reinforcement learning methods, and utilizing the user's physical state and response to changes in information patterns, the degree of change in the information output pattern is determined. Combined with visual, auditory, or tactile information types, accurate estimation of attention is achieved.

Benefits of technology

It can accurately estimate the user's attention intensity and spatial range, improve the effectiveness of information display and user experience, such as reducing the omission of important information in multi-user scenarios and improving the targeting of information display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901055A_ABST
    Figure CN118901055A_ABST
Patent Text Reader

Abstract

[Problem] To accurately estimate the intensity of attention and spatial distribution of attention of a user. [Solution] Provided is an information processing device provided with: an estimation unit that estimates a user's attention level with respect to a specific type of output information on the basis of the physical state of the user, said estimation unit estimating the attention level by using an estimation model, and outputting the estimated attention level to a specific type of output information on the basis of the estimated attention level; the estimation model is used for determining the change degree of the output mode of the output information based on the physical state of the trainee and the experience of whether the trainee perceives the change of the output mode when the output mode of the training information of the same type as the output information changes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0002] In recent years, technologies for estimating how much attention a user pays to output information have been developed. For example, Patent Document 1 discloses a technology for estimating a user's attention level based on changes in the user's eye movements relative to changes in displayed information.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-111291 Summary of the Invention

[0006] Problems to be solved by the present invention

[0007] However, it is difficult to say that the technology disclosed in Patent Document 1 can fully take into account the intensity of attention and the spatial range of attention.

[0008] Solution to the problem

[0009] One aspect of the present disclosure provides an information processing device including an estimation unit configured to estimate a user's attention level for a specific type of output information based on the user's physical state. The estimation unit estimates the attention level using an estimation model that determines a degree of change in an output pattern of the output information based on the trainee's physical state and experience of whether the trainee perceives a change in the output pattern of training information of the same type as the output information when the output pattern of the training information changes.

[0010] Another aspect of the present disclosure provides an information processing method, comprising estimating, using a processor, a user's attention level for a specific type of output information based on the user's physical state. The estimation model determines the degree of change in the output pattern of the output information based on the user's physical state and experience of whether the user perceives a change in the output pattern of training information of the same type as the output information when the output pattern of the training information changes.

[0011] In yet another aspect of the present disclosure, a program is provided that causes a computer to function as an information processing device, the information processing device including an estimation unit that estimates a user's level of attention to a specific type of output information based on the user's physical state. The estimation unit estimates the attention level using an estimation model that determines the degree of change in the output pattern of the output information based on the trainee's physical state and experience of whether the trainee perceives a change in the output pattern of training information of the same type as the output information when the output pattern of the training information changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 3 is a diagram for describing a change in the output mode of the training information 310 according to the embodiment and indicating the change in the output mode by the trainer 20 .

[0013] Figure 2 15 is a diagram for describing an outline of a learning cycle of the estimation model 155 according to the embodiment.

[0014] Figure 3 is a block diagram showing a configuration example of the information processing device 10 according to the embodiment.

[0015] Figure 4 is a sequence diagram illustrating an example of a process of performing deep reinforcement learning on the estimation model 155 according to an embodiment.

[0016] Figure 5 is a diagram for describing dummy information according to an embodiment.

[0017] Figure 6 is a sequence diagram illustrating an example of a process of calculating an attention level using the estimation model 155 and performing display control based on the attention level according to an embodiment.

[0018] Figure 7 is a diagram for describing an example of controlling a dynamic display of continuously estimated attention levels according to an embodiment.

[0019] Figure 8 is another diagram for describing an example of controlling a dynamic display of continuously estimated attention levels according to an embodiment.

[0020] Figure 9 is a diagram for describing an example of output control of content based on an attention level for each of a plurality of content according to an embodiment.

[0021] Figure 10 3 is a diagram for describing an example of display control of a display object based on an attention level according to an embodiment.

[0022] Figure 11 3 is a diagram for describing an example of output control based on the attention level of each of a plurality of users 30 according to an embodiment.

[0023] Figure 12 is a block diagram showing an example of the hardware configuration of the information processing apparatus 90 according to the embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in this specification and the drawings, constituent elements having substantially the same functional configuration will be denoted by the same reference numerals, and redundant descriptions thereof will be omitted.

[0025] Note that the description will be given in the following order.

[0026] 1. Implementation Method

[0027] 1.1. Overview

[0028] 1.2. System Configuration Example

[0029] 1.3. Process of Deep Reinforcement Learning for Estimation Model 155

[0030] 1.4. Process of calculating the attention level using the estimation model 155 and performing display control based on the attention level

[0031] 1.5. Specific Example of Display Control Based on Attention Level

[0032] 2. Hardware Configuration Example

[0033] 3. Conclusion

[0034] <1. Implementation Method>

[0035] <<1.1. Summary>>

[0036] First, an outline of an embodiment of the present disclosure will be described. In recent years, there has been a need to estimate how much attention a user pays to information being output (hereinafter referred to as output information).

[0037] For example, in an online meeting involving information display, it is necessary to understand what information displayed on the screen the user is paying attention to.

[0038] By accurately estimating the area to which the user is paying attention and the degree (intensity) of attention, for example, various advantages can be obtained, such as grasping information in which the user is interested and conversely grasping information ignored by the user.

[0039] Here, in the case where it is desired to simply grasp which point on the screen the user is looking at, various sight line estimation techniques may be used.

[0040] However, typical gaze estimation techniques only provide the coordinates of the end of the gaze. Furthermore, there are cases where a person is not paying attention to the object they are looking at, and cases where a person is paying close attention to an object even when they are not looking at it. Therefore, accurately estimating a user's attention level based solely on their gaze is extremely difficult.

[0041] Furthermore, Patent Document 1 discloses a technique for estimating a user's attention level based on the speed of the user's reaction to a change in information displayed at a certain position.

[0042] However, the technology disclosed in Patent Document 1 does not consider the degree of change that the user can notice and the area of ​​the change on the screen that the user can notice. Therefore, in the technology disclosed in Patent Document 1, it is difficult to accurately estimate the intensity of the user's attention and the spatial range of the attention.

[0043] Furthermore, as another method for estimating the user's attention level, for example, a method focusing on the user's gaze time can also be conceived.

[0044] However, in a case where the level of attention is defined as the more a user looks at an object, the more the user pays attention to the object, it is difficult to estimate the level of attention in a situation where the user is paying close attention to the object although the gaze time is short, or in a situation where the user is not looking at but capturing an object in the peripheral field of vision and paying attention to the object.

[0045] Alternatively, a method can also be conceived in which information indicating a specific state and correct data about the attention level of the user in the state are prepared, and the information and the correct data are learned in pairs.

[0046] However, it is difficult even for a user himself / herself to create correct data that quantitatively reflects the intensity of attention and the spatial range of attention.

[0047] The technical concept according to the embodiment of the present disclosure is conceived by focusing on the above-mentioned points, and can accurately estimate the intensity of the user's attention and the spatial range of the attention.

[0048] To this end, the information processing device 10 according to an embodiment of the present disclosure includes an estimation unit 150 that estimates a user's attention level for a specific type of output information based on the user's physical state. Furthermore, one of the features of the estimation unit 150 according to an embodiment of the present disclosure is that the attention level is estimated using an estimation model 155 that determines the degree of change in the output pattern of the output information based on the trainee's physical state and experience of whether the trainee perceives a change in the output pattern of the training information when the output pattern of the same type of training information as the output information changes.

[0049] Here, the specific type refers to the type of sense used to perceive the output information. That is, the output information according to this embodiment can be visual information, auditory information, tactile information, etc. Note that in the following description, the case where the output information is visual information (i.e., visually perceived information) will be described as the main example.

[0050] First, an overview of learning according to the present embodiment of the estimation model 155 will be described. For example, a deep reinforcement learning method using a neural network is used to learn the estimation model 155 according to the present embodiment.

[0051] As described above, the estimation model 155 according to the present embodiment is learned to determine the degree of change in the output pattern of the output information (training information) based on the trainee's physical condition and the trainee's experience of whether the trainee perceives the change in the output pattern of the training information when the output pattern of the training information changes.

[0052] For example, the estimation model 155 according to the present embodiment may alternatively be learned to determine, based on the above experience, a degree of change in the output pattern of the training information such that the trainee will not perceive the change in the output pattern of the training information.

[0053] Figure 1 3 is a diagram for describing a change in the output mode of the training information 310 according to the present embodiment and indicating the change in the output mode by the trainer 20 .

[0054] Notice, Figure 1 The output mode of the training information 310 is the output (display) position. However, the above is just an example, and when the output information is visual information, the output mode of the training information 310 includes presence / absence, shape, color, brightness, etc.

[0055] Figure 1 The upper portion of shows the training information 310 displayed on the display unit 180 and the trainer 20 holding the pointing unit 110 .

[0056] The pointing unit 110 according to this embodiment is used to point out the change in the output mode of the training information 310 (ie, report that the trainer 20 has perceived the change in the output mode of the training information 310 ) when the trainer 20 notices the change in the output mode of the training information 310 .

[0057] The pointing unit 110 according to the present embodiment may be, for example, a mouse or the like.

[0058] like Figure 1 As shown in the lower right portion of , for example, in a case where the trainer 20 notices a change in the output mode of the training information 310, the trainer 20 operates the pointing unit 110 to point out the change in the output mode.

[0059] On the other hand, Figure 1 As shown in the lower left portion of , in a case where the trainee does not notice the change in the output pattern of the training information 310, the trainee does not point out the change as described above.

[0060] The estimation model 155 according to the present embodiment is learned to determine the degree of change in the output pattern of the training information 310 based on the physical state of the trainee 20 and the presence or absence of indication when the output pattern of the training information 310 changes.

[0061] For example, in a case where the trainer 20 does not point out a change in the output pattern of the training information 310 (ie, the trainer 20 does not notice the change), a higher reward may be given to the estimation model 155 as the degree of change in the output pattern becomes higher.

[0062] exist Figure 1 In the case of the example shown, as the speed of change of the display position of the training information 310 (the moving speed on the display unit 180) becomes higher and the amount of change of the display position of the training information 310 (the moving distance on the display unit 180) becomes larger, a higher reward can be given to the estimation model 155.

[0063] Figure 2 15 is a diagram for describing an outline of a learning cycle of the estimation model 155 according to the present embodiment.

[0064] In the learning of the estimation model 155 according to the present embodiment, first, based on Figure 2 The estimation model 155 shown on the left side of the diagram determines the degree of change in the output pattern of the training information 310 to perform display control of the training information.

[0065] In a case where the trainer 20 perceives a change in the output mode of the training information 310 , the trainer 20 operates the pointing unit 110 to point out the change.

[0066] Furthermore, at this time, the state obtaining unit 120 obtains the sensor information 125 indicating the body information about the trainer 20 .

[0067] exist Figure 2 In the case of the illustrated example, the sensor information 125 may be an image obtained by capturing an image of the face of the trainee 20. Also in this case, the state obtaining unit 120 may be an imaging device.

[0068] Then, if Figure 2 As shown in the center of , the feature value extraction unit 130 extracts feature values ​​related to the physical condition of the trainee 20 from the sensor information 125 .

[0069] In the case where the sensor information 125 is an image obtained by capturing an image of the trainer's 20 face, the feature values ​​may include, for example, the position of the trainer's 20 eyes, line of sight, position of the face, orientation of the face, facial expression, posture, amplitude of movement, etc.

[0070] Then, if Figure 2 As shown on the right side of , the estimation model 155 determines the degree of change of the output pattern of the training information 310 based on the presence or absence indication of the input, the feature value, etc., and updates the estimation model 155.

[0071] Thereafter, based on the determined degree of change in the output pattern, display control of the training information 310 is performed, and the above-mentioned loop is repeatedly performed until the learning is completed.

[0072] As described above, one of the characteristics of the learning of the estimation model 155 according to the present embodiment is that the learning is performed while obtaining the trainer's 20 indication (feedback) of the change in the output pattern of the training information 310 in real time.

[0073] Using the above method, learning can be performed while associating the minimum degree of change in the output pattern of the training information 310 at which the trainee 20 can perceive the change (i.e., the intensity of attention) with the physical state of the trainee 20.

[0074] Furthermore, with the above-described method, learning can be performed while associating changes in the output pattern of the training information 310 in a specific area (ie, the spatial range of attention) with the physical state of the trainer 20 .

[0075] <<1.2 System Configuration Example>>

[0076] Next, a configuration example of a system according to the present embodiment will be described. The system according to the present embodiment includes an information processing apparatus 10 . Figure 3: is a block diagram showing a configuration example of the information processing device 10 according to the present embodiment.

[0077] like Figure 3 As shown, the information processing device 10 according to this embodiment may include a pointing unit 110, a state obtaining unit 120, a feature value extraction unit 130, an estimated position specifying unit 140, an estimating unit 150, a storage unit 160, an output control unit 170 and a display unit 180.

[0078] Furthermore, the information processing device 10 according to the present embodiment may be a personal computer (PC), a smartphone, a tablet, a head-mounted device, a game console, or the like.

[0079] (Pointing to unit 110)

[0080] The pointing unit 110 according to this embodiment is used to point out the change in the output mode of the training information 310 when the trainee 20 perceives the change in the output mode of the training information 310 .

[0081] The pointing unit 110 according to the present embodiment may be, for example, a mouse, a keyboard, a controller, a button, a joystick, a voice recognizer, a gesture recognizer, or the like.

[0082] (Status Acquisition Unit 120)

[0083] The state obtaining unit 120 according to the present embodiment obtains the physical state of the trainer 20 or the user 30. To this end, the state obtaining unit 120 according to the present embodiment includes various sensors.

[0084] Examples of the sensor include an imaging sensor, an infrared sensor, an acceleration sensor, a gyro sensor, a microphone, a pulse sensor, and the like.

[0085] In addition, the state obtaining unit 120 may obtain the environmental state of the trainer 20 or the user 30 .

[0086] Examples of environmental conditions include time, location, illumination, temperature, humidity, intensity of ambient sound, and the like.

[0087] The state obtaining unit 120 may further include a sensor for obtaining the above-mentioned environmental state.

[0088] (Feature Value Extraction Unit 130)

[0089] The feature value extraction unit 130 according to the present embodiment extracts feature values ​​from the sensor information on the physical state of the trainer 20 or the user 30 obtained by the state obtaining unit 120 .

[0090] The feature value extraction unit 130 according to the present embodiment may extract feature values ​​using a technique widely used for processing various types of sensor information.

[0091] In addition, in the case where the state obtaining unit 120 obtains sensor information about the environmental state, the feature value extraction unit 130 may also extract additional feature values ​​from the sensor information.

[0092] (Estimated Position Specifying Unit 140)

[0093] The estimated position specifying unit 140 according to the present embodiment specifies the position where the attention level is to be estimated. The estimated position specifying unit 140 may determine the estimated position according to a predetermined rule or may determine the estimated position according to an instruction from an operator different from the trainer 20.

[0094] (Estimation Unit 150)

[0095] The estimation unit 150 according to the present embodiment estimates the attention level of a user cheering for a specific type of output information based on the user's body state.

[0096] As described above, one of the features of the estimation unit 150 according to the present embodiment is to estimate the user's attention level using the estimation model 155 generated by deep reinforcement learning.

[0097] Furthermore, the estimation unit 150 according to the present embodiment may function as a learning unit that performs deep reinforcement learning on the estimation model 155 .

[0098] The details of the function of the estimation unit 150 according to the present embodiment will be described separately.

[0099] (Storage Unit 160)

[0100] The storage unit 160 according to the present embodiment stores information used by each of the components of the information processing apparatus 10 .

[0101] The storage unit 160 stores, for example, various types of information (network configuration, parameters, etc.) regarding the estimation model 155 .

[0102] (Output Control Unit 170)

[0103] The output control unit 170 according to the present embodiment controls the output of various types of information. For example, the output control unit 170 according to the present embodiment may control the output of output information based on the attention level estimated by the estimation unit 150.

[0104] A specific example of output control according to the present embodiment will be described separately.

[0105] (Display unit 180)

[0106] The display unit 180 according to the present embodiment displays visual information under the control of the output control unit 170. As described above, the visual information is an example of output information according to the present embodiment.

[0107] The display unit 180 according to the present embodiment may be a 2D display, a 3D display, a projector, or the like.

[0108] The configuration example of the information processing apparatus 10 according to the present embodiment has been described above. Figure 3 The described configuration is merely an example, and the configuration of the information processing apparatus 10 according to the present embodiment is not limited to this example.

[0109] For example, the information processing device 10 may further include an audio output unit that outputs auditory information, a tactile presentation unit that presents tactile information, and the like.

[0110] In addition, for example, although the above describes a case where the estimation unit 150 is also used as a training unit for performing deep reinforcement learning on the estimation model 155, the training unit for performing deep reinforcement learning on the estimation model 155 can also be clearly separated from the estimation unit for performing estimation using the estimation model 155.

[0111] In addition, each function of the above-mentioned information processing device 10 can be realized by cooperation of multiple devices. In this case, the multiple devices only need to be able to communicate information through the network and do not need to be set up in the same place.

[0112] For example, the estimation unit 150 may be included in a server, and the output control unit 170 may be included in a local computer.

[0113] The system configuration according to the present embodiment can be flexibly modified according to specifications, operations, and the like.

[0114] <<1.3. Process of Deep Reinforcement Learning for Estimation Model 155>>

[0115] Next, the process of performing deep reinforcement learning on the estimation model 155 according to this embodiment will be described in detail.

[0116] Figure 4 155 is a sequence diagram illustrating an example of a process of performing deep reinforcement learning on the estimation model 155 according to this embodiment.

[0117] exist Figure 4In the case of the illustrated example, first, during training by the trainer 20, the state obtaining unit 120 obtains sensor information on the physical state of the trainer 20, and the feature value extraction unit 130 extracts feature values ​​from the sensor information (S102).

[0118] Here, in the case where the state obtaining unit 120 obtains sensor information about the environmental state, the feature value extraction unit 130 may also extract additional feature values ​​from the sensor information.

[0119] The feature values ​​extracted by the feature value extraction unit 130 are input to the estimation model 155 included in the estimation unit 150 .

[0120] Next, the estimation model 155 included in the estimation unit 150 uses the feature values ​​extracted in step S102 and the estimated position specified by the estimated position specifying unit 140 as inputs, and determines (outputs) the degree of change in the output pattern of the output information ( S104 ).

[0121] That is, it can be said that the estimation model 155 according to the present embodiment determines the degree of change in the output pattern of the training information also based on the output (display) position of the training information.

[0122] At this time, since the feature values ​​extracted from the sensor information about the environmental state are input to the estimation model 155, the estimation model 155 can be learned while the intensity of attention and the spatial range of attention are associated not only with the physical state of the trainee 20 but also with the environmental state.

[0123] Furthermore, the estimation unit 150 may calculate the attention level based on the degree of change in the output pattern of the output information determined by the estimation model 155 .

[0124] For example, the estimation unit 150 according to the present embodiment calculates the attention level based on the degree of change in the output pattern of the output information determined by the estimation model 155 and the maximum amount of change associated with the output pattern.

[0125] Here, the maximum change amount refers to the maximum value of the degree of change that can be determined by the estimation model 155. The maximum change amount can be set according to, for example, the specifications of the display unit 180, the performance of the processor, and the like.

[0126] The attention level according to the present embodiment can be calculated using, for example, the following Expression (1) or Expression (2).

[0127] Attention level = {1-(the degree of change in the determined output mode) / (the maximum amount of change)}*100[%]……(1)

[0128] Attention level = {1-(the degree of change in the output mode determined) 2 / (maximum change) 2}*100[%]……(2)

[0129] However, the above expressions (1) and (2) are merely examples, and the expression for calculating the attention level according to the present embodiment is not limited to these examples. In addition, the attention level according to the present embodiment does not necessarily need to be expressed in percentage.

[0130] Next, the output control unit 170 performs display control of the training information 310 based on the degree of change in the output pattern of the output information determined by the estimation model 155 ( S106 ).

[0131] When the trainer 20 notices the change in the output mode, the trainer 20 operates the pointing unit 110 to change the state of the pointing unit 110 ( S108 ).

[0132] Next, information on the state of the pointing unit 110 is input to the estimation model 155, and the estimation model 155 stores the presence or absence of pointing by the trainee 20 as experience based on the information (S110).

[0133] Furthermore, the estimation model 155 is updated based on the above experience, and information on the estimation model 155 updated as needed is stored in the storage unit 160 ( S112 ).

[0134] The information processing apparatus 10 repeatedly executes the processing in steps S102 to S112 described above until learning is completed.

[0135] Here, updating of the estimation model 155 based on the experience of the presence or absence of indication according to the present embodiment will be described in more detail.

[0136] The estimation model 155 according to the present embodiment may determine the degree of change in the output pattern of the training information 310 using an objective function that differs depending on whether the trainee 20 perceives the change in the output pattern of the training information 310 .

[0137] That is, one of the characteristics of deep reinforcement learning according to this embodiment is that the parameter update rule for estimating the model 155 is switched according to the presence or absence of instruction by the trainer 20 .

[0138] In a case where the trainee 20 does not perceive the change in the output pattern of the training information 310, for example, the estimation model 155 according to the present embodiment can determine the degree of change in the output pattern of the training information 310 in a manner that maximizes an objective function corresponding to a positive number proportional to the degree of change in the output pattern of the training information 310.

[0139] In addition, when the trainee 20 perceives a change in the output pattern of the training information 310, for example, the estimation model 155 according to the present embodiment can determine the degree of change in the output pattern of the training information 310 in a manner that maximizes an objective function corresponding to a negative number proportional to the degree of change in the output pattern of the training information 310.

[0140] As a result of switching the objective function described above, the estimation model 155 is updated so that when there is a point of attention, the degree of change in the output pattern is reduced, and when there is no point of attention, the degree of change in the output pattern is increased. This makes it possible to accurately estimate the intensity of attention.

[0141] The process of deep reinforcement learning according to this embodiment has been described in detail. Figure 4 The described process is merely an example, and the process of deep reinforcement learning according to the present embodiment is not limited to this example.

[0142] In step S106, for example, the output control unit 170 according to the present embodiment may Figure 5 As shown, in addition to the training information 310, the output of dummy information 315 of the same type as the training information 310 (perceived through the same sense organ) is also controlled.

[0143] At this time, the output control unit 170 may, for example, randomly change the output mode of the virtual information 315. However, the trainer 20 can distinguish the training information 310 from the virtual information.

[0144] In the case where only training information 310 is output, the trainee 20 may be sensitive to even a slight change in the output pattern of training information 310. On the other hand, by outputting virtual information as described above, the trainee 20 needs to pay more attention to the training information 310, and the estimation accuracy of the attention level is expected to be improved.

[0145] <<1.4. Process of Calculating Attention Level Using Estimation Model 155 and Performing Display Control Based on the Attention Level>>

[0146] Next, a process of calculating the attention level using the estimation model 155 and performing display control based on the attention level according to the present embodiment will be described in detail.

[0147] Figure 6 15 is a sequence diagram illustrating an example of a process of calculating an attention level using the estimation model 155 and performing display control based on the attention level according to the present embodiment.

[0148] exist Figure 6In the case of the example shown, first, the startup is performed by referring to Figure 4 Describes the application of deep reinforcement learning-generated estimation models 155.

[0149] During use of the application, the state obtaining unit 120 obtains sensor information about the body state of the user 30 , and the feature value extracting unit 130 extracts feature values ​​from the sensor information ( S202 ).

[0150] Note that, in the case where sensor information on the environmental state is used at the time of learning, the sensor information on the environmental state is obtained in step S202 , and feature values ​​are extracted from the sensor information.

[0151] The feature values ​​extracted by the feature value extraction unit 130 are input to the estimation model 155 included in the estimation unit 150 .

[0152] Next, the estimation model 155 included in the estimation unit 150 uses the feature values ​​extracted in step S102 and the estimated position specified by the estimated position specifying unit 140 as input and determines (outputs) the degree of change in the output mode of the output information. In addition, the estimation unit 150 calculates the attention level of the user 30 based on the degree of change in the output mode (S204).

[0153] The estimation unit 150 can calculate the attention level using the above-described Expression (1) or Expression (2) or the like.

[0154] Next, the output control unit 170 performs display control based on the attention level calculated in step S204 ( S206 ), and the display unit 180 displays visual information reflecting the attention level of the user 30 according to control by the output control unit 170 ( S208 ).

[0155] <<1.5. Specific Example of Display Control Based on Attention Level>>

[0156] Next, display control based on the attention level of the user 30 according to the present embodiment will be described with reference to specific examples.

[0157] As described above, the output control unit 170 according to the present embodiment controls the output of the output information based on the attention level of the user 30 calculated by the estimation unit 150 .

[0158] For example, the output control unit 170 according to the present embodiment may control dynamic display of the attention level continuously estimated by the estimation unit 150 .

[0159] Figure 7 and Figure 8is a diagram for describing an example of control of dynamic display of continuously estimated attention levels according to the present embodiment.

[0160] Figure 7 An example of display control in a situation where the user 30 does not pay attention to the display unit 180 at all is shown.

[0161] exist Figure 7 and Figure 8 In the illustrated example, under the control of the output control unit 170 , 28 attention level visualization objects 320 in total are displayed in four rows and seven columns on the display unit 180 .

[0162] The output control unit 170 controls the display mode of each of the attention level visualization objects 320 based on the attention level at the corresponding position calculated by the estimation unit 150 .

[0163] For example, the output control unit 170 may perform control such that the attention level visualization object 320 is more prominently displayed as the corresponding calculated attention level becomes higher.

[0164] For example, Figure 8 The upper part of shows a display example in a case where the user 30 is paying attention to the upper left corner of the display unit 180 .

[0165] In this case, the output control unit 170 performs control such that the attention level visualization object 320 located in the upper left area of ​​the display unit 180 is highlighted based on the attention level calculated by the estimation unit 150 .

[0166] also, Figure 8 The middle portion of shows a display example in a case where the user 30 is paying attention to the upper right corner of the display unit 180 .

[0167] In this case, the output control unit 170 performs control such that the attention level visualization object 320 located in the upper right area of ​​the display unit 180 is highlighted based on the attention level calculated by the estimation unit 150 .

[0168] also, Figure 8 The lower part of shows a display example in a case where the user 30 is paying attention to the upper right corner of the display unit 180 .

[0169] In this case, the output control unit 170 performs control such that the attention level visualization object 320 located in the lower right area of ​​the display unit 180 is highlighted based on the attention level calculated by the estimation unit 150 .

[0170] Notice, Figure 8A case is shown where the output control unit 170 performs control such that the attention level visualizing object 320 becomes larger and the density of the dot pattern becomes higher as the corresponding attention level becomes higher.

[0171] However, Figure 8 The highlighting shown is merely an example, and the attention level visualization object 320 may be highlighted using color, brightness, repetition, blinking speed, etc.

[0172] With such highlighted display of the attention level visualization object 320 based on the attention level, the area to which the user 30 is paying attention can be displayed in real time, and the area to which the user 30 is paying attention can be intuitively grasped.

[0173] Note that the attention level visualization object 320 according to this embodiment may be superimposed on other types of visual information.

[0174] Furthermore, the output control unit 170 according to the present embodiment may control the output of each of a plurality of contents being output based on the attention level to the content.

[0175] Figure 9 1 is a diagram for describing an example of content output control based on an attention level for each of a plurality of contents according to the present embodiment.

[0176] Figure 9 An example of display control of the display unit 180 on the consultant side when online consultation is performed using the Web conference system is shown.

[0177] exist Figure 9 In the case of the illustrated example, the output control unit 170 performs display control based on the attention level of each of the plurality of contents in the first chart area 341 , the second chart area 342 , the text area 343 , the customer image area 344 , and the advisor image area 345 .

[0178] For example, Figure 9 The upper portion shows a situation in which the customer (user 30) pays most attention to the advisor image area 345. In this case, the output control unit 170 may perform control so that the outer edge of the advisor image area 345 is highlighted compared with the outer edges of other contents.

[0179] In addition, for example, Figure 9 The lower portion shows a situation in which the customer (user 30) pays most attention to the second chart area 342. In this case, the output control unit 170 may perform control so that the outer edge of the second chart area 342 is highlighted compared with the outer edges of other contents.

[0180] Note that the presentation of the attention level to each content is not limited to the highlighted display of the outer edge, and the attention level to each content may be presented as a change in one of various display modes including color, brightness, flickering, and the like.

[0181] In addition, despite Figure 9 A case where visibility is prioritized and only the content to which the customer (user 30 ) pays the most attention is highlighted is shown, but the output control unit 170 may simultaneously control display modes of a plurality of contents according to their corresponding attention levels.

[0182] In addition, the output control unit 170 may explicitly present the attention level to each content as a numerical value.

[0183] Use reference Figure 9 With the described display control, the explaining side (eg, consultant) can intuitively grasp which content the listening side (eg, client) is paying attention to, and can give explanations more effectively and efficiently.

[0184] Furthermore, the output control unit 170 according to the present embodiment may control, for example, the behavior of a display object (such as a character in a game application) based on the calculated attention level.

[0185] Figure 10 : is a diagram for describing an example of display control of a display object based on an attention level according to the present embodiment.

[0186] Figure 10 A case where a display object 330 corresponding to a certain character approaches another character operated by the user 30 in a game application is shown.

[0187] When the user 30 has a high attention level on the display object 330 (eg, when the attention level exceeds a threshold), for example, the output control unit 170 according to the present embodiment may perform control so that the display object 330 behaves as described above.

[0188] With the control as described above, even if the user 30 does not intentionally control his / her character so that the character approaches and talks with the character corresponding to the display object 330, the character corresponding to the display object 330 can be made to approach and talk with the character of the user 30.

[0189] On the contrary, in a case where the user 30 has a low attention level to the display object 330 (for example, in a case where the attention level is lower than a threshold value), for example, the output control unit 170 may perform control so that the display object 330 is expressed as described above.

[0190] By using the above-described control, the enemy character corresponding to the display object 330 can be brought close to the character controlled by the user 30 without the user noticing.

[0191] Furthermore, with the above-described control, situations such as the Statues game played around the world (eg, Red Light, Green Light in North America) can be easily realized.

[0192] Furthermore, the output control unit 170 according to the present embodiment can control the output position of output information related to notifications and the like, in particular, based on the attention level of the user 30 .

[0193] For example, when displaying an icon indicating that a mail with a high importance level or urgency level has been received, the output control unit 170 may display the icon in an area on the display unit 180 where the user 30 is paying attention (an area where the attention level exceeds a threshold).

[0194] Through the above control, the possibility of the user 30 ignoring information with a high importance level or a high urgency level can be effectively reduced.

[0195] Conversely, for example, in the case of displaying an icon indicating that an email with a low importance level or urgency level has been received, the output control unit 170 may display the icon in an area on the display unit 180 to which the user 30 is not paying attention (an area where the attention level is below a threshold).

[0196] With the above-described control, the possibility of causing the user 30 who is viewing another output information on the display unit 180 to feel uncomfortable, for example, by unnecessarily drawing the attention of the user 30 , can be effectively reduced.

[0197] Furthermore, the output control unit 170 according to the present embodiment may control the output of the output information based on the attention level of each of the plurality of users 30 .

[0198] Figure 11 3 is a diagram for describing an example of output control based on the attention level of each of the plurality of users 30 according to the present embodiment.

[0199] Figure 11 A high attention level region 371A and a low attention level region 372A of the user 30A and a high attention level region 371B and a low attention level region 372B of the user 30B on the display unit 180 are shown.

[0200] Here, the high attention level area 371A indicated by the dense dots may be an area where the attention level of the user 30A exceeds the first threshold. In addition, the high attention level area 371B indicated by the dense diagonal lines may be an area where the attention level of the user 30B exceeds the first threshold.

[0201] In addition, the low attention level area 372A indicated by the sparse dots may be an area where the attention level of the user 30A is lower than or equal to the first threshold and exceeds the second threshold. In addition, the low attention level area 372B indicated by the sparse diagonal lines may be an area where the attention level of the user 30B is lower than or equal to the first threshold and exceeds the second threshold.

[0202] In the case of displaying information that is important to both the users 30A and 30B under the above situation, the output control unit 170 may display the information in an area where the low attention level areas 372A and 372B overlap.

[0203] With the above-described control, the possibility that both users 30A and 30B overlook important information can be effectively reduced.

[0204] Note that, in the event that there is a region where the high attention regions 371A and 371B overlap, the output control unit 170 may instead display the information in that region.

[0205] On the other hand, when displaying information that is important to user 30A but not important to user 30B, output control unit 170 may display the information in high attention level area 371A. Similarly, when displaying information that is important to user 30B but not important to user 30A, output control unit 170 may display the information in high attention level area 371B.

[0206] With the above-described control, even when there are a plurality of users 30 , it is possible to notify only the target user 30 without disturbing the experience of non-target users 30 , for example.

[0207] The control of output information based on the attention level according to the present embodiment has been described above with reference to specific examples. Note that the output control unit 170 according to the present embodiment can perform various types of output control other than the examples described above.

[0208] For example, the output control unit 170 is also capable of performing control such as displaying an advertisement in an area where the attention level of the user 30 is high.

[0209] Furthermore, for example, the output control unit 170 can also perform display control for drawing attention to an area where the user 30 who is driving has low attention.

[0210] The output control based on the attention level according to the present embodiment can be flexibly modified according to the specifications of the application and the like.

[0211] Furthermore, the attention level calculated by the information processing method according to the present embodiment can be used as knowledge in UI design or the like.

[0212] Furthermore, although the case where the output information is visual information has been described above as a main example, the output information according to the present embodiment may instead be auditory information, tactile information, or the like.

[0213] With the information processing method according to the present embodiment, effective output control of auditory information or tactile information based on the attention intensity of the user 30 or the like can be achieved.

[0214] <2. Hardware Configuration Example>

[0215] Next, an example of the hardware configuration of the information processing device 90 according to an embodiment of the present disclosure will be described. Figure 12 is a block diagram illustrating an example of the hardware configuration of the information processing device 90 according to an embodiment of the present disclosure.

[0216] The information processing device 90 may be a device having a hardware configuration equivalent to that of the information processing device 10 according to the embodiment of the present disclosure.

[0217] like Figure 12 As shown, the information processing device 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a memory 880, a driver 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and a subset of the components may be omitted. In addition, components other than those shown here may be included.

[0218] (Processor 871)

[0219] The processor 871 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation of each component or a part thereof based on various programs stored in the ROM 872 , the RAM 873 , the storage 880 , or the removable storage medium 901 .

[0220] (ROM 872 and RAM 873)

[0221] The ROM 872 is a means for storing programs to be read by the processor 871, data to be used for processing, etc. The RAM 873 temporarily or permanently stores, for example, programs to be read by the processor 871, various parameters that appropriately change when the programs are executed, etc.

[0222] (Host Bus 874, Bridge 875, External Bus 876, and Interface 877)

[0223] The processor 871, the ROM 872, and the RAM 873 are connected to each other via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, the host bus 874 is connected to an external bus 876 having a relatively low data transmission speed via, for example, a bridge 875. In addition, the external bus 876 is connected to various components via an interface 877.

[0224] (Input device 878)

[0225] For example, a mouse, keyboard, touch pad, button, switch, joystick, etc. are used as the input device 878. In addition, a remote controller (hereinafter, referred to as a remote controller) capable of transmitting a control signal using infrared rays or other radio waves may be used as the input device 878. In addition, the input device 878 includes a voice input device such as a microphone.

[0226] (Output device 879)

[0227] The output device 879 is a device capable of visually or auditorily notifying the user of acquired information, such as a display device represented by a cathode ray tube (CRT), LCD, organic EL, etc., an audio output device represented by a speaker, headphones, etc., a printer, a mobile phone, a fax machine, etc. In addition, the output device 879 in the present disclosure includes various vibration devices capable of outputting tactile stimulation.

[0228] (Memory 880)

[0229] The memory 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like is used as the memory 880.

[0230] (Driver 881)

[0231] The drive 881 is a device that reads information stored in the removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901 , for example.

[0232] (Removable Storage Medium 901)

[0233] The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, one of various semiconductor storage media, etc. Needless to say, the removable storage medium 901 may be, for example, an IC card having a non-contact IC chip mounted thereon, an electronic device, or the like.

[0234] (Connection port 882)

[0235] The connection port 882 is, for example, a port for connecting the external connection device 902 , such as a universal serial bus (USB) port, an IEEE 1394 port, a small computer system interface (SCSI), an RS-232C port, or an optical audio terminal.

[0236] (External connection device 902)

[0237] The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, an IC recorder, or the like.

[0238] (communication device 883)

[0239] The communication device 883 is a communication device for connecting to a network, and is, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark) or wireless USB (WUSB), a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for one of various types of communication, etc.

[0240] <3. Conclusion>

[0241] As described above, the information processing device 10 according to an embodiment of the present disclosure includes an estimation unit 150 that estimates a user's attention level for a specific type of output information based on the user's physical state. Furthermore, one of the features of the estimation unit 150 according to an embodiment of the present disclosure is that the attention level is estimated using an estimation model 155 that determines the degree of change in the output pattern of the output information based on the trainee's physical state and experience of whether the trainee perceives a change in the output pattern of the same type of training information when the output pattern of the training information changes.

[0242] With the above configuration, it is possible to accurately estimate the intensity of the user's attention and the spatial range of the attention.

[0243] Although the preferred embodiment of the present disclosure has been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to this example. Obviously, a person skilled in the art with ordinary knowledge in the technical field of the present disclosure can conceive of various changes or modifications within the scope of the technical concept described in the claims, and naturally understand that these changes or modifications also fall within the technical scope of the present disclosure.

[0244] In addition, each step related to the processing described in the present disclosure is not necessarily processed in a time series in the order described in the flowchart or sequence diagram. For example, the various steps related to the processing performed by each device may be processed in a different order than described, or may be processed in parallel with each other.

[0245] In addition, the series of processing operations performed by each device described in the present disclosure can be implemented by a program stored in a non-transitory computer readable storage medium. When a computer executes each program, for example, the program is loaded into RAM and executed by a processor such as a CPU. The storage medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. Alternatively, the program can be distributed, for example, over a network without using a storage medium.

[0246] In addition, the effects disclosed herein are merely illustrative or exemplary, and not restrictive. That is, in addition to or in place of the above effects, the technology in the present disclosure can produce other effects that those skilled in the art can see from the description herein.

[0247] Note that the following configurations also belong to the technical scope of the present disclosure.

[0248] (1) An information processing device comprising:

[0249] an estimating unit that estimates a user's attention level to a specific type of output information based on the user's physical state,

[0250] In which, the estimation unit uses an estimation model to estimate the attention level, and the estimation model determines the degree of change in the output pattern of the output information based on the trainee's physical state and the trainee's experience of whether the trainee perceives the change in the output pattern of the training information when the output pattern of the training information of the same type as the output information changes.

[0251] (2) The information processing device according to (1), wherein

[0252] The estimation model determines, based on the experience, a degree of change in the output pattern of the training information such that a trainee will not perceive the change in the output pattern of the training information.

[0253] (3) The information processing device according to (2), wherein

[0254] The estimation model determines the degree of change in the output pattern of the training information using an objective function that differs depending on whether the trainee perceives the change in the output pattern of the training information.

[0255] (4) The information processing device according to (3), wherein

[0256] When the trainee does not perceive the change in the output pattern of the training information, the estimation model determines the degree of change in the output pattern of the training information by maximizing an objective function corresponding to a positive number proportional to the degree of change in the output pattern of the training information.

[0257] (5) The information processing device according to (3) or (4), wherein

[0258] When the trainee perceives a change in the output pattern of the training information, the estimation model determines the degree of change in the output pattern of the training information in a manner that maximizes an objective function corresponding to a negative number proportional to the degree of change in the output pattern of the training information.

[0259] (6) The information processing device according to any one of (1) to (5), wherein

[0260] The estimation unit calculates the attention level based on a degree of change in an output pattern of output information output from the estimation model, while inputting feature values ​​extracted from sensor information indicating a user's body state into the estimation model.

[0261] (7) The information processing device according to (6), wherein

[0262] The estimation unit calculates the attention level based on a degree of change in an output pattern of output information output from the estimation model and a maximum amount of change associated with the output pattern.

[0263] (8) The information processing device according to any one of (1) to (7), wherein

[0264] The output information includes visual information.

[0265] (9) The information processing device according to (8), wherein

[0266] The estimation model also determines the degree of change in the output pattern of the training information based on the display position of the training information.

[0267] (10) The information processing device according to (8), wherein

[0268] The output mode of the training information includes the display location.

[0269] (11) The information processing device according to (6), wherein

[0270] The sensor information includes an image obtained by capturing an image of the user's face.

[0271] (12) The information processing device according to any one of (1) to (11), further comprising:

[0272] An output control unit that controls output of the output information based on the attention level estimated by the estimation unit.

[0273] (13) The information processing device according to (12), wherein

[0274] The output control unit controls an output position of the output information based on the attention level.

[0275] (14) The information processing device according to (12) or (13), wherein

[0276] The output control unit controls the behavior of the display object based on the attention level.

[0277] (15) The information processing device according to any one of (12) to (14), wherein

[0278] The output control unit controls output of the content based on an attention level for each of the plurality of contents being output.

[0279] (16) The information processing device according to (12), wherein

[0280] The output control unit controls dynamic display of the attention level continuously estimated by the estimation unit.

[0281] (17) The information processing device according to any one of (12) to (16), wherein

[0282] The output control unit controls output of the output information based on an attention level of each of the plurality of users.

[0283] (18) An information processing method comprising:

[0284] estimating, using a processor, a user's level of attention to a particular type of output information based on the user's physical state,

[0285] In which, in the estimation, an estimation model is used to estimate the attention level, and the estimation model determines the degree of change in the output pattern of the output information based on the trainee's physical state and the experience of whether the trainee perceives the change in the output pattern of the training information when the output pattern of the training information of the same type as the output information changes.

[0286] (19) A program for causing a computer to function as:

[0287] An information processing device comprising:

[0288] an estimating unit that estimates a user's attention level to a specific type of output information based on the user's physical state,

[0289] In which, the estimation unit uses an estimation model to estimate the attention level, and the estimation model determines the degree of change in the output mode of the output information based on the trainee's physical state and the trainee's experience of whether the trainee perceives the change in the output mode of the training information when the output mode of the training information of the same type as the output information changes.

[0290] Reference Signs List

[0291] 10Information processing device

[0292] 110 pointing units

[0293] 120 Status Acquisition Unit

[0294] 130 Feature value extraction unit

[0295] 140 estimated position specifying unit

[0296] 150 estimation units

[0297] 155 Estimation Model

[0298] 160 storage units

[0299] 170 output control unit

[0300] 180 display units

Claims

1. An information processing device, comprising: an estimating unit that estimates a user's attention level to a specific type of output information based on the user's physical state, Wherein, the estimation unit estimates the attention level using an estimation model, and the estimation model determines the degree of change in the output mode of the output information based on the trainee's physical state and experience of whether the trainee perceives the change in the output mode of the training information when the output mode of training information of the same type as the output information changes.

2. The information processing device according to claim 1, wherein: The estimation model determines, based on the experience, a degree of change in the output pattern of the training information such that the trainee will not perceive the change in the output pattern of the training information.

3. The information processing device according to claim 2, wherein: The estimation model determines the degree of change in the output pattern of the training information using an objective function that differs depending on whether the trainee perceives the change in the output pattern of the training information.

4. The information processing device according to claim 3, wherein: When the trainee does not perceive the change in the output mode of the training information, the estimation model determines the degree of change in the output mode of the training information in a manner that maximizes an objective function corresponding to a positive number proportional to the degree of change in the output mode of the training information.

5. The information processing device according to claim 3, wherein: When the trainee perceives a change in the output mode of the training information, the estimation model determines the degree of change in the output mode of the training information in a manner that maximizes an objective function corresponding to a negative number proportional to the degree of change in the output mode of the training information.

6. The information processing device according to claim 1, wherein: The estimation unit calculates the attention level based on a degree of change in an output pattern of the output information output from the estimation model, when a feature value extracted from sensor information indicating a physical state of the user is input to the estimation model.

7. The information processing device according to claim 6, wherein: The estimation unit calculates the attention level based on a degree of change in an output pattern of the output information output from the estimation model and a maximum amount of change associated with the output pattern.

8. The information processing device according to claim 1, wherein: The output information includes visual information.

9. The information processing device according to claim 8, wherein: The estimation model also determines a degree of change in an output mode of the training information based on a display position of the training information.

10. The information processing device according to claim 8, wherein: The output mode of the training information includes a display location.

11. The information processing device according to claim 6, wherein: The sensor information includes an image obtained by capturing an image of the user's face.

12. The information processing device according to claim 1, further comprising: An output control unit controls output of the output information based on the attention level estimated by the estimation unit.

13. The information processing device according to claim 12, wherein: The output control unit controls an output position of the output information based on the attention level.

14. The information processing device according to claim 12, wherein: The output control unit controls behavior of a display object based on the attention level.

15. The information processing device according to claim 12, wherein: The output control unit controls output of the content based on an attention level to each of the plurality of contents being output.

16. The information processing device according to claim 12, wherein: The output control unit controls dynamic display of the attention level continuously estimated by the estimation unit.

17. The information processing device according to claim 12, wherein: The output control unit controls output of the output information based on an attention level of each of a plurality of users.

18. An information processing method, comprising: estimating, using a processor, a user's attention level for a particular type of output information based on the user's physical state, Wherein, in the estimation, an estimation model is used to estimate the attention level, and the estimation model determines the degree of change of the output mode of the output information based on the trainee's physical state and the trainee's experience of whether the trainee perceives the change in the output mode of the training information when the output mode of training information of the same type as the output information changes.

19. A program causing a computer to: An information processing device comprising: an estimating unit that estimates a user's attention level to a specific type of output information based on the user's physical state, Wherein, the estimation unit estimates the attention level using an estimation model, and the estimation model determines the degree of change in the output mode of the output information based on the trainee's physical state and experience of whether the trainee perceives the change in the output mode of the training information when the output mode of training information of the same type as the output information changes.