Information processing method, information processing device, and program

Abstract

This information processing device: identifies, on the basis of image information relating to an image corresponding to the visual field of a user and line-of-sight information indicating the direction of the line of sight of the user, a focus object that the user is focusing on among one or more objects included in the image; and calculates, with regard to each of one or more focus objects identified for a plurality of users, a degree of attention on the basis of the number of users focusing on the focus objects.

Description

Information Processing Method, Information Processing Apparatus, and Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program.

[0002] Patent Document 1 discloses a congestion estimation system including a plurality of wearable terminals and a server. The wearable terminal includes a photographing unit that photographs a subject, a position detection unit that calculates a current position, and a terminal communication unit that transmits an image obtained by photographing by the photographing unit and the current position calculated by the position detection unit to the server. The server includes a server communication unit that communicates with the plurality of wearable terminals, a storage unit that stores the image and the current position transmitted from the plurality of wearable terminals and associated with each other, and a server control unit that analyzes the image stored in the storage unit to calculate the number of people and the area of the sidewalk, calculates the congestion degree of pedestrians on the sidewalk from the calculated number of people and the area of the sidewalk, and associates the calculated congestion degree and the current position and transmits them by the server communication unit.

[0003] According to the background art, it is possible to calculate the congestion degree of pedestrians on the sidewalk, but no consideration has been given to calculating the degree of attention of each object and identifying the object being focused on by a plurality of users.

[0004] Japanese Patent No. 6435640

[0005] An object of the present disclosure is to obtain an information processing method, an information processing apparatus, and a program that can simply and highly accurately identify an object being focused on by a plurality of users.

[0006] An information processing method according to an aspect of the present disclosure is such that an information processing apparatus identifies a watched object that the user is watching among at least one object included in the image based on image information regarding an image corresponding to the user's field of view and gaze information indicating the direction of the user's gaze, and calculates a degree of attention based on the number of users watching the watched object for each of at least one watched object identified for a plurality of users.

[0007] This is a simplified diagram showing the first example of the overall configuration of the information processing system. This is a simplified diagram showing the configuration of the management server. This is a simplified diagram showing the configuration of the MR glasses. This is a simplified diagram showing the functional configuration of the processing unit. This is a flowchart showing the processes executed by the processing unit. This is a schematic diagram showing an example of the positional relationship between multiple users wearing MR glasses and multiple objects. This is a schematic diagram showing an image corresponding to the user's field of view, captured by the external camera of the MR glasses. This is a simplified diagram showing an example of the user's field of view. This is a simplified diagram showing the configuration of the MR glasses. This is a schematic diagram showing an example of the positional relationship between multiple users wearing MR glasses and multiple objects. This is a simplified diagram showing the second example of the overall configuration of the information processing system. This is a simplified diagram showing the third example of the overall configuration of the information processing system. This is a simplified diagram showing an example of the user's field of view. This is a simplified diagram showing a modified version of the overall configuration of the information processing system. This is a simplified diagram showing an example of the user's field of view. This is a simplified diagram showing an example of the user's field of view. This diagram shows a simplified functional configuration of the processing unit. This diagram shows a simplified example of a user's field of view. This diagram shows a simplified example of a user's field of view.

[0008] (Knowledge forming the basis of this disclosure) In a system in which multiple users share a limited resource such as space, equipment, or services, a management server manages the use of multiple resources by multiple users. Users check the usage status of each resource, such as congestion level or reservations, using their PC at their desk or their own smartphone. For example, in a meeting room reservation system built within a company with multiple meeting rooms, the management server manages the reservation status of multiple meeting rooms by multiple employees.

[0009] However, unlike a meeting room reservation system where the number of meeting rooms is fixed, if the number of objects to be managed changes irregularly, or if the objects to be managed are fluid, it becomes difficult to easily manage the usage status of each object.

[0010] Furthermore, with digital signage capable of detecting user gaze or analyzing user facial images, attribute information such as the number of users who focused on the advertisement, their age, and gender can be aggregated for each advertisement, and this attribute information can be used for product marketing and other purposes.

[0011] However, in cases where the scope of management is unclear, such as with advertisements using paper posters rather than digital signage, it is difficult to easily manage the attribute information of users who paid attention to the advertisement.

[0012] Furthermore, in a system that manages whether a toilet is occupied or not based on the lock signal of the toilet door and provides users with usage information showing the status of multiple toilets, it is possible that a user may start moving toward an unoccupied toilet, and just as that user is about to arrive, another user may enter that toilet. Moreover, because the subject of management is toilets, making a reservation for use is cumbersome for the user, and reserving future use is not practical given the nature of toilets.

[0013] To solve these problems, the inventors have found that by having each user wear MR glasses or the like, identifying the object that each user is fixated on based on their gaze information, and calculating the degree of attention each object receives based on the number of users who are fixated on each object, it is possible to easily and accurately identify an object that is attracting the attention of multiple users, thereby solving the above problems, and thus conceived this disclosure.

[0014] Next, we will describe each aspect of this disclosure.

[0015] An information processing method according to a first aspect of this disclosure involves an information processing device that identifies a gaze object among at least one object included in an image that the user is looking at, based on image information relating to an image corresponding to the user's field of view and gaze information indicating the direction of the user's gaze, and for each of the at least one gaze object identified for multiple users, calculates the degree of attention based on the number of users who are looking at the gaze object.

[0016] According to the first embodiment, it becomes possible to easily and accurately identify an object that is attracting the attention of multiple users.

[0017] In the information processing method according to a second aspect of this disclosure, in the first aspect, the object being looked at is further identified based on location information indicating the user's location.

[0018] According to the second embodiment, the accuracy of identifying the object being watched can be improved by using location information that indicates the user's position.

[0019] In the information processing method according to the third aspect of this disclosure, in the second aspect, the object being looked at is further identified based on directional information indicating the direction of the user's head.

[0020] According to the third embodiment, the accuracy of identifying the object being gazed upon can be further improved by using directional information indicating the direction of the user's head.

[0021] In the information processing method relating to the fourth aspect of this disclosure, in any one of the first to third aspects, the attention level is further calculated based on the distance between the user and the object being watched.

[0022] According to the fourth embodiment, the accuracy of calculating attention level can be improved by using the distance between the user and the object being gazed upon.

[0023] In the information processing method relating to the fifth aspect of this disclosure, in any one of the first to fourth aspects, the attention level is further calculated based on the direction of movement of the user.

[0024] According to the fifth embodiment, the accuracy of calculating attention can be improved by using the user's direction of movement.

[0025] In the sixth aspect of the information processing method of this disclosure, in the fifth aspect, the attention level is further calculated based on the user's movement speed.

[0026] According to the sixth embodiment, the accuracy of calculating attention can be further improved by using the user's movement speed.

[0027] In the information processing method relating to the seventh aspect of this disclosure, in any one of the first to sixth aspects, it is preferable to further calculate the attention level based on the user's biometric information when calculating the attention level.

[0028] According to the seventh embodiment, the accuracy of calculating attention level can be improved by using the user's biometric information.

[0029] The information processing method according to the eighth aspect of this disclosure may further generate additional information relating to the gaze object and display the additional information in the field of view in association with the gaze object, in any one of the first to seventh aspects.

[0030] According to the eighth aspect, additional information can be presented to the user by displaying additional information about the object being watched within the field of view.

[0031] In the information processing method according to the ninth aspect of this disclosure, in the eighth aspect, the additional information may include the degree of attention with respect to the gaze object.

[0032] According to the ninth embodiment, the level of attention given to each object can be presented to the user by including the level of attention given to the additional information.

[0033] In the tenth aspect of this disclosure, the information processing method may, in the ninth aspect, further include the reliability of the degree of attention with respect to the gaze object.

[0034] According to the tenth embodiment, the reliability of the attention level for each object can be presented to the user by including the reliability of the attention level in the additional information.

[0035] The information processing method according to the eleventh aspect of this disclosure further includes, in any one of the eighth to tenth aspects, determining the category to which the gazed object belongs, identifying other objects belonging to the same category as the gazed object within the image, calculating the degree of attention given to the other objects based on the number of users gazing at those other objects, generating additional information including the degree of attention given to the other objects in the generation of the additional information, and displaying the additional information in association with the other objects within the field of view in the display of the additional information.

[0036] According to the eleventh embodiment, the user can easily compare the level of attention each object receives and utilize the comparison results to select an object.

[0037] The information processing method according to the twelfth aspect of this disclosure further includes, in any one of the first to seventh aspects, determining the category to which the gazed object belongs, identifying other objects belonging to the same category as the gazed object within the image, calculating the degree of attention given to the other objects based on the number of users gazing at the other objects, setting the visibility of the gazed object within the field of view based on the degree of attention given to the gazed object, and setting the visibility of the other objects within the field of view based on the degree of attention given to the other objects.

[0038] According to the twelfth embodiment, the visibility of objects within the field of view can be varied according to the degree of attention they receive, thereby guiding the user in selecting objects.

[0039] The information processing method relating to the 13th aspect of this disclosure may further identify a product related to the object being watched in any one of the 8th to 11th aspects, and the additional information may include product information of the said product.

[0040] According to the 13th embodiment, by including product information in the additional information, product information for products related to each object can be presented to the user.

[0041] In the information processing method according to the 14th aspect of the present disclosure, in the 13th aspect, it is preferable that the product information includes the inventory information of the product.

[0042] According to the 14th aspect, since the product information includes the inventory information of the product, it is possible to stimulate the user's willingness to purchase the product.

[0043] In the information processing method according to the 15th aspect of the present disclosure, in any one of the 8th to 11th aspects, further, the usage order of the object being watched by a plurality of users is adjusted, and it is preferable that the additional information includes information regarding the usage order.

[0044] According to the 15th aspect, since the additional information includes information regarding the usage order of the object, it is possible to present the reservation status regarding the usage of each object to the user.

[0045] In the information processing method according to the 16th aspect of the present disclosure, in the 15th aspect, further, a category to which the object being watched belongs is determined, and in the adjustment of the usage order, it is preferable to determine the necessity of adjusting the usage order based on the degree of attention regarding the object being watched and the category to which the object being watched belongs.

[0046] According to the 16th aspect, by determining the necessity of adjusting the usage order based on the degree of attention and the category, it is possible to execute the adjustment of the usage order for the object with the degree of attention and category that require adjustment.

[0047] The information processing apparatus according to the 17th aspect of the present disclosure is an information processing apparatus having a circuit configuration, and the circuit configuration identifies the object being watched by the user among at least one object included in the image based on the image information regarding the image corresponding to the user's field of view and the gaze information indicating the direction of the user's line of sight, and calculates the degree of attention based on the number of users who are watching each of the at least one object being watched identified for a plurality of users.

[0048] According to the 17th aspect, it is possible to simply and highly accurately identify the object being watched by a plurality of users.

[0049] A program according to the 18th aspect of this disclosure is a program for causing an information processing device to perform processing, wherein the processing identifies a gaze object that the user is looking at from among at least one object included in the image, based on image information relating to an image corresponding to the user's field of view and gaze information indicating the direction of the user's gaze, and for each of the at least one gaze object identified for a plurality of users, the degree of attention is calculated based on the number of users who are looking at the gaze object.

[0050] According to the 18th aspect, it becomes possible to easily and accurately identify an object that is attracting the attention of multiple users.

[0051] This disclosure can also be implemented as a program that causes a computer to execute each characteristic configuration included in such a method or apparatus, or as a system that operates using such a program. It goes without saying that such a computer program can be distributed via a computer-readable, non-temporary recording medium such as a CD-ROM, or via a communication network such as the Internet.

[0052] (Embodiments of the Disclosure) Embodiments of the Disclosure will be described in detail below with reference to the drawings. Elements denoted by the same reference numeral in different drawings will indicate the same or corresponding elements. Each embodiment can be arbitrarily combined and applied. Furthermore, the components, their arrangement, connection configurations, and operation sequences shown in the following embodiments are examples and are not intended to limit the Disclosure. The Disclosure is limited only by the claims. Therefore, among the components in the following embodiments, those not described in the independent claims representing the highest-level concepts of the Disclosure are described as constituting a more preferable configuration, even though they are not necessarily required to achieve the object of the Disclosure.

[0053] Figure 1 is a simplified diagram showing a first example of the overall configuration of the information processing system 100. The information processing system 100 comprises a management server 1, multiple MR (Mixed Reality) glasses 2 worn by multiple users U, and a communication network 3. In the example shown in Figure 1, for the sake of simplification, users U include three users U1 to U3, and the MR glasses 2 include MR glasses G1 to G3 worn by users U1 to U3, respectively. Note that the terminal used by users U is not limited to MR glasses, but may also be AR glasses or VR goggles, etc. Furthermore, it is not limited to wearable devices, but may also be a smartphone or tablet terminal, etc.

[0054] Management server 1 may be a cloud server, on-premises server, or edge server, etc. Communication network 4 may be a WAN or LAN, etc.

[0055] Figure 2 is a simplified diagram showing the configuration of the management server 1. The management server 1 comprises a processing unit 11, a storage unit 12, and a communication unit 13. The processing unit 11 is configured using a processor (information processing device) such as a CPU. The storage unit 12 is configured using an HDD, SSD, or semiconductor memory, etc. The storage unit 12 includes a computer-readable non-volatile storage medium, and the program 19 is stored in the storage medium. The communication unit 13 is configured using a communication module, etc., corresponding to the communication method of the communication network 3.

[0056] Figure 3 is a simplified diagram showing the configuration of the MR glasses 2. The MR glasses 2 comprises a processing unit 21, a storage unit 22, a communication unit 23, an imaging unit 24, and a display unit 25. The processing unit 21 is configured using a processor such as a CPU. The storage unit 22 is configured using semiconductor memory or the like. The communication unit 23 is configured using a communication module or the like that corresponds to the communication method of the communication network 3. The imaging unit 24 has an external camera. The external camera is configured using an optical system and a CMOS image sensor or the like, and captures an image corresponding to the field of view of user U wearing the MR glasses 2. The imaging unit 24 also has an internal camera. The internal camera is configured using an optical system and a CMOS image sensor or the like, and captures an image of the eye of user U wearing the MR glasses 2. The display unit 25 is configured using a liquid crystal display or an organic EL display or the like.

[0057] The processing unit 21 includes a gaze detection unit 29. The gaze detection unit 29 may be implemented as a software function by the processor executing a program, or it may be implemented using hardware such as an FPGA or ASIC. The gaze detection unit 29 detects the direction of the user U's gaze by calculating the direction of the user U's eyeball using an arbitrary algorithm related to gaze tracking, based on the image data of the user U's eye input from the internal camera of the imaging unit 24. The gaze detection unit 29 may also detect the direction of the gaze based on electrooculography or the like instead of the image data of the eye.

[0058] Figure 4 is a simplified diagram showing the functional configuration of the processing unit 11. The functional configuration shown in Figure 4 is realized when the processing unit 11 executes the program 19 read from the storage unit 12. However, a similar functional configuration may be realized using hardware such as an FPGA or ASIC. The processing unit 11 comprises a control unit 41, an acquisition unit 42, a determination unit 43, a identification unit 44, a calculation unit 45, a generation unit 46, and an output unit 47. Details of the processing content executed by each of these units will be described later.

[0059] Figure 5 is a flowchart showing the process executed by the processing unit 11.

[0060] First, in step SP01, the acquisition unit 42 acquires various types of information. This information includes image information relating to images captured by the external camera of the imaging unit 24, which correspond to the field of view of user U wearing the MR glasses 2. The information also includes gaze information indicating the direction of the gaze of user U wearing the MR glasses 2, which is detected by the gaze detection unit 29. This information is transmitted from the communication unit 23 of the MR glasses 2 to the management server 1 via the communication network 3 and received by the communication unit 13 of the management server 1.

[0061] Next, in step SP02, the determination unit 43 determines at least one object included in the image corresponding to the user U's field of view using an arbitrary algorithm for object detection, such as a CNN (Convolutional Neural Network).

[0062] Next, in step SP03, the identification unit 44 identifies a gaze object based on the image information and gaze information acquired in step SP01 and the object information determined in step SP02. A gaze object means an object that at least one user U is gazing at, which is included in the image corresponding to the user U's field of view. The identification unit 44 identifies an object as the gaze object of a user U if the dwell time of the user U's gaze on that object exceeds a predetermined threshold. The acquisition unit 42 acquires image information and gaze information from multiple MR glasses 2 worn by multiple users U, and the identification unit 44 identifies the gaze object that each user U is gazing at based on the image information and gaze information acquired from each MR glasses 2.

[0063] Figure 6 schematically shows an example of the positional relationship between multiple users U1 to U3 wearing MR glasses G1 to G3 and multiple objects OA to OC. Objects OA to OC are, for example, products displayed for sale. Users U1 to U3 are facing towards objects OA to OC, and each user's field of view includes objects OA to OC. The objects may also be digital signage or paper advertising posters. If the object is a paper poster, the determination unit 43 can determine the product associated with the object by analyzing the image of the object captured by the external camera of the MR glasses 2. Furthermore, the acquisition unit 42 may acquire user information such as the age or gender of user U from the MR glasses 2 for use in product marketing, etc.

[0064] Figure 7 schematically shows image IM1 corresponding to the field of view of user U1, captured by the external camera of MR glasses G1. Image IM1 includes objects OA to OC. User U1, wearing MR glasses G1, is fixated on object OA, and the gaze detection unit 29 of MR glasses G1 detects user U1's gaze V1 at the position in image IM1 corresponding to object OA. Based on image information showing image IM1 corresponding to user U1's field of view and gaze information showing the direction of user U1's gaze V1, the identification unit 44 identifies object OA as the object that user U1 is fixated on. The identification unit 44 also identifies the objects that users U2 and U3 are fixated on, similar to user U1. The objects of fixation may or may not overlap for multiple users U.

[0065] Next, in step SP04, the calculation unit 45 calculates the degree of attention for each of the at least one gazed-on object identified for each of the multiple users U, based on the number of users U who are gazing at the gazed-on object. The calculation unit 45 may, for example, calculate the degree of attention for each gazed-on object as the total number of users U who are simultaneously gazing at that object. For example, the degree of attention for an object that is simultaneously gazing at by three users U will be "3", and the degree of attention for an object that is not gazing at by even one user U will be "0".

[0066] Next, in step SP05, the generation unit 46 generates additional information with respect to the gaze object identified in step SP03, according to the level of attention calculated in step SP04.

[0067] Next, in step SP06, the output unit 47 outputs the additional information generated in step SP05. The control unit 41 overlays the additional information output from the output unit 47 onto the corresponding gaze object within the field of view of each user U and displays it on the display unit 25 of each MR glasses 2.

[0068] For example, when superimposing additional information onto user U1's field of view FV1, the determination unit 43 determines the category to which user U1's gaze object OA belongs (in this example, a specific product). The identification unit 44 identifies other objects OB and OC belonging to the same category as gaze object OA within the image IM1 corresponding to user U1's field of view FV1. The calculation unit 45 calculates the degree of attention given to object OA based on the number of users U who are gazing at object OA. Similarly, the calculation unit 45 calculates the degree of attention given to objects OB and OC based on the number of users U who are gazing at objects OB and OC. The generation unit 46 generates additional information including the degree of attention given to object OA, and the output unit 47 outputs the generated additional information. Similarly, the generation unit 46 generates other additional information including the degree of attention given to other objects OB and OC, and the output unit 47 outputs the generated additional information. The control unit 41 displays the additional information in the field of view FV1, associating it with object OA. Similarly, the control unit 41 displays other additional information within the field of view FV1, associating it with other objects OB and OC.

[0069] Figure 8 is a simplified diagram showing an example of the field of view FV1 of user U1. Additional information and other additional information include the degree of attention given to the object being gazed upon. In the example shown in Figure 8, the degree of attention is represented by the size of the flame image around the object. For example, the degree of attention for object OA is "2", the degree of attention for object OB is "0", and the degree of attention for object OC is "1". In this case, the control unit 41 superimposes a large flame image ZA1 onto object OA and displays it on the display unit 25, and superimposes a smaller flame image ZC1 onto object OC and displays it on the display unit 25. Note that no flame image is superimposed on object OB, which has a degree of attention of "0". Alternatively, instead of superimposed display, the control unit 41 may display a map on the display unit 25 of the MR glasses 2 and display the user U's current position, the position of each object, and the degree of attention on the map.

[0070] According to this embodiment, the processing unit 11, acting as an information processing device, identifies a gaze object that user U is fixated on from at least one object included in an image, based on image information relating to an image corresponding to user U's field of view and gaze information indicating the direction of user U's gaze. Furthermore, for each of the at least one gaze object identified for multiple user Us, the processing unit 11 calculates the degree of attention based on the number of user Us who are fixated on the gaze object. Thus, according to this embodiment, it is possible to easily and accurately identify an object that is attracting the attention of multiple user Us. Specifically, according to this embodiment, by automatically managing an object that is attracting the attention of multiple user Us, the degree of attention can be easily visualized even for fluid or unclear resources without manually constructing a management system. In particular, in a system that manages the usage status of multiple toilets, it is possible to visualize toilets with high levels of attention before the toilet doors are locked, thereby streamlining the management of usage status and eliminating the need for users to make cumbersome reservations.

[0071] Furthermore, the processing unit 11 generates additional information about the gazed object and displays the additional information within the field of view FV1, associating it with the gazed object. By displaying the additional information about the gazed object within the field of view FV1, the additional information can be easily presented to the user U.

[0072] Furthermore, by including the level of attention given to each object as additional information, the level of attention given to each object can be easily presented to the user U.

[0073] Furthermore, user U can easily compare the level of attention each object receives and use the comparison results to select an object.

[0074] The following describes various modifications of the above embodiment. The modifications described below can be applied in any combination.

[0075] (First Modification) Figure 9 is a simplified diagram showing the configuration of MR glasses 2. Compared to the configuration shown in Figure 3, MR glasses 2 further includes a position sensor 31, a direction sensor 32, a distance sensor 33, a speed sensor 34, and a biosensor 35.

[0076] The position sensor 31 detects the position of user U wearing the MR glasses 2 in real space using a positioning method that utilizes a satellite positioning system such as GPS, or by position estimation based on external camera images used in SLAM (Simultaneous Localization and Mapping), and outputs position information indicating that position.

[0077] The direction sensor 32 detects the direction of the head of the user U wearing the MR glasses 2 by estimating the posture using a gyro sensor or an angular velocity sensor, and outputs direction information indicating that direction.

[0078] The distance sensor 33 measures the distance between the user U wearing the MR glasses 2 and the object using a distance measurement method such as laser light, millimeter waves, or ultrasound, and outputs distance information indicating that distance.

[0079] The speed sensor 34 detects the movement speed (walking speed) of user U wearing the MR glasses 2 and outputs speed information indicating that speed.

[0080] The biosensor 35 detects the biosignals of user U wearing the MR glasses 2 and outputs biosignals indicating those biosignals. The biosignals include electromyography, electrooculography, electroencephalography, changes in muscle shape, changes in muscle stiffness, sounds generated by the body, skin temperature, skin conductivity, respiration, sweating, heart rate, or blood pressure.

[0081] Note that some of the sensors shown in Figure 9, such as the position sensor 31, direction sensor 32, distance sensor 33, speed sensor 34, and biosensor 35, may be omitted, or other sensors may be added.

[0082] Figure 10 schematically shows an example of the positional relationship between multiple users U1 to U3 wearing MR glasses G1 to G3 and multiple objects OA to OC.

[0083] The position sensor 31 of the MR glasses G1 detects the position of user U1 in real space and outputs position information indicating that position. The position information includes position coordinates (X1, Y1, Z1). The direction sensor 32 of the MR glasses G1 detects the direction of user U1's head and outputs direction information indicating that direction. The direction information includes the angle between a predetermined reference direction and the front direction A1 of the MR glasses G1.

[0084] Similarly, the position sensor 31 of the MR glasses G2 detects the position of user U2 in real space and outputs position information indicating that position. The position information includes position coordinates (X2, Y2, Z2). The direction sensor 32 of the MR glasses G2 detects the direction of user U2's head and outputs direction information indicating that direction. The direction information includes the angle between a predetermined reference direction and the front direction A2 of the MR glasses G2.

[0085] Similarly, the position sensor 31 of the MR glasses G3 detects the position of user U3 in real space and outputs position information indicating that position. The position information includes position coordinates (X3, Y3, Z3). The direction sensor 32 of the MR glasses G3 detects the direction of user U3's head and outputs direction information indicating that direction. The direction information includes the angle between a predetermined reference direction and the front direction A3 of the MR glasses G3.

[0086] The acquisition unit 42 acquires image information, gaze information, position information, direction information, distance information, speed information, and biometric information from each of the multiple MR glasses 2 worn by multiple users U. This information is transmitted from the communication unit 23 of the MR glasses 2 to the management server 1 via the communication network 3 and received by the communication unit 13 of the management server 1.

[0087] The identification unit 44 identifies the object that each user U is fixated on based on image information, gaze information, position information, and direction information acquired from each MR glasses 2. The identification unit 44 may further identify the object based on feature information. The feature information indicates the feature quantities of objects around the point of fixation of each user U. The identification unit 44 estimates the feature quantities of objects using any image recognition technology such as CNN. By using feature quantities, the accuracy of distinguishing each object can be improved even when multiple objects are similar, and as a result, the accuracy of identifying the object of fixation can be improved.

[0088] The calculation unit 45 calculates the degree of attention for each gazed object based on the number of users U who are gazing at the gazed object, distance information, speed information, and biometric information. The calculation unit 45 evaluates the degree of attention to be higher the closer the distance between the user U and the gazed object, and lower the degree of attention to be higher the further the distance between the user U and the gazed object. The calculation unit 45 evaluates the degree of attention to be higher the greater the degree of agreement between the direction of movement of the user U and the direction in which the user U is approaching the gazed object, and lower the degree of attention to be lower the less the degree of agreement between the direction of movement of the user U and the direction in which the user U is approaching the gazed object. The calculation unit 45 evaluates the degree of attention to be higher the greater the speed at which the user U is moving in the direction of approaching the gazed object is, and lower the degree of attention to be lower the greater the speed at which the user U is moving in the direction of approaching the gazed object is. Regarding the biometric signals of the user U gazing at the gazed object, the calculation unit 45 evaluates the degree of attention to be higher the greater the change from normal, and lower the degree of attention to be lower the greater the change from normal.

[0089] According to this modified example, the accuracy of identifying the object being watched can be improved by using location information that indicates the position of user U.

[0090] Furthermore, according to this modified version, the accuracy of identifying the object being gazed upon can be further improved by using directional information indicating the direction of the user U's head.

[0091] Furthermore, according to this modified version, the accuracy of calculating attention level can be improved by using the distance between user U and the object being gazed upon.

[0092] Furthermore, according to this modified version, the accuracy of calculating attention can be improved by using the direction of movement of user U.

[0093] Furthermore, according to this modified example, the accuracy of calculating attention can be further improved by using the movement speed of user U.

[0094] Furthermore, according to this modified version, the accuracy of calculating attention can be improved by using the biometric information of user U.

[0095] (Second Modification) Figure 11 is a simplified diagram showing a second example of the overall configuration of the information processing system 100. Figure 1 shows a server-type system configuration with a management server 1, but as shown in Figure 11, an ad-hoc system configuration in which the MR glasses 2 communicate directly with each other without a management server 1 is also possible. The functions of the management server 1 are implemented in each MR glass 2.

[0096] Figure 12 is a simplified diagram showing a third example of the overall configuration of the information processing system 100. As shown in Figure 12, the system configuration may be a mixture of server-type and ad-hoc-type systems. In the example shown in Figure 12, MR glasses G2 communicates with the management server 1 via the communication network 3, while MR glasses G1 and G3 communicate directly with MR glasses G2.

[0097] (Third Modification) In the above embodiment, the control unit 41 displays additional information in association with the object being watched and in the field of view, and displays other additional information in association with other objects and in the field of view, but the system is not limited to this example. The control unit 41 may set the visibility of the object being watched in the field of view based on the degree of attention given to the object being watched, and set the visibility of other objects in the field of view based on the degree of attention given to the other objects.

[0098] Figure 13 is a simplified diagram showing an example of the field of view FV1 of user U1. The control unit 41 performs image processing on objects OA to OC so that the visibility of objects of high importance decreases, and the visibility of objects of low importance improves. For example, visibility can be reduced by lowering the clarity, brightness, or transparency of the image area corresponding to an object, and conversely, visibility can be improved by increasing the clarity, brightness, or transparency of the image area corresponding to an object.

[0099] According to this modified version, the visibility of objects within the field of view FV1 can be varied according to their level of attention, thereby guiding the user's object selection. For example, by decreasing the visibility of highly attention-grabbing objects and increasing the visibility of less attention-grabbing objects, it is possible to guide users to select less attention-grabbing objects (i.e., objects that are not crowded or objects with no competition).

[0100] However, the control unit 41 may apply image processing to objects OA to OC such that the visibility of objects of high attention improves, and the visibility of objects of low attention decreases. In this case, the user can intuitively recognize which objects are attracting attention.

[0101] (Fourth Modification) In the above embodiment, the additional information included the degree of attention given to the object being watched, but the invention is not limited to this example. The additional information may include product information related to the object being watched. The product information may also include inventory information for the product.

[0102] Figure 14 is a simplified diagram showing a modified configuration of the information processing system 100. The information processing system 100 further includes a product information management server 8 and an inventory information management server 9 in addition to the configuration shown in Figure 1. The product information management server 8 manages product information such as the product name, product code, and sales price for multiple pre-registered products. The inventory information management server 9 manages inventory information such as the product name, product code, and remaining stock quantity for multiple pre-registered products.

[0103] For example, when displaying additional information superimposed on the user U1's field of view FV1, the determination unit 43 determines the category to which the object OA that user U1 is fixated on belongs (in this example, a specific product). The identification unit 44 identifies other objects OB and OC belonging to the same category as the object OA within the image IM1 corresponding to user U1's field of view FV1. The identification unit 44 also identifies the objects OA and OC that are being fixed on by at least one user U. The calculation unit 45 calculates the degree of attention given to object OA based on the number of users U who are fixated on object OA. Similarly, the calculation unit 45 calculates the degree of attention given to object OC based on the number of users U who are fixated on object OC. The acquisition unit 42 acquires product information from the product information management server 8 and inventory information from the inventory information management server 9 regarding the objects OA and OC. The generation unit 46 generates additional information for the gaze objects OA and OC, including product information (e.g., product name), attention level information (e.g., attention ranking), and inventory information (e.g., remaining inventory). The output unit 47 outputs the generated additional information. The control unit 41 displays the additional information in the field of view FV1, associating it with the objects OA and OC. If a product is out of stock or its sales period has ended, the generation unit 46 may generate additional information including product information for similar or related products.

[0104] Figure 15 is a simplified diagram showing an example of user U1's field of view FV1. The additional information includes product information, attention level information, and inventory information related to the object being watched. The control unit 41 displays image ZA2, which shows the additional information related to the object being watched OA, on the display unit 25 in association with the object being watched OA. The control unit 41 also displays image ZC2, which shows the additional information related to the object being watched OC, on the display unit 25 in association with the object being watched OC. Note that no additional information is displayed for object OB, which has an attention level of "0". The display position of images ZA2 and ZC2 may be at the top or bottom edge of the screen, etc., and the display method of images ZA2 and ZC2 may be scrolling display, etc.

[0105] Furthermore, the attention level information described above does not necessarily have to be calculated using the attention level explained in the first modified example. For example, assuming object OA is product X, it may be calculated based on the sum of the attention levels of multiple objects that are the same product X. In other words, the attention level information superimposed on object OA, which is product X displayed in a certain store, may be calculated based on the sum of the attention levels of product X displayed in all other stores. Moreover, if the sum of attention levels of product X is low in the region where the store displaying object OA is located (for example, City A), but is ranked first in the sum of attention levels in another region where other stores are located (for example, City B), the attention level information may be displayed as, for example, "Ranked first in attention levels in City B."

[0106] According to this modified version, by including product information in the additional information, product information for products related to each object can be presented to user U.

[0107] Furthermore, according to this modified version, by including product inventory information, it is possible to stimulate user U's desire to purchase the product.

[0108] (Fifth variation) The additional information may include not only the degree of attention given to the object being watched, but also the reliability of the degree of attention given to the object being watched.

[0109] The generation unit 46 measures the elapsed time from the time the calculation unit 45 last calculated the level of attention for each gazed object to the present time. The longer the elapsed time, the older the information is, and the reliability decreases as the information deteriorates. For example, if communication between the MR glasses G1 and the communication network 3 is interrupted, the information regarding the gazed objects of user U1 is not updated, so the information becomes old and its reliability decreases. The generation unit 46 generates additional information for each gazed object according to the level of attention and reliability.

[0110] Figure 16 is a simplified diagram showing an example of user U1's field of view FV1. In the example shown in Figure 16, the degree of attention is represented by the size of the flame image around the object, and the reliability of attention is represented by the transparency of the flame image. The shorter the elapsed time and the higher the reliability of attention, the lower the transparency of the flame image, as in image ZC1. On the other hand, the longer the elapsed time and the lower the reliability of attention, the higher the transparency of the flame image, as in image ZA1. When the elapsed time reaches a certain period, the transparency becomes 100%, and image ZA1 disappears.

[0111] According to this modified example, by including the reliability of the level of attention in the additional information, the reliability of the level of attention for each object can be presented to the user U.

[0112] (Sixth variation) The additional information may include information on how multiple users coordinate the order in which they use the object they are looking at.

[0113] Figure 17 is a simplified diagram showing the functional configuration of the processing unit 11. The processing unit 11 further includes an adjustment unit 48 in addition to the functional configuration shown in Figure 4.

[0114] The identification unit 44 identifies the attention object OA from among objects OA to OC that is being viewed by multiple users U. The determination unit 43 determines the category to which the attention object OA belongs. The calculation unit 45 calculates the level of attention given to object OA based on the number of users U who are viewing it.

[0115] The adjustment unit 48 determines whether or not to adjust the order of use based on the level of attention given to the attention object OA and the category to which the attention object OA belongs. The adjustment unit 48 may also determine that adjustment of the order of use is necessary if the category to which the attention object OA belongs is one of the pre-registered categories such as a conference room, studio, or common space. Furthermore, the adjustment unit 48 may also determine that adjustment of the order of use is necessary if the level of attention is above a predetermined threshold.

[0116] If it is determined that the order of use needs to be adjusted, the generation unit 46 generates additional information including information regarding the order in which the gazed object OA is used. The output unit 47 outputs the generated additional information. The control unit 41 displays the additional information in the field of view FV1, associating it with the object OA.

[0117] Figure 18 is a simplified diagram showing an example of user U1's field of view FV1. The control unit 41 displays image ZA3, which shows additional information regarding the acceptance of the reservation of the first conference room, on the display unit 25 in association with the gaze object OA. The first conference room corresponds to object OA. Image ZA3 includes the name "First Conference Room" and the inquiry message "Do you want to make a reservation?". Image ZA3 also includes an icon labeled "YES" and an icon labeled "NO". When user U1 selects the icon labeled "YES", the adjustment unit 48 completes the reservation of the first conference room by user U1 and determines the usage time. The usage time may be specified by user U1, or it may be automatically determined by the adjustment unit 48 in predetermined time units set according to the category. The adjustment unit 48 may also confirm whether user U has the authority to use the conference room by using the user ID obtained from the MR glasses 2.

[0118] Figure 19 is a simplified diagram showing an example of user U1's field of view FV1. The control unit 41 displays image ZA4, which indicates additional information regarding the completion of the reservation of the first conference room, on the display unit 25 in association with the gaze object OA. Image ZA4 includes a completion message, "Reservation made," and the reserved usage time (in this example, from 11:00 to 12:00). Image ZA4 also includes an icon labeled "OK." When user U1 selects the icon labeled "OK," the control unit 41 stops displaying image ZA4 on the display unit 25. The adjustment unit 48 may also display an image on the display unit 25 of the MR glasses G1 that includes a message prompting user U1 to wait until the reservation time if user U1 is attempting to start using the conference room before the reservation time. Furthermore, the adjustment unit 48 may also display an image on the display unit 25 of the MR glasses G1 that includes a message prompting user U1 to start using the conference room if user U1 has not started using the conference room even after the reservation time has passed. In this case, the adjustment unit 48 may impose a penalty on user U1 who is late in starting to use the system, such as restricting the operation of the MR glasses G1.

[0119] According to this modified example, by including information about the order in which objects are used as additional information, the reservation status of each object can be presented to user U.

[0120] Furthermore, according to this modified example, by determining whether or not adjustment of the usage order is necessary based on attention level and category, adjustment of the usage order can be performed for objects with attention level and category that require adjustment.

[0121] (Seventh variation) The term "object" is not limited to real objects existing in real space, but may also include virtual objects existing in a virtual space created by computer graphics, etc.

[0122] If user U's avatar exists in the virtual space, the location information may include location information indicating user U's location in the real world and location information indicating the avatar's location in the virtual space.

[0123] In this case, in relation to the digital signage seen by the avatar in the virtual space, product information of goods sold in the vicinity of user U's current location or place of residence may be presented to user U.

[0124] This disclosure is broadly applicable to MR glasses, AR glasses, VR glasses, smart glasses, head-mounted displays, computer user interfaces, digital assistants, information terminals, or wearable devices.

Claims

1. An information processing method comprising: an information processing device identifying a gaze object that a user is fixated on from at least one object included in an image, based on image information relating to an image corresponding to a user's field of view and gaze information indicating the direction of the user's gaze; and for each of the at least one gaze object identified for multiple users, calculating the degree of attention based on the number of users who are fixated on the gaze object.

2. The information processing method according to claim 1, further comprising identifying the object of attention based on location information indicating the user's location.

3. The information processing method according to claim 2, wherein the object of gaze is identified based on directional information indicating the direction of the user's head.

4. The information processing method according to claim 1, wherein the attention level is further calculated based on the distance between the user and the object being watched.

5. The information processing method according to claim 1, wherein the attention level is further calculated based on the user's direction of movement.

6. The information processing method according to claim 5, wherein the attention level is further calculated based on the user's movement speed.

7. The information processing method according to claim 1, wherein the attention level is further calculated based on the user's biometric information.

8. The information processing method according to claim 1, further comprising generating additional information relating to the gaze object and displaying the additional information in association with the gaze object within the field of view.

9. The information processing method according to claim 8, wherein the additional information includes the degree of attention given to the object being watched.

10. The information processing method according to claim 9, wherein the additional information further includes the reliability of the degree of attention with respect to the object being watched.

11. The information processing method according to claim 8, further comprising: determining the category to which the gazed object belongs; identifying other objects belonging to the same category as the gazed object within the image; calculating the degree of attention given to the other objects based on the number of users gazing at the other objects; generating additional information, including the degree of attention given to the other objects; and displaying the additional information, further displaying the additional information within the field of view in association with the other objects.

12. The information processing method according to claim 1, further comprising: determining the category to which the object being watched belongs; identifying other objects belonging to the same category as the object being watched within the image; calculating the degree of attention given to the other objects based on the number of users watching those other objects; setting the visibility of the object being watched within the field of view based on the degree of attention given to the object being watched; and setting the visibility of the other objects within the field of view based on the degree of attention given to those other objects.

13. The information processing method according to claim 8, further comprising identifying a product related to the object being observed, wherein the additional information includes product information of the product.

14. The information processing method according to claim 13, wherein the product information includes inventory information of the product.

15. The information processing method according to claim 8, further comprising adjusting the order in which multiple users use the gazed object, wherein the additional information includes information regarding the order of use.

16. The information processing method according to claim 15, further comprising determining the category to which the object being watched belongs, and in adjusting the order of use, determining whether or not to adjust the order of use based on the degree of attention to the object being watched and the category to which the object being watched belongs.

17. An information processing device comprising a circuit configuration, wherein the circuit configuration identifies a gaze object among at least one object included in an image that the user is looking at, based on image information relating to an image corresponding to the user's field of view and gaze information indicating the direction of the user's gaze, and calculates the degree of attention for each of the at least one gaze object identified for multiple users, based on the number of users looking at the gaze object.

18. A program for causing an information processing device to perform processing, wherein the processing involves identifying a gaze object among at least one object included in an image that the user is looking at, based on image information relating to an image corresponding to the user's field of view and gaze information indicating the direction of the user's gaze, and for each of the at least one gaze object identified for multiple users, calculating the degree of attention based on the number of users looking at the gaze object.

Images