Information processing system, information processing method, and program

Abstract

In order to solve the problem of presenting information with which it is possible to more fully identify work performed by a user in a virtual space, an information processing system (1) comprises: a work information acquisition unit (11) that acquires work information regarding work performed by a user in a virtual space; a line-of-sight information acquisition unit (12) that acquires line-of-sight information regarding the line of sight of the user; an emotion information acquisition unit (13) that acquires emotion information regarding the user's emotion; and a relevance information output unit (14) that outputs information indicating the relevance between changes in the work information, changes in the line-of-sight information, and changes in the emotion information that occur as the work progresses.

Description

Technical Field

[0001] The present invention relates to a technique for presenting information acquired when a user performs work in a virtual space.

Background Art

[0002] A technique for presenting information acquired when a user performs work in a virtual space is known. For example, Patent Document 1 describes an apparatus that constructs a training environment in a virtual space and sequentially records the position coordinates, actions, and utterances of a trainer (user) in the virtual space together with time when the trainer performs training actions (work), and stores them as training data. Further, this apparatus reproduces a past training situation by calling and reproducing the stored training data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the apparatus described in Patent Document 1, there is a problem that it is not possible to sufficiently grasp the training actions of the trainer (work performed by the user) only by reproducing the past training situation.

[0005] One aspect of the present invention has been made in view of the above problems, and an example of the object is to provide a technique for presenting information that can more sufficiently grasp the work performed by a user in a virtual space.

Means for Solving the Problems

[0006] An information processing system according to an aspect of the present invention includes a work information acquisition unit that acquires work information related to work performed by a user in a virtual space, a gaze information acquisition unit that acquires gaze information related to the gaze of the user, an emotion information acquisition unit that acquires emotion information related to the emotion of the user, and a correlation information output unit that outputs information indicating the correlation between changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses.

[0007] An information processing method according to an aspect of the present invention includes acquiring work information related to work performed by a user in a virtual space, acquiring gaze information related to the gaze of the user, acquiring emotion information related to the emotion of the user, and outputting information indicating the correlation between changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses.

[0008] A program according to an aspect of the present invention is a program for causing a computer to function as an information processing system, and causes the computer to function as a work information acquisition unit that acquires work information related to work performed by a user in a virtual space, a gaze information acquisition unit that acquires gaze information related to the gaze of the user, an emotion information acquisition unit that acquires emotion information related to the emotion of the user, and a correlation information output unit that outputs information indicating the correlation between changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses.

Advantages of the Invention

[0009] According to one aspect of the present invention, it is possible to present information that can more fully grasp the work performed by the user in the virtual space.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Best Mode for Carrying Out the Invention

[0011] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for the exemplary embodiments described later.

[0012] <Configuration of Information Processing System 1> The configuration of the information processing system 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing system 1. As shown in FIG. 1, the information processing system 1 includes a work information acquisition unit 11, a gaze information acquisition unit 12, an emotion information acquisition unit 13, and a relevance information output unit 14. The work information acquisition unit 11 is an example of a configuration that realizes the work information acquisition means described in the claims. The gaze information acquisition unit 12 is an example of a configuration that realizes the gaze information acquisition means described in the claims. The emotion information acquisition unit 13 is an example of a configuration that realizes the emotion information acquisition means described in the claims. The relevance information output unit 14 is an example of a configuration that realizes the relevance information Output means described in the claims.

[0013] The work information acquisition unit 11 acquires work information related to the work performed by the user in the virtual space. The gaze information acquisition unit 12 acquires gaze information related to the user's gaze. The emotion information acquisition unit 13 acquires emotion information related to the user's emotion. The relevance information output unit 14 outputs information indicating the relevance of changes in work information, changes in gaze information, and changes in emotion information as the work progresses. Details of each of these functional blocks will be described in the "Flow of Information Processing Method S1" described later.

[0014] <Flow of Information Processing Method S1> The information processing system 1 configured as described above executes the information processing method S1 according to this exemplary embodiment. The flow of the information processing method S1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the information processing method S1. As shown in FIG. 2, the information processing method S1 includes steps S11 to S14.

[0015] In step S11, the work information acquisition unit 11 acquires work information regarding the work performed by the user in the virtual space. For example, the work information may include information indicating the start, end, or type of work, but is not limited thereto. Also, for example, the work information acquisition unit 11 may acquire work information based on an operation performed by the user in the virtual space. Further, the work information acquisition unit 11 may acquire work information by referring to information associated with work that can be performed by the user in the virtual space. However, the method of acquiring work information is not limited to this.

[0016] In step S12, the gaze information acquisition unit 12 acquires gaze information regarding the user's gaze. Here, for example, the gaze information acquisition unit 12 may acquire gaze information based on the orientation of the virtual reality device worn by the user to perform work in the virtual space. Also, for example, the gaze information acquisition unit 12 may acquire gaze information by referring to a captured image including the user as a subject. However, the method of acquiring gaze information is not limited to this.

[0017] In step S13, the emotion information acquisition unit 13 acquires emotion information regarding the user's emotion. Here, for example, the emotion information can be acquired using known emotion recognition techniques. As an example of an emotion recognition technique, there is a technique of analyzing the user's physiological indicators and acquiring emotion information such as concentration and stress level. Also, as an example of the user's physiological indicators, there are pulse wave, brain wave, heartbeat, and sweating. However, the emotion recognition technique is not limited to this.

[0018] The processes of steps S11 to S13 are repeatedly executed, for example, at each elapsed time point of the work while the user is performing work in the virtual space. For example, the information processing system 1 stores the work information, gaze information, and emotion information acquired in steps S11 to S13 in the memory in association with the elapsed time point at which the information was acquired. Note that the memory may be inside or outside the information processing system 1.

[0019] In step S14, the relevance information output unit 14 outputs information indicating the relevance among changes in work information, changes in line-of-sight information, and changes in emotion information as the work progresses. Here, for example, the relevance information output unit 14 may output work information, line-of-sight information, and emotion information in a manner associated with the elapsed time as information indicating the relevance. As an example of such a manner, for example, graphs showing changes in work information, changes in line-of-sight information, and changes in relevant information with respect to the horizontal axis (or vertical axis) indicating the elapsed time are superimposed while sharing the horizontal axis (or vertical axis). However, the information indicating the relevance is not limited to this.

[0020] <Example of implementation in a program> When the information processing system 1 is configured by a computer, the following program is stored in the memory referred to by the computer. The program is a program for causing the computer to function as the information processing system 1, and causes the computer to function as work information acquisition means for acquiring work information related to work performed by a user in a virtual space, line-of-sight information acquisition means for acquiring line-of-sight information related to the user's line of sight, emotion information acquisition means for acquiring emotion information related to the user's emotion, and relevance information output means for outputting information indicating the relevance among changes in work information, changes in line-of-sight information, and changes in emotion information as the work progresses.

[0021] <Effects of this exemplary embodiment> As described above, in the information processing system 1, information processing method S1, and program according to this exemplary embodiment, a configuration for acquiring work information related to work performed by a user in a virtual space, a configuration for acquiring line-of-sight information related to the user's line of sight, a configuration for acquiring emotion information related to the user's emotion, and a configuration for outputting information indicating the relevance among changes in work information, changes in line-of-sight information, and changes in emotion information as the work progresses are adopted.

[0022] Therefore, by using this exemplary embodiment, the work performed by the user in the virtual space IndustryIt is possible to know the correlations among changes in work information, changes in gaze information, and changes in emotional information as the process progresses, and it is possible to more fully grasp the work performed by the user. In this way, this exemplary embodiment can present information that enables a more complete understanding of the work performed by the user in the virtual space.

[0023] 〔Exemplary Embodiment 2〕 The second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same functions as those described in Exemplary Embodiment 1 are denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0024] <Overview of Information Processing System 1A> The overview of the information processing system 1A according to this exemplary embodiment will be described with reference to FIG. 3. FIG. 3 is a schematic diagram showing the overview of the information processing system 1A. As shown in FIG. 3, the information processing system 1A is a system that analyzes the progress of the work performed by the user U in the virtual space VR1 by the server 10A and displays the analysis result G1 on the terminal 50. Although FIG. 3 shows one user U, the information processing system 1A can analyze the progress of the work performed by a plurality of users U respectively.

[0025] The virtual space VR1 is displayed on the head-mounted display (hereinafter, HMD) 20 worn by the user U. Thereby, the user U can experience virtual reality as if he / she exists in the virtual space VR1.

[0026] In the example of FIG. 3, the virtual space VR1 includes objects OBJ1, OBJ2, and OBJ3 as work targets. The objects OBJ1, OBJ2, and OBJ3 are virtual objects displayed in the virtual space VR1. Hereinafter, when there is no need to particularly distinguish these objects, they will also be simply referred to as object OBJ.

[0027] User U wears the HMD 20 and the sensor 40, and operates the operation device 30 to perform work in the virtual space VR1. The work performed by user U in the virtual space VR1 includes operations on each of the objects OBJ. In other words, user U can perform a plurality of operations in the virtual space VR1. For example, the operation on the object OBJ may be to display a pointer object (not shown) displayed in the virtual space VR1 at the position of the object OBJ by operating the operation device 30. Also, for example, the operation on the object may be to virtually bring the position of the operation device 30 in the virtual space VR1 close to the position of the object OBJ and make contact. However, the operation on the object OBJ is not limited to these, and may be other operations.

[0028] Based on the orientation of the HMD 20, the sensor information from the sensor 40, and the operation information by the operation device 30, the server 10A analyzes the relevance among the changes in the work information, the changes in the line-of-sight information, and the changes in the emotion information as the work progresses, and displays the analysis result G1 on the terminal 50. The analysis result G1 is an example of the "information indicating relevance" described in the claims. Thereby, the information processing system 1A can present information that enables a more comprehensive understanding of the work performed by user U in the virtual space VR1.

[0029] <Configuration of the information processing system 1A> The configuration of the information processing system 1A according to this exemplary embodiment will be described with reference to FIG. 4. FIG. 4 is a block diagram showing the configuration of the information processing system 1A. As shown in FIG. 4, the information processing system 1A includes a server 10A, an HMD 20, an operation device 30, a sensor 40, and a terminal 50.

[0030] Server 10A is connected to each of the HMD 20 and the terminal 50 via the network N1. The network N1 is composed of, for example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public switched telephone network, a mobile data communication network, other networks, or a combination of some or all of these. The HMD 20 is communicably connected to each of the operation device 30 and the sensor 40. For example, the HMD 20 and the operation device 30 and the sensor 40 may be connected by short-range wireless communication. Although FIG. 4 shows one HMD 20 and one terminal 50 each, the number of these devices connected to the server 10A is not limited. Also, although FIG. 4 shows one operation device 30 and one sensor 40 each, the number of these devices connected to the HMD 20 is not limited.

[0031] (Configuration of Server 10A) Server 10A is a computer that analyzes the progress of the work of the user U in the virtual space VR1. The configuration of the server 10A will be described with reference to FIG. 4. As shown in FIG. 4, the server 10A includes a control unit 110A, a storage unit 120A, and a communication unit 130A. The control unit 110A controls each part of the server 10A in an overall manner. The storage unit 120A stores various data used by the control unit 110A. The communication unit 130A transmits and receives data to and from other devices under the control of the control unit 110A.

[0032] The control unit 110A includes a work information acquisition unit 11A, a gaze information acquisition unit 12A, an emotion information acquisition unit 13A, a related information output unit 14A, and a content execution unit 15A. The work information acquisition unit 11A acquires work information indicating the work that the user U is performing among a plurality of works that can be performed in the virtual space VR1. Further, the work information acquisition unit 11A acquires work information based on operation information indicating an operation on the operation device 30. The gaze information acquisition unit 12A acquires gaze information based on the orientation of the HMD 20. The emotion information acquisition unit 13A acquires emotion information based on the sensor information measured by the sensor 40. The related information output unit 14A outputs an analysis result G1 including a graph showing the change in emotion information with respect to the elapsed time of the work, and the work information or gaze information acquired at an arbitrary elapsed time point indicated by the graph. The content execution unit 15A generates a virtual space VR1 in which the user U can perform work. The content execution unit 15A is an example of a configuration that realizes the virtual space generation means described in the claims. Details of each unit included in the control unit 110A will be described in the "flow of the information processing method S1A" described later.

[0033] Also, as shown in FIG. 4, the various data stored in the storage unit 120A include a work content AP1, content data DT1, and a work progress database DB1.

[0034] (Work content AP1) The work content AP1 is an application program for providing a work environment in the virtual space VR1 to the user U. When the work content AP1 is executed by the content execution unit 15A, the work environment is provided to the user U. Hereinafter, the work that can be performed in the work environment is also described as "the work included in the work content AP1". Also, the start of providing the work environment to the user U is also described as "the user U starts the work content AP1". Also, the user U performing work in the work environment is also described as "the user U executes the work content AP1".

[0035] (Content Data DT1) Content data DT1 is data that is referred to when the content execution unit 15A executes the work content AP1. An example of the content data DT1 will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the content data DT1. As shown in FIG. 5, the content data DT1 includes information indicating the type and content of the work. Hereinafter, the work whose work type is "A", "B", or "C" may also be described as work A, work B, or work C. Work A is work for operating the object OBJ1. Work B is work for operating the object OBJ2. Work C is work for operating the object OBJ3. Work A, work B, and work C are included in the work content AP1. By referring to the content data DT1, the content execution unit 15A can determine that work A has been performed, for example, when an operation on the object OBJ1 is performed. Note that the information included in the content data DT1 is not limited to the above-described example.

[0036] (Work Progress Database DB1) The work progress database DB1 stores the work information, eye gaze information, and emotion information acquired at each point in time of the work for each user U. An example of the work progress database DB1 will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the work progress database DB1. As shown in FIG. 6, the work progress database DB1 includes a user ID, an execution date and time, an elapsed time, work information, eye gaze information, and emotion information (in this example, concentration). Hereinafter, the user U with user IDs "U1" and "U2" may also be described as user U1 and user U2. The work progress database DB1 includes a record group R1 related to user U1 and a record group R2 related to user U2. In other words, the work information includes work information related to a plurality of users U1 and U2. Also, the emotion information includes emotion information related to a plurality of users U1 and U2. Also, the eye gaze information includes eye gaze information related to a plurality of users U1 and U2. Details of the record groups R1 and R2 will be described in the "Flow of Information Processing Method S1A" described later. Note that the information stored in the work progress database DB1 and its data structure are not limited to the above-described example.

[0037] (Work information) The work information includes information indicating the start, end, or type of work. In the example of FIG. 6, the work information "Work A To Start" included in records R13 and R23 includes information indicating the start of the work and the type of work "A". Also, the work information "Work A To End" included in record R14 includes information indicating the end of the work and the type of work "A".

[0038] (Eye gaze information) The eye gaze information includes information indicating a virtual object OBJ arranged on the line of sight of the user U in the virtual space VR1. In the example of FIG. 6, the eye gaze information "Viewed OBJ1" included in records R12 and R22 indicates that the object OBJ1 was arranged on the line of sight of users U1 and U2 in the virtual space VR1.

[0039] (Emotion information) The emotional information includes information indicating the magnitude of a predetermined emotion. In the example of FIG. 6, the predetermined emotion is an emotion indicating "concentration". In this case, for example, the magnitude of the predetermined emotion is represented as "concentration degree". Also, the predetermined emotion may be information indicating "stress". In this case, for example, the magnitude of the predetermined emotion is represented as "stress degree". However, the predetermined emotion is not limited to these.

[0040] (Configuration of HMD20) The HMD20 is a virtual reality device worn by the user U to experience virtual reality. The configuration of the HMD20 will be described with reference to FIG. 4. As shown in FIG. 4, the HMD20 includes a control unit 210, a storage unit 220, a communication unit 230, a display unit 240, and a sensor 250. The control unit 210 comprehensively controls each part of the HMD20. The storage unit 220 stores various data used by the control unit 210. The communication unit 230 transmits and receives data to and from other devices under the control of the control unit 210.

[0041] Also, as shown in FIG. 3, the HMD20 is configured to be wearable on the head. The display unit 240 is arranged to be located in front of both eyes of the user U when the HMD20 is worn on the head of the user U. The display unit 240 may be a non-transmissive type or a transmissive type. Also, the HMD20 may be a sealed type that covers both eyes of the user U, or an open type like glasses. The sensor 250 is a sensor that detects the orientation of the HMD20 in the real space, and is composed of, for example, an acceleration sensor, a gyro sensor, etc. Note that the sensor 250 may be arranged outside the HMD20. In this case, for example, the sensor 250 may be a camera arranged around the user U.

[0042] The control unit 210 receives information indicating the virtual space VR1 from the server 10A and displays it on the display unit 240. Also, the control unit 210 transmits the orientation of the HMD20 (detected by the sensor 250), the orientation of the operation device 30 (detected by the operation device 30 described later), and the pulse wave of the user U (detected by the sensor 40 described later) to the server 10A.

[0043] (Operation device 30) The operation device 30 is an input device that receives the operations of the user U in the virtual space VR1. For example, the operation device 30 is configured to be holdable by the user U. Also, for example, the operation device 30 includes a sensor (not shown) that detects the orientation of the operation device 30 in the real space. In this case, based on the orientation of the operation device 30, the position indicated by the user U in the virtual space VR1 may be specified. Also, for example, a virtual pointer object (not shown) may be displayed at the position indicated by the user U in the virtual space VR1 according to the orientation of the operation device 30.

[0044] (Sensor 40) The sensor 40 measures sensor information for recognizing the emotion of the user U. In this example, the sensor 40 is a wristband-type sensor that measures the pulse wave of the user U. However, the sensor 40 does not necessarily have to be of the wristband type. Also, the sensor 40 may be any other type of sensor as long as it measures information that can be referred to in the emotion recognition technology for recognizing the emotion of the user U. Specific examples of the sensor information that can be referred to in the emotion recognition technology are as described in the exemplary embodiment 1.

[0045] (Terminal 50) The terminal 50 is a computer for browsing the analysis result G1. Hereinafter, a user who browses the analysis result G1 using the terminal 50 is referred to as a viewer. For example, the viewer may be the user U who performed the work related to the analysis result G1, or may be another user U different from the user U. Also, the viewer may be an administrator who manages a plurality of users U. In other words, the user U can view the analysis result G1 related to the work performed by himself / herself on the terminal 50. Also, the user U can view the analysis result G1 related to the work performed by another user U different from himself / herself on the terminal 50. Also, the administrator can view the analysis result G1 related to the work performed by a plurality of users U on the terminal 50.

[0046] The configuration of the terminal 50 will be described with reference to FIG. 4. As shown in FIG. 4, the terminal 50 includes a control unit 510, a storage unit 520, a communication unit 530, a display unit 540, and an input unit 550. The control unit 510 controls each part of the terminal 50 in an overall manner. The storage unit 520 stores various data used by the control unit 510. The communication unit 530 transmits and receives data to and from other devices under the control of the control unit 510. The display unit 540 is constituted by, for example, a display, and displays information under the control of the control unit 510. The input unit 550 is constituted by, for example, a touch pad or the like, and receives the input of the viewer.

[0047] Note that the display unit 540 and the input unit 550 may be integrally formed, for example, a touch panel or the like. Also, one or both of the display unit 540 and the input unit 550 may not be included inside the terminal 50, and may be connected to the outside of the terminal 50 as peripheral devices.

[0048] <Flow of the information processing method S1A> The information processing system 1A configured as described above executes the information processing method S1A according to this exemplary embodiment. The flow of the information processing method S1A will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the flow of the information processing method S1A. As shown in FIG. 7, the information processing method S1A includes steps S101 to S114.

[0049] (Step S101) In step S101, the content execution unit 15A of the server 10A executes the work content AP1 to generate a virtual space VR1 in which the user U can perform work. Also, the content execution unit 15A transmits information indicating the generated virtual space VR1 to the HMD 20.

[0050] For example, step S101 may be executed in response to an operation by the user U instructing the start of the work content AP1. Such an operation may be performed on the operation device 30, for example.

[0051] (Step S102) In step S102, the control unit 210 of the HMD 20 displays information indicating the virtual space VR1 on the display unit 240. Thereby, a work environment in which the user U wearing the HMD 20 can perform work A, work B, and work C in the virtual space VR1 is provided.

[0052] (Step S103) In step S103, the control unit 210 acquires operation information according to the operation of the user U on the operation device 30. For example, the operation information is information including the orientation of the operation device 30. Further, the control unit 210 transmits the operation information to the server 10A.

[0053] (Step S104) In step S104, the work information acquisition unit 11A of the server 10A acquires work information based on the received operation information.

[0054] For example, the work information acquisition unit 11A identifies the object OBJ operated by the user U based on the operation information. Further, the work information acquisition unit 11A refers to the content data DT1 to identify the type of work corresponding to the identified object OBJ. Here, it is assumed that the work of this type was not identified in the immediately preceding period. In this case, the work information acquisition unit 11A may determine that the work of this type has started and acquire work information indicating "starting the work of this type".

[0055] Also, for example, it is assumed that the work information acquisition unit 11A determines that there is no object OBJ operated by the user U based on the operation information. Further, it is assumed that some type of work was identified in the immediately preceding period. In this case, the work information acquisition unit 11A may determine that the work of this type has ended and acquire work information indicating "ending the work of this type".

[0056] Note that the work information acquisition unit 11A does not have to acquire work information when there is no object OBJ operated by the user U and the type of work in the immediately preceding period is not specified.

[0057] (Step S105) In step S105, the control unit 210 of the HMD 20 transmits information indicating the orientation of the HMD 20 detected by the sensor 250 to the server 10A.

[0058] (Step S106) In step S106, the line-of-sight information acquisition unit 12A of the server 10A acquires line-of-sight information based on the received information indicating the orientation of the HMD 20. For example, the line-of-sight information acquisition unit 12A calculates the line of sight of the user U in the virtual space VR1 based on the received information indicating the orientation of the HMD 20. Also, the line-of-sight information acquisition unit 12A identifies the object OBJ arranged on the line of sight. In this case, the work information acquisition unit 11A may acquire line-of-sight information of "visually recognizing the identified object OBJ".

[0059] (Step S107) In step S107, the control unit 210 of the HMD 20 transmits the sensor information acquired by the sensor 40 to the server 10A. Specifically, information indicating the pulse wave of the user U is transmitted to the server 10A.

[0060] (Step S108) In step S108, the emotion information acquisition unit 13A of the server 10A acquires the magnitude of a predetermined emotion included in the emotion information based on the received sensor information. As a method of acquiring the magnitude of a predetermined emotion based on the sensor information, a known emotion recognition technique can be adopted.

[0061] (Step S109) In step S109, the content execution unit 15A updates the virtual space VR1 according to the operation information received in step S103. Also, the content execution unit 15A transmits information indicating the updated virtual space VR1 to the HMD 20.

[0062] For example, the content execution unit 15A updates the position of the pointer object in the virtual space VR1 based on the operation information. Also, for example, the content execution unit 15A may update the display mode, position, or display mode of the virtual space VR1, etc. of the object OBJ based on the operation information according to the work content AP1.

[0063] (Step S110) In step S110, the control unit 210 of the HMD 20 displays the information indicating the updated virtual space VR1 on the display unit 240. Thereby, as a response to the operation on the operation device 30 by the user U, the virtual space VR1 in which the user U virtually exists is updated.

[0064] (Step S111) In step S111, the control unit 110A of the server 10A stores the work information, gaze information, and emotion information acquired in steps S104, S106, and S108 in the work progress database DB1 in association with the acquired elapsed time.

[0065] Note that some or all of steps S103 to S111 can be executed in a swapped order or in parallel. Also, the process including steps S103 to S111 is referred to as step S10A. Step S10A is repeated while the user U is executing the work content AP1. A specific example of the information stored in the work progress database DB1 due to the repetition of step S10A will be described with reference to FIG. 6. As shown in FIG. 6, a record group R1 related to the user U1 and a record group R2 related to the user U2 are stored in the work progress database DB1.

[0066] (Specific example of information stored in the work progress database DB1) The record group R1 shows information about the work performed by user U1 when starting work content AP1 at 12:30 on February 28, 2022. The record group R1 includes records R11, R12, R13, and R14. Record R11 indicates that information on the emotional information (concentration in this example) of "70%" was obtained at the elapsed time of "0 minutes" since the start of work content AP1. Record R12 indicates that information on the line-of-sight information of "visually recognizing object OBJ1" and the concentration of "60%" was obtained at the elapsed time of "5 minutes". Record R13 indicates that information on the work information of "starting work A" and the concentration of "50%" was obtained at the elapsed time of "10 minutes". Record R14 indicates that information on the work information of "ending work A" and the concentration of "60%" was obtained at the elapsed time of "15 minutes".

[0067] The record group R2 shows information about the work performed by user U2 when starting work content AP1 at 13:15 on February 1, 2022. The record group R2 includes records R21, R22, and R23. Record R21 indicates that information on the concentration of "65%" was obtained at the elapsed time of "0 minutes" since the start of work content AP1. Record R22 indicates that information on the line-of-sight information of "visually recognizing object OBJ1" and the concentration of "63%" was obtained at the elapsed time of "3 minutes". Record R23 indicates that information on the work information of "starting work A" and the concentration of "67%" was obtained at the elapsed time of "7 minutes".

[0068] (Step S112) In step S112, the control unit 510 of the terminal 50 acquires the condition information input by the viewer through the input unit 550. The condition information indicates conditions related to at least any one of work information, line-of-sight information, emotional information, and information related to user U. Also, the condition information indicates conditions for extracting the information to be output as the analysis result G1. The control unit 510 transmits the acquired condition information to the server 10A.

[0069] (Specific examples of condition information) Specific examples of the condition information will be described. For example, the condition information may be information indicating a condition related to the work information. Such condition information includes, for example, information specifying the start, end, or type of work. Also, for example, the condition information may be information indicating a condition related to the gaze information. Such condition information includes, for example, information specifying the object OBJ that has been visually recognized. Also, for example, the condition information may be information indicating a condition related to the emotion information. Such condition information includes, for example, the magnitude of a predetermined emotion (e.g., concentration level, stress level, etc.) being equal to or greater than a threshold value, equal to or less than a threshold value, etc. Also, there is information specifying the magnitude of a predetermined emotion (e.g., concentration level, stress level, etc.) from the highest among a plurality of users U in order up to a predetermined number of people, from the lowest in order up to a predetermined number of people, etc. Also, for example, the condition information may be information indicating a condition related to the user U. Such condition information includes, for example, information specifying the user U who is a model or a user U other than the model. In this case, which user U is the model may be set separately. Also, such condition information includes, for example, information specifying the attributes of the user U. In this case, the attributes of the user U may be acquirable. However, the condition information is not limited to the examples described above.

[0070] (Step S113) In step S113, the relevance information output unit 14A analyzes the relevance among changes in work information, changes in gaze information, and changes in emotion information as the work by the user U progresses, and generates an analysis result G1. Also, the relevance information output unit 14A transmits the analysis result G1 to the terminal 50.

[0071] Specifically, for example, the relevance information output unit 14A may output, as the analysis result G1, information that satisfies the condition indicated by the received condition information. Also, for example, the relevance information output unit 14A may output the analysis results regarding a plurality of each user U in a comparable manner.

[0072] Further, for example, the relevant information output unit 14A may output, as the analysis result G1, information including a graph showing changes in emotion information with respect to the elapsed time of the work, and work information or gaze information acquired at an arbitrary elapsed time point indicated by the graph.

[0073] Further, for example, the relevant information output unit 14A may output, as the analysis result G1, information including work information or gaze information acquired at an elapsed time point when the change in emotion information accompanying the progress of the work is large compared to before and after.

[0074] Further, for example, the relevant information output unit 14A may output, as the analysis result G1, information including information indicating the cause of a large change in emotion information accompanying the progress of the work compared to before and after. For example, as the information indicating the cause, gaze information or work information may be specified. Note that a known technique can be adopted for the technique of identifying the cause of a large change in emotion compared to before and after.

[0075] (Step S114) In step S114, the control unit 510 of the terminal 50 displays the received analysis result G1 on the display unit 540. Hereinafter, displaying on the display unit 540 is also simply described as displaying on the terminal 50.

[0076] (Specific example 1 of analysis result G1) A specific example 1 of the analysis result G1 displayed on the terminal 50 will be described with reference to FIG. 8. FIG. 8 is a diagram showing a specific example 1 of the analysis result G1 displayed on the terminal 50.

[0077] In FIG. 8, the specific example 1 of the analysis result G1 includes an analysis result G10 regarding a modeler and an analysis result G20 regarding others (excluding the modeler). For example, the modeler may be a user U set as a modeler by the administrator. Such a specific example 1 of the analysis result G1 may be displayed when information specifying "modeler" and "others (excluding the modeler)" is acquired as condition information regarding the user U in step S112.

[0078] The analysis result G10 includes a graph G1a, a speech bubble G1b indicating gaze information, and a speech bubble G1c indicating work information. The analysis result G20 includes a graph G2a, a speech bubble G2b indicating gaze information, and a speech bubble G2c indicating work information. The graphs G1a and G2a share the vertical axis and the horizontal axis. In the graphs G1a and G2a, the horizontal axis indicates the elapsed time since the start of the work content AP1, and the vertical axis indicates the concentration level (the magnitude of a predetermined emotion, which is an example of emotion information). Here, the graphs G1a, G2a sharing the horizontal axis (elapsed time) does not necessarily indicate that the modeler and those other than the modeler started the work content AP1 simultaneously. The timings at which the modeler and those other than the modeler started the work content AP1 may be the same or different. In this specific example, the analysis result G10 regarding the modeler and the analysis result G20 regarding those other than the modeler are made comparable by being displayed including the graphs G1a, G2a which share the horizontal axis and the vertical axis.

[0079] The speech bubble G1b indicates that the object OBJ1 was visually recognized at the elapsed time point t1b after the modeler started the work content AP1. The speech bubble G1c indicates that work A was started at the elapsed time point t1c after the modeler started the work content AP1.

[0080] For example, the speech bubbles G1b and G1c may be displayed when the time points t1b and t1c are specified as the elapsed time points at which the change in the concentration level of the modeler is large compared to before and after in step S113. The speech bubbles G1b and G1c may also be displayed as "information indicating the reason why the change in the concentration level is large compared to before and after".

[0081] Similarly, for the speech bubbles G2b and G2c, the same explanation applies by replacing G1 with G2 and replacing the modeler with those other than the modeler in the description regarding the speech bubbles G1b and G1c described above.

[0082] As a result, viewers can compare and understand the progress of work by the modelers and others (the rest). For example, viewers can understand that "the modelers visually recognize the object OBJ1 later than others", "the modelers start work A later than others", and "for both the modelers and others, the concentration changes significantly when they visually recognize the object OBJ1 or start work A".

[0083] (Specific Example 2 of Analysis Result G1) A specific example 2 of the analysis result G1 displayed on the terminal 50 will be described with reference to FIG. 9. FIG. 9 is a diagram showing a specific example 2 of the analysis result G1 displayed on the terminal 50.

[0084] In FIG. 9, a specific example 2 of the analysis result G1 includes an analysis result G30 regarding the user U1 and an analysis result G40 regarding the user U2. Such a specific example 2 of the analysis result G1 may be displayed when information specifying "user U1" and "user U2" is acquired as condition information regarding the user U in step S112.

[0085] Regarding the analysis results G30 and G40, they can be explained in substantially the same way by replacing "G10, G20, modelers, others, OBJ1, work A, and concentration" in the description of the specific example 1 of the analysis result G1 with "G30, G40, user U1, user U2, OBJ3, work C, and stress level".

[0086] However, a specific example 2 of the analysis result G1 is different from the specific example 1 in the following points. In the specific example 1, it was explained that the balloons G1b, G1c, G2b, and G2c are displayed for time points when the change in concentration is large compared to before and after. In the specific example 2, the balloons G3b and G4b are displayed when information specifying "visually recognizing the object OBJ3" is acquired as condition information regarding the gaze information in step S112. The balloons G3c and G4c are displayed when information specifying "starting work C" is acquired as condition information regarding the work information in step S112.

[0087] As a result, the viewer can compare and understand the progress of the work by the users U1 and U2. For example, the viewer can understand that "while user U2 starts work C after visually recognizing object OBJ3, user U1 visually recognizes object OBJ3 after starting work C", etc.

[0088] Note that in the specific example 2 of the analysis result G1, the analysis results G30 and G40 may be analysis results regarding the progress of work performed by the same user U at different timings. In this case, the viewer can compare and understand the progress of work performed by the same user U at different timings.

[0089] <Effects of this exemplary embodiment> As described above, in this exemplary embodiment, as the analysis result G1 (information indicating relevance), information including a graph showing the change in emotional information with respect to the elapsed time of work and the work information or gaze information acquired at an arbitrary elapsed time point shown by the graph is output.

[0090] Therefore, the viewer who views the analysis result G1 can more fully understand what changes in work information or gaze information there are according to the change in emotional information accompanying the progress of work.

[0091] Also, in this exemplary embodiment, as the analysis result G1 (information indicating relevance), information including the work information or gaze information acquired at the elapsed time point when the change in emotional information accompanying the progress of work is large compared to before and after is output. For example, in the graph described above, a balloon indicating the work information or gaze information is associated and displayed at a point where the change is large compared to before and after.

[0092] Therefore, the viewer who views the analysis result G1 can more fully understand what changes in work information or gaze information there are at the elapsed time point when the change in emotional information is large compared to before and after.

[0093] Also, in this exemplary embodiment, as the analysis result G1 (information indicating relevance), information including information indicating the reason why the change in emotional information over the course of work is large compared to before and after is output. For example, in the graph described above, a balloon indicating work information or gaze information as information indicating the reason is associated and displayed at points where the change is large compared to before and after.

[0094] Therefore, a viewer who views the analysis result G1 can more fully grasp the reason why the change in emotional information is large compared to before and after during the course of work (for example, whether the reason is work information or gaze information).

[0095] Also, in this exemplary embodiment, as the analysis result G1 (information indicating relevance), a configuration is adopted in which information that satisfies at least one of work information, gaze information, emotional information, and conditions related to the user U is output. Such conditions can be inputtable by the viewer, for example.

[0096] Therefore, a viewer who inputs conditions and views the analysis result G1 can view the analysis result G1 regarding the information of interest. For example, the viewer can view the analysis result G1 regarding the work information of interest (for example, the type of work), the gaze information of interest (for example, the object OBJ that was visually recognized), or the user U of interest (for example, a model user), etc.

[0097] Also, in this exemplary embodiment, the work information includes work information regarding a plurality of users U, the emotional information includes emotional information regarding a plurality of users U, the gaze information includes gaze information regarding a plurality of users U, and a configuration is adopted in which the analysis results regarding each user U are output in a comparable manner.

[0098] Therefore, a viewer who views the analysis result G1 including the analysis results for each user U in a comparable manner can more fully understand by comparing the progress of the work done by each user U. For example, the viewer can understand by comparing the progress of the work done by a plurality of users U respectively. Also, the viewer can understand by comparing the progress of the work done by the same user U at different timings respectively.

[0099] Also, in this exemplary embodiment, a configuration is adopted in which the emotion information includes information indicating the magnitude of a predetermined emotion.

[0100] Therefore, a viewer who views the analysis result G1 can grasp the correlation between the change in the magnitude of a predetermined emotion (for example, concentration level, stress level, etc.) of the user U as the work progresses and the change in the work information or the change in the gaze information.

[0101] Also, in this exemplary embodiment, a configuration is adopted in which the gaze information includes information indicating virtual objects arranged on the line of sight in the virtual space VR1.

[0102] Therefore, a viewer who views the analysis result G1 can grasp the correlation between the change in the virtual objects viewed by the user U as the work progresses and the change in the work information or the change in the emotion information.

[0103] Also, in this exemplary embodiment, a configuration is adopted in which the work information includes information indicating the start, end, or type of the work.

[0104] Therefore, a viewer who views the analysis result G1 can grasp the correlation between changes such as the start, end, or type of the work and the change in the gaze information or the change in the emotion information.

[0105] 〔Exemplary Embodiment 3〕 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0106] <Configuration of Information Processing System 1B> The configuration of the information processing system 1B according to this exemplary embodiment will be described with reference to FIG. 10. FIG. 10 is a block diagram showing the configuration of the information processing system 1B. As shown in FIG. 10, the information processing system 1B is configured substantially the same as the information processing system 1A, except that it includes a server 10B instead of the server 10A. For the same configuration as in the first exemplary embodiment, detailed description will not be repeated.

[0107] (Configuration of Server 10B) The server 10B is a computer that analyzes the progress of the work of the user U in the virtual space VR2. The configuration of the server 10B will be described with reference to FIG. 10. As shown in FIG. 10, the server 10B includes a control unit 110B, a storage unit 120B, and a communication unit 130B. The control unit 110B controls each part of the server 10B in an overall manner. The storage unit 120B stores various data used by the control unit 110B. The communication unit 130B transmits and receives data to and from other devices under the control of the control unit 110B.

[0108] The control unit 110B includes a work information acquisition unit 11B, a gaze information acquisition unit 12B, an emotion information acquisition unit 13B, a relevance information output unit 14B, a content execution unit 15B, and a video generation unit 16B. The work information acquisition unit 11B, the gaze information acquisition unit 12B, and the emotion information acquisition unit 13B are each configured in the same manner as the corresponding functional blocks in the exemplary embodiment 2. The relevance information output unit 14B is configured substantially the same as the relevance information output unit 14A, except that it outputs the analysis result G2 instead of the analysis result G1. The analysis result G2 includes a moving image described later. The content execution unit 15B is configured substantially the same as the content execution unit 15A, except that it executes the work content AP2 instead of the work content AP1. The video generation unit 16B generates a moving image obtained by photographing the virtual space VR2 using a virtual camera. Details of each unit included in the control unit 110B will be described in the "flow of the information processing method S1B" described later.

[0109] In addition, the various data stored in the storage unit 120B includes a work progress database DB2, a video database DB3, content data DT2, and work content AP2.

[0110] (Work progress database DB2) The work progress database DB2 is configured substantially the same as the work progress database DB1, and stores operation information acquired at each point in time during the work. The operation information is information indicating an operation object operated by the user U. The operation object will be described later. For example, the work progress database DB2 may store records in which each record of the work progress database DB1 shown in the specific example in FIG. 6 further includes an item of operation information. However, the information stored in the work progress database DB2 and its data structure are not limited to the above-described example.

[0111] (Video database DB3) The video database DB3 stores the recorded video that records the progress of the work by the user U. The recorded video is generated by photographing the progress of the work in the virtual space VR2 using a virtual camera. For example, the video database DB3 stores the recorded video in association with the user ID and the execution date and time. For example, the information on the progress of the work recorded in the recorded video can be obtained from the work progress database DB1 based on the user ID and the execution date and time associated with the recorded video.

[0112] For example, the virtual camera may be placed at the position of the user U in the virtual space VR2. Also, the shooting direction of the virtual camera may be set in the direction of the line of sight of the user U. In this case, the recorded video captured by the virtual camera records the work space VR2 that the user U visually recognized while performing the work. Also, for example, the virtual camera may be placed at a predetermined position in the virtual space VR2. Also, the shooting direction of the virtual camera may be a direction that includes the user U within the angle of view. In this case, the recorded video captured by the virtual camera is a video of the user U performing the work in the work space VR2 taken from a predetermined position. Such a recorded video is generated by the video generation unit 16B controlling the position, shooting direction, angle of view, etc. of the virtual camera. However, the recorded video stored in the video database DB3 and its data structure are not limited to the above-described examples.

[0113] (Work content AP2) The work content AP1 is an application program for providing a user U with a work environment in the virtual space VR2. The work content AP2 will be described in substantially the same manner as the work content AP1 in the exemplary embodiment 1. However, the virtual space VR2 generated by executing the work content AP2 is slightly different from the virtual space VR1. The virtual space VR2 will be described with reference to FIG. 11. FIG. 11 is a schematic diagram showing an example of the virtual space VR2. The virtual space VR2 includes objects OBJ1, OBJ2, and OBJ3. Further, the object OBJ1 includes operation objects OBJ1-1 and OBJ1-2 for operating the object OBJ1. The object OBJ2 includes operation objects OBJ2-1, OBJ2-2, and OBJ2-3 for operating the object OBJ2. The object OBJ3 includes operation objects OBJ3-1 and OBJ3-2 for operating the object OBJ3.

[0114] (Content data DT2) The content data DT2 is data that the content execution unit 15B refers to when executing the work content AP2. An example of the content data DT2 will be described with reference to FIG. 12. FIG. 12 is a diagram showing an example of the content data DT2. As shown in FIG. 12, the content data DT2 includes information indicating the type of work, the target object OBJ, and the correct operation pattern. For example, Work A is a work targeting the object OBJ1, and the correct pattern is to operate the operation objects OBJ1-1 and OBJ1-2 in this order. Work B is a work targeting the object OBJ2, and the correct pattern is to operate the operation objects OBJ2-1, OBJ2-3, and OBJ2-2 in this order. Work C is a work targeting the object OBJ3, and the correct pattern is to operate the operation objects OBJ3-2 and OBJ3-1 in this order. Work A, Work B, and Work C are included in the work content AP2. By referring to the content data DT2, the content execution unit 15B can determine whether the pattern of an operation is correct when an operation targeting the object OBJ is performed.

[0115] <Flow of Information Processing Method S1B> The information processing system 1B configured as described above executes the information processing method S1B according to this exemplary embodiment. The flow of the information processing method S1B will be described with reference to FIG. 13. FIG. 13 is a flowchart showing the flow of the information processing method S1B. As shown in FIG. 13, the information processing method S1B includes steps S101 to S111B and S201 to S206.

[0116] (Steps S101 to S111B) Regarding steps S101 to S111B, in the description of steps S101 to S111 of the exemplary embodiment 2, it can be explained in substantially the same way by reading "A, reference signs VR1, AP1 at the end of the reference signs" as "B, VR2, AP2". However, the difference is that steps S104B and S111B are executed instead of steps S104 and S111.

[0117] (Step S104B) In step S104B, the work information acquisition unit 11B of the server 10B acquires work information based on the received operation information. The work information includes information indicating the appropriateness of an operation associated with the work, the start, end, or type of the work. For a specific example of the process of acquiring work information including the start, end, or type of the work, it is as described in step S104. Here, a specific example of the process of acquiring work information including the appropriateness of the operation will be described. The work information acquisition unit 11B calculates the appropriateness of the operation associated with the work by comparing the received operation information with the correct operation pattern included in the content data DT2.

[0118] For example, in the work progress database DB2, it is assumed that operation information "OBJ1-2" and "OBJ1-1" are sequentially stored in association with each time point from the start to the end of work A. In this case, the operation pattern of user U is "OBJ1-2⇒OBJ1-1". The operation pattern of this user U does not match the correct operation pattern "OBJ1-1⇒OBJ1-2" of work A included in the content data DT2. In this case, the work information acquisition unit 11B acquires work information including information indicating that the operation is inappropriate (for example, "operation error") and stores it in the work progress database DB2.

[0119] (Step S111B) In step S111B, the control unit 110B of the server 10B stores the operation information, work information, gaze information, and emotion information acquired in S104B, S106, and S108 in the work progress database DB2.

[0120] (Step S201) In step S201, the video generation unit 16B controls the virtual camera to generate a recorded video of the progress of the work in the virtual space VR2.

[0121] Note that some or all of steps S103 to S111B and S201 can be executed in a different order or in parallel.

[0122] Also, the process including steps S103 to S111B and S201 is referred to as step S10B. Step S10B is repeatedly executed while user U is executing the work content AP2. By repeating step S10B, records adding operation information to the information exemplified with reference to FIG. 6 are stored in the work progress database DB2. Also, by repeating step S10B, recorded videos of the work performed by user U are stored in the video database DB3.

[0123] (Step S202) In step S202, the relevance information output unit 14B analyzes the relevance among changes in work information, changes in gaze information, and changes in emotion information as the work performed by the user U progresses, and generates an analysis result G2. Further, the relevance information output unit 14B transmits the analysis result G2 to the terminal 50.

[0124] Specifically, for example, the relevance information output unit 14B outputs, as the analysis result G2, information including (i) a recorded video stored in the video database DB3 (a moving image captured by a virtual camera arranged in the virtual space VR2 as the work progresses), and (ii) work information, emotion information, or gaze information acquired at the time point of the progress of the work corresponding to an arbitrary playback time point of the recorded video. Further, the work information included in the analysis result G2 may include the appropriateness of an operation associated with the work. Specific examples of the appropriateness of the operation are as described above.

[0125] (Step S203) In step S203, the control unit 510 of the terminal 50 displays the received analysis result G2 on the display unit 540.

[0126] (Specific example of analysis result G2) A specific example of the analysis result G2 displayed on the terminal 50 will be described with reference to FIG. 14. FIG. 14 is a diagram showing a specific example of the analysis result G2 displayed on the terminal 50.

[0127] In FIG. 14, a specific example of the analysis result G2 includes a playback area G50 of the recorded video, a seek bar G51, a marker G52 indicating the playback position, a marker G53 indicating the gaze information, a graph G54 indicating the change in emotion information, and balloons G55, G56, G57 indicating the work information.

[0128] The seek bar G51 is a figure having a width corresponding to the length of the entire playback time of the recorded video played in the playback area G50. The seek bar G51 includes a marker G52 indicating the current playback time. Also, the playback time of the recorded video corresponds to the elapsed time of the work recorded in the recorded video. In the example of FIG. 14, the marker G52 indicates that the current playback time corresponds to the elapsed time t55. The marker G52 moves as the recorded video is played and accepts an operation to change the playback time.

[0129] The marker G53 indicates the gaze information acquired at the past elapsed time t55 corresponding to the current playback time. In this example, the marker G53 is displayed superimposed on the playback area G50. The display position of the marker G53 corresponds to the position where the user U directed the line of sight in the virtual space VR2 shown by the recorded video being played in the playback area G50. In this example, the marker G53 is superimposed in the vicinity of the operation object OBJ2-2. That is, the marker G53 indicates that the user U visually recognized the operation object OBJ2-2 at the elapsed time t55. The display position of the marker G53 can move with the change of the gaze information.

[0130] The graph G54 is a graph showing the change in concentration, drawn with the seek bar G51 regarded as the horizontal axis (the axis indicating the elapsed time of the work). The balloon G55 indicates the work information "work B 、operation mistake" acquired at the past elapsed time t55 corresponding to the current playback time. An operation mistake means that the operation pattern performed by the user U for the work B does not match the correct operation pattern associated with the work B . The balloons G56 and G57 indicate the work information acquired at the past elapsed times t56 and t57.

[0131] For example, by visually recognizing the analysis result G2, the viewer can more details of the progress of the work by playing the recorded video. Also, when the user U who performed the work recorded in the recorded video is the viewer, the user U can review the progress of the work he / she performed by the recorded video.

[0132] Also, for example, by visually recognizing the analysis result G2, the viewer can recognize the change in concentration accompanying the progress of the work while playing back the recorded video. Also, the viewer can recognize that there was an operation error in Work B at the elapsed time t55 corresponding to the current playback time.

[0133] (Step S204) In step S204, the control unit 510 of the terminal 50 acquires re-execution information indicating that the work is to be performed again, which is input in response to the output of the analysis result G2 (information indicating relevance). Also, the control unit 510 transmits the acquired re-execution information to the server 10B. For example, in the example of FIG. 14, the control unit 510 accepts operations on the balloons G55, G56, G57 indicating work information as operations instructing to perform the work indicated by the balloon again. For example, among a plurality of works A, B, and C, the viewer B performs an operation on the balloon G55 in order to perform again the work B with an operation error. As a result, re-execution information indicating that the work

[0134] (Step S205) In step S205, when the content execution unit 15B of the server 10B receives the re-execution information, it generates a virtual space VR2 in which the work indicated by the re-execution information can be performed again. Also, the content execution unit 15B transmits information indicating the generated virtual space VR2 to the HMD 20. For example, B when receiving re-execution information indicating a work B in this step S205, a virtual space VR2 in which the work B can be performed is generated. For example, "a virtual space VR2 in which the work B can be performed" may be a virtual space VR2 in which only the work B can be performed and other works cannot be performed. Also, for example, when it is necessary to complete other works in advance in order to perform the work BThe "virtual space VR2" that enables the operation may be the virtual space VR2 in a state where other operations have been completed.

[0135] (Step S206) In step S206, the control unit 210 of the HMD 20 displays information indicating the virtual space VR2 on the display unit 240. Thereby, a working environment is provided for the user U wearing the HMD 20 to be able to perform the specified operation again in the virtual space VR2.

[0136] <Effects of this exemplary embodiment> As described above, in this exemplary embodiment, as the analysis result G2 (information indicating relevance), information including a recorded video (moving image) captured by a virtual camera arranged in the virtual space VR2 of the progress of the operation, and operation information, emotion information, or gaze information acquired at the time point of the progress of the operation corresponding to an arbitrary playback position of the recorded video is output.

[0137] Therefore, the viewer who views the analysis result G2 can review the operation performed by the user U by playing back the recorded video while recognizing the relevance of changes in operation information, changes in emotion information, and changes in gaze information.

[0138] Also, in this exemplary embodiment, a configuration is adopted in which when information instructing to perform the operation again is input corresponding to the output of the analysis result G2, a virtual space VR2 in which the operation can be performed again is generated. Further, when the analysis result G2 includes a plurality of operations, the input information may include information specifying the operation to be performed again, and such a configuration is adopted.

[0139] Therefore, when the viewer who views the analysis result G2 wants to perform the operation again based on the analysis result G2, the viewer can call up the virtual space VR2 in which the operation can be performed again.

[0140] In addition, in the present exemplary embodiment, a configuration is adopted in which the work information includes information indicating the appropriateness of operations associated with the work.

[0141] Therefore, the viewer who views the analysis result G2 can grasp the correlation between the change in the appropriateness of the operation and the change in the line-of-sight information or the change in the emotion information.

[0142] [Modification Example 1] Instead of outputting the analysis result G2 to the terminal 50, the present exemplary embodiment can be modified to output information indicating the correlation to the virtual space VR2 in which the process of the work performed in the past is being reproduced. That is, the information indicating the correlation is displayed on the HMD20. In this modification example, the information processing system 1B executes the information processing method S1C. The information processing method S1C according to this modification example will be described with reference to FIG. 15. FIG. 15 is a flowchart showing the flow of the information processing method S1C according to Modification Example 1. In FIG. 15, the information processing method S1C includes steps S101 to S102, S301 to S304. Steps S101 to S102 are as described with reference to FIG. 13.

[0143] (Step S301) In step S301, the content execution unit 15B updates the virtual space VR2 on the assumption that the operation indicated by the operation information stored in the work progress database DB2 has been performed. Thereby, the content execution unit 15B reproduces the process of the work performed in the past in the virtual space VR2. In addition, the content execution unit 15B transmits information indicating the updated virtual space VR2 to the HMD20.

[0144] (Step S302) In step S302, the control unit 210 of the HMD20 displays the information indicating the updated virtual space VR2 on the display unit 240. Thereby, the user U wearing the HMD20 experiences the virtual reality of looking back on the process of the work performed in the past in the virtual space VR2.

[0145] (Step S303) In step S303, the relevance information output unit 14B of the server 10B outputs information indicating relevance to the virtual space VR2 in which the progress of the work is being reproduced. The information indicating the relevance shows the relevance of the changes in the work information, the emotional information, and the gaze information obtained when the work being reproduced was performed in the past. Specifically, the relevance information output unit 14B outputs the work information, the gaze information, and the emotional information obtained at the corresponding elapsed time at the reproduction time corresponding to the elapsed time of the past work. Further, the content execution unit 15B updates the virtual space VR2 in response to the output of the work information, the gaze information, and the emotional information, and transmits information indicating the updated virtual space VR2 to the HMD 20.

[0146] For example, the relevance information output unit 14B may arrange a virtual signboard object for displaying the type of work in the virtual space VR2. In this case, the relevance information output unit 14B may display the character "Work A" on the signboard object at the reproduction time corresponding to the elapsed time of the past work when the work information "Started Work A" was obtained. Also, for example, the relevance information output unit 14B may arrange a virtual gaze object indicating the position where the user U directed the gaze in the virtual space VR2. In this case, the relevance information output unit 14B may change the position of the gaze object based on the gaze information obtained at each elapsed time of the past work. Also, for example, the relevance information output unit 14B may arrange a virtual indicator object indicating the magnitude of the emotional information in the virtual space VR2. In this case, the relevance information output unit 14B may change the magnitude indicated by the indicator object based on the emotional information obtained at each elapsed time of the past work.

[0147] (Step S304) In step S304, the control unit 210 of the HMD 20 displays the information indicating the updated virtual space VR2 on the display unit 240.

[0148] Here, the process including steps S301 to S304 is referred to as step S10C. Step S10C is repeatedly executed while the user U is executing the work content AP2. As a result, in the virtual space VR2, the work performed in the past is reproduced. Also, as the reproduction time elapses, work information, gaze information, and emotion information are output while changing. As a result, the user U can grasp the relevance of the changes in work information, gaze information, and emotion information accompanying the progress of the work while looking back on the work performed in the past in the virtual space VR2.

[0149] Note that in this modification example, the user U wearing the HMD20 and the user U who performed the reproduced work may be different. In this case, the user U wearing the HMD20 can grasp the information indicating the above-mentioned relevance while experiencing the work performed by another user U in the past in the virtual space VR2, and thus can use it as a reference when performing work in the future.

[0150] [Modification Example 2] In this exemplary embodiment, instead of outputting the analysis result G2 to the terminal 50, it can be modified to output the information indicating the relevance to the virtual space VR2 while the user U is executing the work. That is, the information indicating the relevance is displayed on the HMD20. In this modification example, the information processing system 1B executes the information processing method S1D. The information processing method S1D according to this modification example will be described with reference to FIG. 16. FIG. 16 is a flowchart showing the flow of the information processing method S1D according to Modification Example 2. In FIG. 16, the information processing method S1D includes steps S101 to S111B, S201, S401 to S402. Steps S101 to S111B and S201 are as described with reference to FIG. 13.

[0151] (Step S401) In step S401, the relevance information output unit 14B of the server 10B outputs information indicating relevance to the virtual space VR2 in which the user U is performing work. The information indicating the relevance shows the relevance of changes in work information, changes in emotion information, and changes in gaze information obtained when the same work as the work being executed was performed in the past. Specifically, the relevance information output unit 14B outputs work information, gaze information, and emotion information obtained at the elapsed time points of past work corresponding to the elapsed time points of the work being executed at each elapsed time point of the work being executed. Further, the content execution unit 15B updates the virtual space VR2 in response to the output of the work information, gaze information, and emotion information, and transmits information indicating the updated virtual space VR2 to the HMD 20. Note that the content execution unit 15B may associate the "elapsed time points of the work being executed" with the "elapsed time points of past work" based on the elapsed time of the entire work content AP2, or may associate them based on the elapsed time for each type of work.

[0152] For example, the relevance information output unit 14B may arrange a signboard object, a gaze object, and an indicator object similar to those in Modification 1 in the virtual space VR2. In this case, the relevance information output unit 14B may display characters indicating the type of work on the signboard object based on the work information obtained at the elapsed time point of past work corresponding to the elapsed time point of the work being executed. Also, for example, the relevance information output unit 14B may change the position of the gaze object based on the gaze information obtained at the elapsed time point of past work corresponding to the elapsed time point of the work being executed. Also, for example, the relevance information output unit 14B may change the size indicated by the indicator object based on the work information obtained at the elapsed time point of past work corresponding to the elapsed time point of the work being executed.

[0153] (Step S402) In step S402, the control unit 210 of the HMD 20 displays information indicating the updated virtual space VR2 on the display unit 240.

[0154] Here, the processes including steps S103 to S111B, S201, and S401 to S402 are referred to as step S10D. Step S10D is repeatedly executed while the user U is executing the work content AP2. As a result, in the virtual space VR2, as the work in progress by the user U progresses, the work information, gaze information, and emotion information acquired when the same work was performed in the past are output while changing. As a result, the user U can perform the work again while recognizing the relevance of the changes in the work information, gaze information, and emotion information when performing the work in the past.

[0155] Note that in this modified example, the user U who wears the HMD20 and performs the work may be different from the user U who performed the past work. In this case, the user U who wears the HMD20 can perform the work while referring to the information indicating the relevance of the work performed by another user U in the past.

[0156] 〔Other Modified Examples〕 In the above-described exemplary embodiments 2 and 3, other virtual reality devices may be used instead of the HMD20. Further, the HMD20 may be connected to an audio output device to reproduce the audio in the virtual space VR2. Further, the HMD20 may be connected to an audio input device, and the emotion information acquisition units 13A and 13B may acquire emotion information based on the information acquired by the sensor 40 and the audio input device. Further, in the above-described exemplary embodiments 2 and 3, part or all of the work progress databases DB1 and DB2, the video database DB3, the work contents AP1 and AP2, and the content data DT1 and DT2 may be stored in the storage unit 220 of the HMD20, or may be arranged outside the information processing systems 1A and 1B. Further, part or all of the functional blocks included in the control units 110A and 110B of the servers 10A and 10B may be arranged in other devices included in the information processing systems 1A and 1B.

[0157] 〔Examples of Realization by Software〕 Part or all of the functions of each device that constitutes the information processing systems 1, 1A, and 1B may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0158] In the latter case, each device that constitutes the information processing systems 1, 1A, and 1B is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 17. Computer C includes at least one processor C1 and at least one memory C2. A program P for operating computer C as each device that constitutes the information processing systems 1, 1A, and 1B is recorded in memory C2. In computer C, processor C1 reads and executes program P from memory C2, whereby each function of each device that constitutes the information processing systems 1, 1A, and 1B is realized.

[0159] As processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), a microcontroller, or a combination thereof can be used. As memory C2, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0160] Note that the computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and temporarily storing various data. Further, the computer C may further include a communication interface for transmitting and receiving data to and from other devices. Further, the computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0161] Also, the program P can be recorded on a non-transitory tangible recording medium M readable by the computer C. As such a recording medium M, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit can be used. The computer C can acquire the program P via such a recording medium M. Also, the program P can be transmitted via a transmission medium. As such a transmission medium, for example, a communication network or a broadcast wave can be used. The computer C can also acquire the program P via such a transmission medium.

[0162] 〔Supplementary Note 1〕 The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0163] 〔Supplementary Note 2〕 Some or all of the above-described embodiments can also be described as follows. However, the present invention is not limited to the aspects described below.

[0164] (Supplementary Note 1) work information acquisition means for acquiring work information related to work performed by a user in a virtual space; gaze information acquisition means for acquiring gaze information related to the gaze of the user; emotion information acquisition means for acquiring emotion information related to the emotion of the user; Relevance information output means for outputting information indicating the relevance among changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses; An information processing system comprising the above.

[0165] (Appendix 2) The emotion information includes information indicating the magnitude of a predetermined emotion. The information processing system according to Appendix 1.

[0166] (Appendix 3) The gaze information includes information indicating a virtual object arranged on the line of sight in the virtual space. The information processing system according to Appendix 1 or 2.

[0167] (Appendix 4) The work information includes information indicating the appropriateness of an operation associated with the work, the start, end, or type of the work. The information processing system according to any one of Appendices 1 to 3.

[0168] (Appendix 5) The relevance information output means outputs, as the information indicating the relevance, A graph showing the change in the emotion information with respect to the elapsed time of the work; Work information or gaze information acquired at an arbitrary elapsed time point indicated by the graph. The information processing system according to any one of Appendices 1 to 4 outputs information including the above. The information processing system according to any one of Appendices 1 to 4.

[0169] (Appendix 6) The relevance information output means outputs, as the information indicating the relevance, A moving image captured by a virtual camera arranged in the virtual space as the work progresses; Work information, emotion information, or gaze information acquired at the elapsed time point of the work corresponding to an arbitrary playback position of the moving image. The information processing system according to any one of Appendices 1 to 4 outputs information including the above. The information processing system according to any one of Appendices 1 to 4.

[0170] (Appendix 7) The relevant information output means outputs information indicating the relevance to the virtual space in which the process of the work performed in the past is being reproduced, and the information indicating the relevance shows the relevance of the changes in the work information, the changes in the emotion information, and the changes in the gaze information obtained when the work being reproduced was performed in the past. The information processing system according to any one of Appendices 1 to 4.

[0171] (Appendix 8) The relevant information output means outputs information indicating the relevance to the virtual space in which the user is executing the work, and the information indicating the relevance shows the relevance of the changes in the work information, the changes in the emotion information, and the changes in the gaze information obtained when the same work as the current work was performed in the past. The information processing system according to any one of Appendices 1 to 4.

[0172] (Appendix 9) further includes virtual space generation means for generating the virtual space, and the virtual space generation means generates a virtual space in which the work can be performed again when information instructing to perform the work again is input in response to the output of the information indicating the relevance. The information processing system according to any one of Appendices 1 to 8.

[0173] (Appendix 10) The relevant information output means outputs, as the information indicating the relevance, information that satisfies at least one condition regarding the work information, the gaze information, the emotion information, and the user. The information processing system according to any one of Appendices 1 to 9.

[0174] (Appendix 11) The work information includes the work information regarding a plurality of users, The emotional information includes the emotional information about the plurality of users. The gaze information includes the gaze information about the plurality of users. The relevance information output means outputs information indicating the relevance about each user in a comparable manner. The information processing system according to any one of Appendices 1 to 10.

[0175] (Appendix 12) The relevance information output means, as the information indicating the relevance, outputs information including the work information or the gaze information obtained at a point in time when the change in the emotional information with the progress of the work is large compared to before and after. The information processing system according to any one of Appendices 1 to 11.

[0176] (Appendix 13) The relevance information output means, as the information indicating the relevance, outputs information including information indicating the cause of the large change in the emotional information with the progress of the work compared to before and after. The information processing system according to any one of Appendices 1 to 12.

[0177] (Appendix 14) Obtaining work information related to work performed by a user in a virtual space, Obtaining gaze information related to the gaze of the user, Obtaining emotional information related to the emotion of the user, Outputting information indicating the relevance of the change in the work information, the change in the gaze information, and the change in the emotional information with the progress of the work, An information processing method including:

[0178] (Appendix 15) A program for causing a computer to function as an information processing system, the program causing the computer to work information acquisition means for acquiring work information related to work performed by a user in a virtual space, A line-of-sight information acquisition means for acquiring line-of-sight information regarding the line of sight of the user; An emotion information acquisition means for acquiring emotion information regarding the emotion of the user; A relevance information output means for outputting information indicating the relevance of changes in the work information, changes in the line-of-sight information, and changes in the emotion information as the work progresses; A program that functions as such.

[0179] 〔Supplementary Note 3〕 Part or all of the above-described embodiments can also be expressed as follows.

[0180] An information processing system including at least one processor, the processor performing a work information acquisition process for acquiring work information regarding work performed by a user in a virtual space, a line-of-sight information acquisition process for acquiring line-of-sight information regarding the line of sight of the user, an emotion information acquisition process for acquiring emotion information regarding the emotion of the user, and a relevance information output process for outputting information indicating the relevance of changes in the work information, changes in the line-of-sight information, and changes in the emotion information as the work progresses. Note that this information processing system may further include a memory, and a program for causing the processor to execute the work information acquisition process, the line-of-sight information acquisition process, the emotion information acquisition process, and the relevance information output process may be stored in this memory. Further, this program may be recorded on a non-transitory tangible computer-readable recording medium.

Explanation of Reference Numerals

[0181] 1, 1A, 1B Information processing system 10A, 10B Server 20 HMD 30 Operating device 40, 250 Sensor 50 Terminal 110A, 110B, 210, 510 Control unit 120A, 120B, 220, 520 Storage unit Communication units 130A, 130B, 230, 530 Operation information acquisition units 11, 11A, 11B Line-of-sight information acquisition units 12, 12A, 12B Emotion information acquisition units 13, 13A, 13B Relevant information output units 14, 14A, 14B Content execution units 15A, 15B Video generation unit 16B Display units 240, 540 Input unit 550 Processor C1 Memory C2

Claims

1. A work information acquisition means for acquiring work information related to work performed by a user in a virtual space; A gaze information acquisition means for acquiring gaze information related to the gaze of the user; An emotion information acquisition means for acquiring emotion information related to the emotion of the user; A relevance information output means for outputting information indicating the relevance of changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses; An information processing system comprising the above.

2. The emotion information includes information indicating the magnitude of a predetermined emotion, The information processing system according to claim 1.

3. The gaze information includes information indicating a virtual object arranged on the line of sight in the virtual space, and the information processing system according to claim 1 or 2.

4. The work information includes information indicating the appropriateness of an operation associated with the work, the start, end, or type of the work, The information processing system according to any one of claims 1 to 3.

5. The relevance information output means outputs, as the information indicating the relevance, A graph showing the change in the emotion information with respect to the elapsed time of the work, Work information or gaze information acquired at an arbitrary point in time indicated by the graph, and outputs information including the above, The information processing system according to any one of claims 1 to 4.

6. The relevance information output means outputs, as the information indicating the relevance, A moving image captured by a virtual camera arranged in the virtual space as the work progresses, The work information, the emotion information, or the gaze information acquired at the point in time when the work progresses corresponding to an arbitrary playback position of the moving image, and outputs information including the above, The information processing system according to any one of claims 1 to 4.

7. The relevance information output means Outputs the information indicating the relevance to a virtual space during reproduction of the progress of work performed in the past, The information indicating the relevance shows the relevance of changes in the work information, changes in the emotion information, and changes in the gaze information acquired when the work being reproduced was performed in the past, The information processing system according to any one of claims 1 to 4.

8. The relevance information output means Outputs the information indicating the relevance to a virtual space in which the user is executing the work, The information indicating the relevance shows the relevance of changes in the work information, changes in the emotion information, and changes in the gaze information acquired when the same work as the current work was performed in the past. The information processing system according to any one of claims 1 to 4.

9. Obtaining work information related to work performed by a user in a virtual space, Obtaining gaze information related to the gaze of the user, Obtaining emotion information related to the emotion of the user, Outputting information indicating the relevance of changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses, An information processing method comprising:

10. A program for causing a computer to function as an information processing system, the computer being A work information acquisition means for acquiring work information related to work performed by a user in a virtual space, A gaze information acquisition means for acquiring gaze information related to the gaze of the user, An emotion information acquisition means for acquiring emotion information related to the emotion of the user, A relevance information output means for outputting information indicating the relevance of changes in the work information, changes in the gaze information, and changes in the emotion information as the work progresses, A program for causing the computer to function as such.

Images