Information processing device, method for controlling information processing device, and program

Abstract

To provide an information processing device capable of easily performing photographing of a virtual camera without emitting a sound.SOLUTION: An information processing device 100 displays a virtual space in a display part 107. The information processing device 100 acquires line-of-sight information in the virtual space of a first user wearing the information processing device 100, and specifies a gazing target object gazed by the first user in the virtual space on ts of the line-of-sight information. The information processing device 100 acquires biological information of the first user, and estimates emotion information of the first user on the basis of the biological information of the first user. The information processing device 100 determines whether or not to satisfy a photographing condition on the basis of the line-of-sight information and the emotion information of the first user, determines a view point of a virtual camera on the basis of the emotion information of the first user, and performs photographing of the virtual camera on the basis of the determined view point in the case of determining that the photographing condition is satisfied.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, the metaverse in which multiple users communicate and experience content within the same virtual space on the Internet has attracted attention. The metaverse enables communication without being restricted by location or time. By placing and operating an avatar, which is a virtual representation of oneself, in the virtual space, users can communicate and work as if they were actually present there. In addition, users can take pictures of avatars and objects placed in the virtual space using a virtual camera.

[0003] When taking pictures in such a virtual space, generally, users need to actively activate the virtual camera in the virtual space to perform the shooting operation. However, when users are concentrating on the content in the virtual space, they cannot perform the time-consuming shooting operation simultaneously. If users can enjoy the content in the virtual space and leave a photo of the happy memories at that time, the user experience in the virtual space can be improved.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, Patent Document 1 proposes a technique for taking a photo in a virtual space using an audio signal as a trigger. Also, it is possible to control a virtual camera to take a photo with oneself and other users who can communicate with oneself as the subjects to be photographed. However, in a public place in the real world or in content where work or viewing is being done, there are situations where it is difficult to make a sound, and it may not be suitable to use the sound as a shooting trigger.

[0006] An object of the present invention is to provide an information processing apparatus, a control method for the information processing apparatus, and a program that can easily perform shooting of a virtual camera without making a sound.

Means for Solving the Problems

[0007] In order to achieve the above object, an information processing apparatus according to the present invention is an information processing apparatus that performs control to display a virtual space on a display means worn by a user, and includes means for acquiring line-of-sight information of the user in the virtual space, means for specifying an object that the user is gazing at in the virtual space based on the line-of-sight information, means for acquiring biometric information of the user, means for estimating emotion information of the user based on the biometric information of the user, means for determining whether shooting conditions are satisfied based on the line-of-sight information and the emotion information of the user, means for determining a viewpoint of a virtual camera that shoots inside the virtual space based on the emotion information of the user, and means for performing control of shooting of the virtual camera based on the determined viewpoint when it is determined that the shooting conditions are satisfied.

Effects of the Invention

[0008] According to the present invention, shooting of a virtual camera can be easily performed without making a sound.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, an information processing apparatus and its control method according to the first embodiment of the present invention will be described.

[0011] FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 is, for example, a wearable display device such as a head-mounted display or smart glasses, and is worn on the user's head.

[0012] In FIG. 1, the information processing apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an operation unit 105, a sensor unit 106, a display unit 107, and a communication I / F 108. These are connected to each other via a bus 104.

[0013] The CPU 101 is an arithmetic unit that comprehensively controls the system of the information processing apparatus 100, and executes programs stored in the ROM 102 and the like to perform various processes. The ROM 102 is a read-only non-volatile memory device that stores programs and parameters that do not require modification, such as image processing programs and initial data. The RAM 103 is a memory device that temporarily stores input information from various devices, calculation results in image processing, etc., and provides a working area for the CPU 101. The operation unit 105 is a device that receives instructions from a user, such as a switch or a controller. The sensor unit 106 includes a gyro sensor that detects the posture information of the user (hereinafter referred to as the "first user") wearing the information processing apparatus 100, a gaze sensor that detects the gaze information of the first user, an electroencephalogram sensor that detects the electroencephalogram information of the first user, and the like. The display unit 107 is a display device that displays a virtual space. The communication I / F 108 is an interface unit capable of transmitting and receiving data to and from an external device via a network.

[0014] FIG. 2 is a block diagram schematically showing the software configuration of the information processing apparatus 100 in FIG. 1. As software, the information processing apparatus 100 includes a virtual space control unit 201, a gaze information acquisition unit 202, a biological information acquisition unit 203, a fixation target acquisition unit 204, an emotion information acquisition unit 205, a shooting timing determination unit 206, a virtual camera movement unit 207, and a shooting unit 208, as shown in FIG. 2. The processing of the software shown in FIG. 2 is realized by the CPU 101 executing a program stored in the ROM 102 or the like.

[0015] The virtual space control unit 201 controls the display unit 107 to display a virtual space such as VR that holds 3D CG information. This virtual space includes virtual objects and avatars of users who have entered this virtual space. The gaze information acquisition unit 202 acquires the gaze information of the first user from the gaze sensor of the sensor unit 106. The biological information acquisition unit 203 acquires the biological information of the first user from the electroencephalogram sensor or the like of the sensor unit 106. The biological information of the first user includes, for example, the electroencephalogram data of the first user shown in FIG. 3(a) described later.

[0016] The fixation target acquisition unit 204 performs ray casting in the virtual space from the gaze information of the first user acquired by the gaze information acquisition unit 202, and identifies an object (hereinafter referred to as the "fixation target object") that the first user is fixating on in the virtual space. Examples of the fixation target object of the first user in the virtual space include virtual objects included in the virtual space and avatars of other users. The fixation target acquisition unit 204 acquires information on the fixation target object. The information on the fixation target object includes, for example, an ID for identifying the fixation target object, position and orientation information of this object, and size information of this object.

[0017] The emotion information acquisition unit 205 acquires the brain wave data of the first user from the brain wave sensor of the sensor unit 106, and estimates the emotion information of the first user from this brain wave data using a machine learning algorithm. The shooting timing determination unit 206 determines the shooting timing based on the information of the object to be gazed at obtained from the gaze target acquisition unit 204 and the emotion information of the first user obtained from the emotion information acquisition unit 205.

[0018] The virtual camera movement unit 207 determines the viewpoint of the virtual camera and controls the movement of the virtual camera so as to have the determined viewpoint. Specifically, when the shooting timing is determined, the virtual camera movement unit 207 acquires the position and orientation information of the first user and the object to be gazed at, and controls the movement of the virtual camera so that the avatar of the first user and the object to be gazed at are included in the shooting range. In this embodiment, it is assumed that the virtual camera is set to be invisible from the first user. The shooting unit 208 shoots a still image or a moving image with the virtual camera and controls to save the still image data or the moving image data in the RAM 103.

[0019] FIG. 3(a) is a diagram showing an example of the brain wave data of the first user acquired by the information processing apparatus 100 of FIG. 1 as biological information. In FIG. 3(a), the horizontal axis represents time, and the vertical axis represents the potential difference of the brain wave. The sensor unit 106 can acquire brain wave data as shown in FIG. 3(a) by observing the potential difference generated by the brain activity of the first user from the electrode unit of the brain wave sensor of the sensor unit 106. Further, the information processing apparatus 100 can estimate the emotion information of the user from the brain wave data using a machine learning algorithm. As the emotion information, that modeled by the Russell emotion circumplex model or the like can be used. Also, when estimating such emotion information, in addition to the brain wave data, other biological information such as the heartbeat and body temperature of the first user may also be used together.

[0020] Figure 3(b) represents the emotions modeled by Russell's Affect Circumplex Model. Emotions are expressed by two parameters: a valence value representing the degree of pleasure - displeasure (positive - negative), and an arousal value representing the degree of arousal - calmness. Based on these two parameters, the user's emotions can be classified into several emotions such as "joy", "sadness", "fear", and "relief".

[0021] Figure 4 is a flowchart showing the procedure of the shooting control process executed by the information processing apparatus 100 in Figure 1. The shooting control process in Figure 4 is realized by the CPU 101 executing a program stored in the ROM 102 or the like. The shooting control process in Figure 4 starts when the first user enters the virtual space and is periodically executed at a predetermined time interval while the first user is in the virtual space.

[0022] In Figure 4, first, the CPU 101 acquires the line - of - sight information of the first user from the line - of - sight sensor of the sensor unit 106 (S401). Next, the CPU 101 performs ray casting in the virtual space based on the acquired line - of - sight information and acquires the information of the object being gazed at (S402). Note that the information of the object being gazed at includes, for example, the ID for identifying the object being gazed at as described above, the position and orientation information of this object, the size information of this object, etc.

[0023] Next, the CPU 101 acquires biometric information from the biometric sensor of the sensor unit 106 (S403). The biometric information is, for example, the brain wave data of the first user as described above. Note that the biometric information may include information such as the heart rate and body temperature of the first user in addition to the brain wave data of the first user.

[0024] Next, the CPU 101 estimates the emotion information of the first user using a machine - learning algorithm from the acquired biometric information (S404). Note that the emotion information is expressed by two parameters: a valence value representing the positive - negative degree and an arousal value representing the arousal - calmness degree, by utilizing Russell's Affect Circumplex Model as described above.

[0025] Next, the CPU 101 determines whether the shooting conditions are satisfied based on the line-of-sight information of the first user and the emotional information of the first user (S405). The shooting conditions are, for example, that the object of fixation is the same for a first time interval or longer, and that the first user continues to show a positive emotion for a second time interval or longer. With these conditions, it is possible to determine whether the degree of interest of the first user in the object of fixation is high. Note that the first time interval and the second time interval may be the same or different. That is, in S405, when the object of fixation is the same for a first time interval or longer and the first user continues to show a positive emotion for a second time interval or longer, it is determined that the shooting conditions are satisfied, and this process proceeds to S406. On the other hand, when the object of fixation is not the same for a first time interval or longer, or when the first user does not continue to show a positive emotion for a second time interval or longer, it is determined that the shooting conditions are not satisfied, and this process returns to S401.

[0026] In S406, the CPU 101 determines the viewpoint of the virtual camera and moves the virtual camera so as to have the determined viewpoint. Specifically, the CPU 101 determines the viewpoint of the virtual camera so that the avatar of the first user and the object of fixation are included in the shooting range, and moves the virtual camera so as to have the determined viewpoint. Note that the emotional information of the first user may be considered in determining the viewpoint of the virtual camera.

[0027] For example, when the arousal degree of the emotional information of the first user is high, it can be inferred that the first user is engrossed in enjoying the content and is in a state of moving the body actively. In such a case, in order to capture the moving figure of the first user, the viewpoint of the virtual camera is determined so that the avatar of the first user becomes the main subject.

[0028] Also, when the arousal level of the emotional information of the first user is low, it can be inferred that the first user is relaxed and enjoying the content, and is in a state of appreciating the virtual space without moving the body. In such a case, in order to clearly understand the object being appreciated by the first user, the viewpoint of the virtual camera is determined so that the object of the fixation target becomes the main subject.

[0029] Also, the shooting direction of the previous shooting may be held, and the shooting direction of the virtual camera may be changed to a shooting direction different from the previous one for shooting. For example, the shooting direction of the virtual camera seen from the first user may be classified into a left direction, a right direction, an upper left direction, an upper right direction, etc., and the shooting direction may be changed by rotation. Also, when the avatar of the first user was shot as the main subject at the previous shooting, the control may be such that the object of the fixation target is shot as the main subject in the next shooting. Also, in the present embodiment, in addition to moving the virtual camera so that the avatar of the first user and the object of the fixation target are included in the shooting range, the virtual camera may be moved for shooting so that shooting can be performed with the same angle of view as the field of view of the first user. Also, the virtual camera may shoot not only still images but also moving images. When shooting a moving image, the virtual camera is moved so that the avatar of the first user, which is the shooting target, and the object of the fixation target are always included in the shooting range.

[0030] Next, the CPU 101 executes shooting of the virtual camera (S407). The captured image generated by this shooting is stored in the RAM 103. Then, this process ends.

[0031] FIG. 5 is a diagram showing an example of a captured image generated by the capture in S407 of FIG. 4. FIG. 5(a) is an example of a captured image generated by the capture when the first user has a positive emotion and a high level of arousal. 501 represents the avatar of the first user. 502 is a CG object that the first user is working on in the virtual space. When the first user has a positive emotion and a high level of arousal, it is presumed that the first user is concentrating on the experience in the virtual space and actively working. When recording the experience at such a time as a record, by recording the image of the user who is enjoying as the main subject in the photo, the state of enjoyment itself can be looked back on later, so the experience in the virtual space can be further improved. Also, by taking a photo together with the object that the first user is gazing at at that time, the situation at the time of photo taking can also be saved. In FIG. 5(a), the object that the first user is gazing at is the CG object that the user is working on.

[0032] FIG. 5(b) is an example of a captured image generated by the capture when the first user has a positive emotion and a low level of arousal. 503 represents the avatar of the first user. 504 is a CG object that the first user is browsing in the virtual space. When the first user has a positive emotion and a low level of arousal, it is presumed that the first user is browsing the CG object in the virtual space in a relaxed state. When recording the experience at that time as a record, by recording the CG object that the first user is browsing as the main subject in the photo, what the user was gazing at at that time can be looked back on later, so the experience in the virtual space can be further improved. Also, by taking a photo together with the avatar of the first user at that time, the situation at the time of photo taking can also be saved.

[0033] According to the above-described embodiment, it is determined whether the shooting conditions are satisfied based on the line-of-sight information of the first user and the emotional information of the first user, and the viewpoint of the virtual camera is determined based on the emotional information of the first user. When it is determined that the shooting conditions are satisfied, shooting by the virtual camera is performed based on the determined viewpoint. Thereby, shooting by the virtual camera can be easily performed without making a sound.

[0034] Also, in the above-described embodiment, since the information processing apparatus 100 includes the display unit 107 on which the virtual space is displayed, in the virtual space displayed on the display unit 107 of the information processing apparatus 100, shooting by the virtual camera can be easily performed without making a sound.

[0035] Note that the shooting conditions in S405 are not limited to the above-described conditions. For example, in addition to the above-described conditions, it may be set that a predetermined time has elapsed since the previous shooting as a shooting condition. That is, in S405, when the object of interest is the same for a first time interval or more and the first user continues to show a positive emotion for a second time interval or more and a predetermined time has elapsed since the previous shooting, it is determined that the shooting conditions are satisfied. By adding such conditions, it is possible to avoid continuously shooting the same scene.

[0036] Also, in addition to the above-described conditions, it may be set that the object of interest has changed since the previous shooting as a shooting condition. That is, in S405, when the object of interest is the same for a first time interval or more and the first user continues to show a positive emotion for a second time interval or more and the object of interest has changed since the previous shooting, it is determined that the shooting conditions are satisfied. By adding such conditions, it is possible to leave photos with various objects as records.

[0037] In addition to the above-described conditions, it may be set that the avatar of the first user and the object of attention approaching within a predetermined distance in the virtual space is set as a shooting condition. That is, in S405, when the object of attention is the same for the first time interval or more and the first user continues to show positive emotions for the second time interval or more and the avatar of the first user and the object of attention approach within a predetermined distance in the virtual space, it is determined that the shooting condition is satisfied. By adding such conditions, a photo of the scene where the avatar of the first user approaches the object of attention to a predetermined distance can be left as a record.

[0038] In addition to the above-described conditions, it may be set that the approach motion of the avatar of the first user and the object of attention stopping is set as a shooting condition. That is, in S405, when the object of attention is the same for the first time interval or more and the first user continues to show positive emotions for the second time interval or more and the approach motion of the avatar of the first user and the object of attention stops, it is determined that the shooting condition is satisfied. By adding such conditions, a photo of the scene where the approach motion of the avatar of the first user and the object of attention stops can be left as a record.

[0039] In addition, in the present embodiment, in S405, instead of the above-described conditions, it may be set that the timing when it is determined that the transition of emotions is predicted from the tendency of the time change of emotion information and the degree of positive emotion becomes the highest is set as a shooting condition. That is, in S405, when it is determined that the degree of positive emotion of the first user becomes the highest, it is determined that the shooting condition is satisfied. By setting such timing as a shooting condition, a photo of the scene where the degree of positive emotion of the first user becomes the highest can be left as a record.

[0040] In addition, in the present embodiment, emotional information of the first user used at the time of shooting the captured image may be added to the metadata of the captured image. Thereby, together with the captured image, the emotional information of the first user at the time of shooting the captured image can be left as a record.

[0041] Further, in the present embodiment, when a plurality of captured images are generated at short shooting intervals, only the captured image generated by shooting when the first user shows the most positive emotion among the plurality of captured images may be retained. Thereby, it is possible to prevent the memory of the information processing apparatus 100 from being strained by retaining unnecessary captured images.

[0042] In addition, in the present embodiment, the captured image may be recorded as a moving image instead of a still image. When recording as a moving image, time-series data of the emotional information of the acquired first user may also be recorded together. Note that the time-series data of the emotional information may be recorded in a file separate from the moving image, or may be included in the moving image file. By recording the time-series data of the emotional information of the first user together with the moving image in this way, not only the moving image but also the emotional information of the first user at the time of shooting the moving image can be left as a record.

[0043] Next, an information processing apparatus and a control method thereof according to a second embodiment of the present invention will be described. The second embodiment is basically the same as the first embodiment described above in terms of its configuration and operation, and is different from the first embodiment described above in that the shooting timing and the viewpoint of the virtual camera are controlled using the emotional information of the second user who operates the avatar that is the object to be gazed at. Therefore, the description of the overlapping configurations and operations will be omitted, and the following describes the different configurations and operations. Note that the hardware configuration of the information processing apparatus 100b according to the second embodiment is the same as the configuration shown in FIG. 1 described above. Hereinafter, the components with “b” appended to the end of the components shown in FIG. 1 will be described as the components of the information processing apparatus 100b. In addition, since the software configuration of the information processing apparatus 100b is different from the configuration shown in FIG. 2 described above, its configuration will be described with reference to FIG. 6.

[0044] FIG. 6 is a block diagram schematically showing the software configuration of the information processing apparatus 100b according to the second embodiment. As software, the information processing apparatus 100b includes, as shown in FIG. 6, the virtual space control unit 201, the gaze information acquisition unit 202, the biological information acquisition unit 203, the fixation target acquisition unit 204, the emotion information acquisition unit 205, and the imaging unit 208 described above. Further, as software, the information processing apparatus 100b includes a fixation target emotion information acquisition unit 601, a shooting timing determination unit 602, and a virtual camera movement unit 603. The processing of the software shown in FIG. 6 is realized by the CPU 101b of the information processing apparatus 100b executing a program stored in the ROM 102b or the like.

[0045] When the object of the fixation target is an avatar, the fixation target emotion information acquisition unit 601 acquires the emotion information of the second user who operates this avatar. The shooting timing determination unit 602 determines the shooting timing based on the gaze information of the first user, the emotion information of the first user, and the emotion information of the second user. The virtual camera movement unit 603 performs control to move the virtual camera in consideration of the emotion information of the first user and the emotion information of the second user.

[0046] FIG. 7 is a flowchart showing the procedure of the shooting control process executed by the information processing apparatus 100b according to the second embodiment. Note that the shooting control process in FIG. 7 is a process similar to the shooting control process in FIG. 4 described above, and hereinafter, the content different from the shooting control process in FIG. 4 described above will be particularly described. The shooting control process in FIG. 7 is realized by the CPU 101b of the information processing apparatus 100b executing a program stored in the ROM 102b or the like. Similar to the shooting control process in FIG. 4 described above, the shooting control process in FIG. 7 is also started when the first user enters the virtual space and is periodically executed at a predetermined time interval while the first user is in the virtual space.

[0047] In FIG. 7, first, the processes of S401 to S404 described above are performed. Next, when the object to be gazed at is an avatar, the CPU 101b acquires the emotion information of the second user who operates this avatar (S701). In S701, for example, the CPU 101b directly acquires the emotion information of the second user from the information processing device worn by the second user, or indirectly via a management server that provides content for experiencing the virtual space.

[0048] Next, the CPU 101b determines whether the shooting conditions are satisfied based on the gaze information of the first user, the emotion information of the first user, and the emotion information of the second user (S702). The shooting conditions in S702 are that the object to be gazed at is the same for a first time interval or more, and both the first user and the second user show a certain level or more of positive emotion. That is, in S702, when the object to be gazed at is the same for a first time interval or more and both the first user and the second user show a certain level or more of positive emotion, it is determined that the shooting conditions are satisfied, and this process proceeds to S703. On the other hand, when the object to be gazed at is not the same for a first time interval or more, or at least one of the first user and the second user does not show a certain level or more of positive emotion, it is determined that the shooting conditions are not satisfied, and this process returns to S401.

[0049] In S703, the CPU 101b determines the viewpoint of the virtual camera and moves the virtual camera so as to have the determined viewpoint. Specifically, the CPU 101b determines the viewpoint of the virtual camera so that the first user and the second user are included in the shooting range, and moves the virtual camera so as to have the determined viewpoint. Regarding the determination of the viewpoint of the virtual camera, the emotion information of the first user and the emotion information of the second user may be considered. Next, the CPU 101b executes shooting with the virtual camera (S704). The captured image generated by this shooting is stored in the RAM 103b. Then, this process ends.

[0050] FIG. 8 is a diagram showing an example of a captured image generated by the photographing in S704 of FIG. 7. FIG. 8(a) is an example of a captured image generated by photographing when both the first user and the second user have positive emotions and their arousal levels are both high. 801 represents the avatar of the first user, and 802 represents the avatar of the second user. When both the first user and the second user have positive emotions and their arousal levels are both high, it is presumed that both the first user and the second user are actively engaged in the experience of the content. In such a case, in the present embodiment, photographing with the virtual camera is performed from a viewpoint that provides a composition in which the states of each other at that time are easily visible.

[0051] FIG. 8(b) is an example of a captured image generated by photographing when the first user has a positive emotion and a high arousal level, and the second user has a positive emotion and a low arousal level. 803 represents the avatar of the first user, and 804 represents the avatar of the second user. When the arousal level of the first user is high and the arousal level of the second user is low, it is presumed that among the first user and the second user, the first user is actively engaged in the experience of the content. In such a case, photographing with the virtual camera is performed from a viewpoint that takes the avatar of the first user at that time as the main subject and also includes the avatar of the second user in the photograph.

[0052] FIG. 8(c) is an example of a captured image generated by photographing when the first user has a positive emotion and a low arousal level, and the second user has a positive emotion and a high arousal level. 805 represents the avatar of the first user, and 806 represents the avatar of the second user. When the arousal level of the first user is low and the arousal level of the second user is high, it is presumed that among the first user and the second user, the second user is actively engaged in the experience of the content. In such a case, photographing with the virtual camera is performed from a viewpoint that takes the avatar of the second user at that time as the main subject and also includes the avatar of the first user in the photograph.

[0053] By controlling the viewpoint of the virtual camera based on the emotion information of the first user and the emotion information of the second user in this way, it is possible to leave various photos with different compositions as records according to the changes in the emotions of the first user and the second user.

[0054] In addition, in the shooting of S704, when the avatars of the first user and the second user are separated by a predetermined distance in the virtual space, if they are adjusted so that their postures are within the angle of view of the same virtual camera, their postures will become small. On the contrary, the avatars of the first user and the second user may be photographed with separate virtual cameras, and two photographed images obtained by photographing with each virtual camera may be combined to generate one photographed image.

[0055] FIG. 9 is another example of a photographed image generated in the case where both the first user and the second user have positive emotions and a high degree of arousal. 901 represents the avatar of the first user, and 902 represents the avatar of the second user. When the avatars of the first user and the second user are separated by a predetermined distance in the virtual space, in S704, the CPU 101b photographs the avatar of the first user and the avatar of the second user with separate virtual cameras at the same timing. The CPU 101b stores the photographed image of FIG. 9 obtained by combining the two photographed images obtained by photographing with each virtual camera in the RAM 103b. By performing shooting in this way, it is possible to leave as a record a photo in which the avatars of the first user and the second user are easily visible.

[0056] Next, an information processing apparatus and its control method according to the third embodiment of the present invention will be described. The third embodiment is basically the same as the first and second embodiments described above in terms of its configuration and operation, and is different from the first and second embodiments described above in that the first user is photographed so as to appear together with a plurality of objects of attention directed with positive emotions. Therefore, the description of the overlapping configuration and operation will be omitted, and the following will describe the different configuration and operation. The hardware configuration of the information processing apparatus 100c according to the third embodiment is the same as the configuration shown in FIG. 1. Hereinafter, the components with "c" added to the end of the components shown in FIG. 1 will be described as the components of the information processing apparatus 100c. Further, since the software configuration of the information processing apparatus 100c is different from the configurations shown in FIGS. 2 and 6 described above, the configuration will be described with reference to FIG. 10.

[0057] FIG. 10 is a block diagram schematically showing the software configuration of the information processing apparatus 100c according to the third embodiment. As software, the information processing apparatus 100c includes, as shown in FIG. 10, the virtual space control unit 201, the gaze information acquisition unit 202, the biological information acquisition unit 203, the attention target acquisition unit 204, the emotion information acquisition unit 205, the emotion information acquisition unit 601 for the attention target, the shooting timing determination unit 602, and the shooting unit 208 described above. Further, as software, the information processing apparatus 100b includes an emotion information storage unit 1001 for the attention target and a virtual camera movement unit 1002. The software processing shown in FIG. 10 is realized by the CPU 101c of the information processing apparatus 100c executing a program stored in the ROM 102c or the like.

[0058] When the object of the attention target is an avatar, the emotion information storage unit 1001 for the attention target stores the ID for identifying the avatar and the emotion information of the first user with respect to the avatar. The virtual camera movement unit 1002 controls the movement of the virtual camera so that the avatar corresponding to the emotion information indicating a positive emotion of a certain level or more among the plurality of emotion information recorded in the emotion information storage unit 1001 for the attention target and the avatar of the first user appear together.

[0059] FIG. 11 is a flowchart showing the procedure of the shooting control process executed by the information processing apparatus 100c according to the third embodiment. Note that the shooting control process in FIG. 11 is a process similar to the shooting control processes in FIGS. 4 and 7 described above. Hereinafter, particularly, the content different from the shooting control processes in FIGS. 4 and 7 described above will be described. The shooting control process in FIG. 11 is realized by the CPU 101c of the information processing apparatus 100c executing a program stored in the ROM 102c or the like. The shooting control process in FIG. 11 is also started when the first user enters the virtual space, and is periodically executed at a predetermined time interval while the first user is in the virtual space, similar to the shooting control processes in FIGS. 4 and 7 described above.

[0060] In FIG. 11, first, the processes of S401 to S404 and S701 described above are performed. Next, when the object to be gazed at is an avatar, the CPU 101c accumulates the ID for identifying the avatar and the emotion information of the first user with respect to the avatar (S1101). Thus, in the present embodiment, the emotion information of the first user with respect to the object gazed at by the first user is accumulated.

[0061] Next, the CPU 101c determines whether or not shooting conditions are satisfied based on the gaze information of the first user, the emotion information of the first user, and the emotion information of the second user (S1102). The shooting conditions in S1102 are that the object to be gazed at is the same for a first time interval or longer, and both the emotion information of the first user and the emotion information of the second user indicate a positive emotion of a certain level or more. That is, in S1102, when the object to be gazed at is the same for a first time interval or longer and both the first user and the second user show a positive emotion of a certain level or more, it is determined that the shooting conditions are satisfied, and this process proceeds to S1103. On the other hand, when the object to be gazed at is not the same for a first time interval or longer, or when at least one of the first user and the second user does not show a positive emotion of a certain level or more, it is determined that the shooting conditions are not satisfied, and this process returns to S401.

[0062] In S1103, the CPU 101c determines the viewpoint of the virtual camera and moves the virtual camera so as to have the determined viewpoint. Specifically, the CPU 101c determines the viewpoint of the virtual camera based on the avatar of the first user, the avatar of the second user, and the avatar identified based on the accumulated emotion information, where the avatar shown by the first user with a positive emotion of a certain level or more is included in the shooting range, and moves the virtual camera so as to have the determined viewpoint. Regarding the determination of the viewpoint of the virtual camera, the shooting direction is determined in consideration of the emotion information of each user. In addition, for the selection of the avatar included in the shooting range, in addition to the condition that the avatar is within a predetermined value of the distance from the avatar of the first user to the avatar of the second user among the avatars shown by the first user with a positive emotion of a certain level or more, this may be added as a condition.

[0063] Next, the CPU 101c executes shooting with the virtual camera (S1104). The captured image generated by this shooting is stored in the RAM 103c. After that, this process ends.

[0064] FIG. 12 is a diagram showing an example of a captured image generated by the shooting in S1104 of FIG. 11. 1201 represents the avatar of the first user. 1202 represents the avatar of the second user who is the object of gaze of the first user. 1203 and 1204 respectively represent the avatar of the third user and the avatar of the fourth user, for which the first user has shown a positive emotion of a certain level or more previously. By accumulating the avatar and the emotion information associated therewith, in addition to the current object of gaze, a plurality of avatars shown by the first user with a positive emotion of a certain level or more previously can be added to the shooting target, and a photo in which a plurality of avatars appear together can be left as a record.

[0065] Note that in the present embodiment, the information processing apparatus 100 is not limited to a configuration including the display unit 107 that displays the virtual space as described above. For example, the information processing apparatus 100 may be a management server that provides content for experiencing the virtual space. A user wearing a wearable display device such as a head-mounted display accesses the management server from this wearable display device or via a PC or the like connected to this wearable display device. The management server controls the wearable display device that has accessed to display the virtual space, and performs the shooting control processes of FIGS. 4, 7, and 11 described above. For example, in S401 to S403, the management server acquires various information from the wearable display device worn by the first user or a PC or the like connected to the wearable display device. Also, in S701, the management server acquires the emotional information of the object to be gazed at from the wearable display device worn by the second user or a PC or the like connected to the wearable display device. Also, in S406, S703, and S1103, the management server determines the viewpoint of the virtual camera and transmits an instruction to move the virtual camera so as to be at the determined viewpoint to the wearable display device worn by the first user or a PC or the like connected to the wearable display device. Also, in S407, S704, and S1104, the management server transmits an instruction to execute shooting of the virtual camera to the wearable display device worn by the first user or a PC or the like connected to the wearable display device. By performing the shooting control processes of FIGS. 4, 7, and 11 described above in this way, in a configuration where the management server controls the wearable display device to display the virtual space, shooting of the virtual camera can be easily performed without making a sound.

[0066] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0067] Note that the disclosure of the present embodiment includes the following configurations and methods. (Configuration 1) An information processing apparatus that controls the display of a virtual space on a display means worn by a user, the apparatus comprising: means for acquiring line-of-sight information of the user in the virtual space; means for identifying an object being gazed at by the user in the virtual space based on the line-of-sight information; means for acquiring biometric information of the user; means for estimating emotion information of the user based on the biometric information of the user; means for determining whether shooting conditions are satisfied based on the line-of-sight information and the emotion information of the user; means for determining a viewpoint of a virtual camera that shoots inside the virtual space based on the emotion information of the user; and means for controlling shooting of the virtual camera based on the determined viewpoint when it is determined that the shooting conditions are satisfied. (Configuration 2) The information processing apparatus according to Configuration 1, wherein the determining means determines that the shooting conditions are satisfied when the object being gazed at by the user remains the same for a first time interval or more and the user continues to show a positive emotion for a second time interval or more. (Configuration 3) The information processing apparatus according to Configuration 1, wherein the determining means determines that the shooting conditions are satisfied when the object being gazed at by the user remains the same for a first time interval or more, the user continues to show a positive emotion for a second time interval or more, and the avatar of the user and the identified object approach within a predetermined distance in the virtual space. (Configuration 4) The information processing apparatus according to Configuration 1, wherein the determining means determines that the shooting conditions are satisfied when the object being gazed at by the user remains the same for a first time interval or more, the user continues to show a positive emotion for a second time interval or more, and the approaching movement of the avatar of the user and the identified object stops in the virtual space. (Configuration 5) The information processing apparatus according to Configuration 1, wherein the determining means predicts the transition of emotion from the tendency of the time change of the emotion information of the user, and determines that the shooting conditions are satisfied when it is determined that the degree of positive emotion is the highest. Configuration 6: The information processing apparatus according to any one of Configurations 1 to 5, characterized in that control is performed to retain only the captured image generated by the capture of the virtual camera when the user showed the most positive emotion among the captured images generated by the capture of the virtual camera. Configuration 7: The information processing apparatus according to any one of Configurations 1 to 6, characterized in that the emotional information of the user used at the time of the capture is attached as metadata to the captured image generated by the capture of the virtual camera. Configuration 8: The information processing apparatus according to any one of Configurations 1 to 7, characterized in that the captured image generated by the capture of the virtual camera is a still image. Configuration 9: The information processing apparatus according to any one of Configurations 1 to 8, characterized in that the captured image generated by the capture of the virtual camera is a moving image. Configuration 10: The information processing apparatus according to Configuration 9, further comprising means for performing control to save the moving image generated by the capture of the virtual camera, wherein the means for performing the saving control saves the time-series data of the emotional information of the user together with the moving image. Configuration 11: The information processing apparatus according to any one of Configurations 1 to 10, characterized in that the means for determining the viewpoint of the virtual camera determines the viewpoint of the virtual camera such that the avatar of the user and the specified object are included in the capture range in the virtual space. Configuration 12: The information processing apparatus according to any one of Configurations 1 to 11, further comprising means for acquiring the emotional information of another user when the specified object is an avatar of another user different from the user, wherein the means for determining determines whether the capture conditions are satisfied based on the gaze information, the emotional information of the user, and the emotional information of the other user, and the means for determining the viewpoint of the virtual camera determines the viewpoint of the virtual camera based on the emotional information of the user and the emotional information of the other user. (Configuration 13) When the avatar of the user and the specified object are separated by a predetermined distance in the virtual space, the means for controlling the shooting of the virtual camera performs control to cause the avatar of the user and the specified object to be shot by separate virtual cameras and to combine two shot images generated by the shooting of each virtual camera to generate one shot image. The information processing apparatus according to any one of Configurations 1 to 12, characterized in that. (Configuration 14) The information processing apparatus further includes means for accumulating emotional information directed by the user to the specified object, and the means for determining the viewpoint of the virtual camera is based on the avatar of the user, the specified object, and the accumulated emotional information in the virtual space. The information processing apparatus according to any one of Configurations 1 to 13, characterized in that the viewpoint of the virtual camera is determined so that an object showing a positive emotion by the user is included in the shooting range. (Configuration 15) The information processing apparatus includes the display means. The information processing apparatus according to any one of Configurations 1 to 14, characterized in that. (Configuration 16) The information processing apparatus is a server separate from the display means. The information processing apparatus according to any one of Configurations 1 to 14, characterized in that.

Explanation of Signs

[0068] 100, 100b, 100c Information processing apparatus 101, 101b, 101c CPU 106, 106b, 106c Sensor unit 107, 107b, 107c Display unit 201 Virtual space control unit 202 Gaze information acquisition unit 203 Biometric information acquisition unit 204 Fixation target acquisition unit 205 Emotional information acquisition unit 206, 602 Shooting timing determination unit 207, 603, 1002 Virtual camera movement unit 208 Shooting unit 601 Fixation target emotional information acquisition unit 1001 Emotional information accumulation unit for the object of gaze

Claims

1. An information processing apparatus that controls the display of a virtual space on a display means worn by a user, comprising: means for acquiring the user's gaze information in the virtual space; means for identifying an object in the virtual space that the user is gazing at based on the gaze information; means for acquiring the user's biometric information; means for estimating the user's emotional information based on the user's biometric information; means for determining whether the shooting conditions are satisfied based on the gaze information and the user's emotional information; means for determining the viewpoint of a virtual camera that shoots inside the virtual space based on the user's emotional information; and means for controlling the shooting of the virtual camera based on the determined viewpoint when it is determined that the shooting conditions are satisfied. The information processing apparatus is characterized by comprising the above.

2. The determining means determines that the shooting conditions are satisfied when the object that the user is gazing at is the same for a first time interval or more and the user shows a positive emotion for a second time interval or more. The information processing apparatus according to Claim 1, characterized in that.

3. The determining means determines that the shooting conditions are satisfied when the object that the user is gazing at is the same for a first time interval or more, the user shows a positive emotion for a second time interval or more, and the avatar of the user and the identified object in the virtual space approach within a predetermined distance. The information processing apparatus according to Claim 1, characterized in that.

4. The determining means determines that the shooting conditions are satisfied when the object that the user is gazing at is the same for a first time interval or more, the user shows a positive emotion for a second time interval or more, and the approaching movement of the avatar of the user and the identified object in the virtual space stops. The information processing apparatus according to Claim 1, characterized in that.

5. The determining means predicts the transition of the emotion from the tendency of the time change of the user's emotional information, and determines that the shooting conditions are satisfied when it is determined that the degree of positive emotion is the highest. The information processing apparatus according to Claim 1, characterized in that.

6. The information processing apparatus according to claim 1, wherein control is performed to hold only a captured image generated by capturing with the virtual camera when the user shows the most positive emotion among the captured images generated by the capturing with the virtual camera.

7. The information processing apparatus according to claim 1, wherein emotion information of the user used at the time of capturing is added as metadata to a captured image generated by capturing with the virtual camera.

8. The information processing apparatus according to claim 1, wherein the captured image generated by capturing with the virtual camera is a still image.

9. The information processing apparatus according to claim 1, wherein the captured image generated by capturing with the virtual camera is a moving image.

10. The information processing apparatus further includes means for performing control to save a moving image generated by capturing with the virtual camera, The means for performing the saving control saves time-series data of the emotion information of the user together with the moving image. The information processing apparatus according to claim 9.

11. The means for determining the viewpoint of the virtual camera determines the viewpoint of the virtual camera so that the avatar of the user and the specified object are included in the shooting range in the virtual space. The information processing apparatus according to claim 1.

12. When the specified object is an avatar of another user different from the user, the information processing apparatus further includes means for acquiring emotion information of the other user, The determining means determines whether or not the shooting conditions are satisfied based on the line-of-sight information, the emotion information of the user, and the emotion information of the other user, The means for determining the viewpoint of the virtual camera determines the viewpoint of the virtual camera based on the emotion information of the user and the emotion information of the other user. The information processing apparatus according to claim 1.

13. When the avatar of the user and the specified object are separated from each other by a predetermined distance in the virtual space, the means for controlling the shooting of the virtual camera causes the avatar of the user and the specified object to be shot with separate virtual cameras, and combines two captured images generated by the shooting of each virtual camera to generate one captured image. The information processing apparatus according to claim 1.

14. The information processing apparatus further includes means for accumulating emotion information of the user directed to the specified object. The means for determining the viewpoint of the virtual camera determines the viewpoint of the virtual camera in the virtual space such that an object that the user has shown positive feelings based on the user's avatar, the specified object, and the accumulated emotion information in the virtual space is included in the shooting range. The information processing apparatus according to claim 1.

15. The information processing apparatus according to claim 1, wherein the information processing apparatus includes the display means.

16. The information processing apparatus according to claim 1, wherein the information processing apparatus is a server separate from the display means.

17. A control method for an information processing apparatus that controls to display a virtual space on a display means worn by a user, a step of acquiring the user's line-of-sight information in the virtual space; a step of specifying an object that the user is gazing at in the virtual space based on the line-of-sight information; a step of acquiring the user's biological information; a step of estimating the user's emotion information based on the user's biological information; a step of determining whether shooting conditions are satisfied based on the line-of-sight information and the user's emotion information; a step of determining the viewpoint of a virtual camera that shoots within the virtual space based on the user's emotion information; and a step of controlling the shooting of the virtual camera based on the determined viewpoint when it is determined that the shooting conditions are satisfied. A control method for an information processing apparatus characterized by having.

18. A program for causing a computer to execute a control method for an information processing apparatus that controls to display a virtual space on a display means worn by a user, wherein the control method of the information processing apparatus includes a step of acquiring the user's line-of-sight information in the virtual space; a step of specifying an object that the user is gazing at in the virtual space based on the line-of-sight information; a step of acquiring the user's biological information; a step of estimating the user's emotion information based on the user's biological information; a step of determining whether shooting conditions are satisfied based on the line-of-sight information and the user's emotion information; a step of determining the viewpoint of a virtual camera that shoots within the virtual space based on the user's emotion information; and a step of controlling the shooting of the virtual camera based on the determined viewpoint when it is determined that the shooting conditions are satisfied. A program characterized by having.

Images