Information processing method, information processing system, and program

Abstract

The present technology relates to an information processing system and an information processing method that make it possible to suitably reduce the immersion of a user. An information control system according to the present technology comprises a presentation control unit that presents a user with content that includes an object and, at timing that corresponds to the immersion of the user in the content, changes the object in response to the stage of the interaction of the user with the object. The presentation control unit changes a plurality of types of visual characteristics of the object. The presentation control unit changes visual characteristics that include at least one of the position, depth, movement, size, and shape of the object. The present technology can be applied to systems that provide a VR or AR experience.

Description

Information processing system and information processing method

[0001] The present technology relates to an information processing system and an information processing method, and more particularly to an information processing system and an information processing method that can suitably reduce a user's sense of immersion.

[0002] There are technologies that allow users to experience virtual reality (VR) and augmented reality (AR) by presenting images that correspond to the field of view in a virtual space or by superimposing virtual objects on reality.

[0003] A user experiencing VR may experience so-called VR sickness. As a technology for reducing VR sickness, for example, Patent Document 1 describes a technology for reducing the visibility of objects. Also, as a technology for reducing VR sickness, Patent Document 2 describes a technology for performing processing on video data to reduce the sense of immersion.

[0004] JP 2019-75091 A JP 2021-180425 A

[0005] As the development of VR and AR technologies progresses, the reality of virtual objects and virtual spaces increases, blurring the boundary between reality and virtual reality, which could lead to adverse effects on the user's mind and body. To mitigate the effects on the user's mind and body, it is necessary to reduce excessive immersion and realism.

[0006] However, with conventional technology, the visual presentation is controlled from the beginning, which may make it difficult for the user to achieve a sense of immersion. Also, with conventional technology, the visual presentation is controlled without taking into account human cognitive characteristics, so the visual presentation is suddenly controlled, which may make it difficult for the user to achieve a sense of immersion or may place a psychological burden on the user.

[0007] The present technology has been made in view of such circumstances, and is intended to suitably reduce the sense of immersion felt by the user.

[0008] An information control system according to one aspect of the present technology includes a presentation control unit that presents content including an object to a user and changes the object in accordance with the stage of the user's interaction with the object at a timing that corresponds to the user's level of immersion in the content.

[0009] An information control method according to one aspect of the present technology includes an information processing system presenting content including an object to a user, and changing the object according to the stage of the user's interaction with the object at a timing according to the user's level of immersion in the content.

[0010] In one aspect of the present technology, content including an object is presented to a user, and the object is changed according to the stage of the user's interaction with the object at a timing according to the user's level of immersion in the content.

[0011] 1 is a block diagram showing an example configuration of an information processing system according to an embodiment of the present technology; FIG. 2 is a diagram showing an example of a method for detecting an interaction; FIG. 3 is a diagram showing an example of a scale of immersion; FIG. 4 is a diagram showing pathways in the human brain that process visual information; FIG. 5 is a diagram showing an example of information required at each stage of a gripping motion; FIG. 6 is a diagram showing an example of a method for detecting each stage of a gripping motion in an information processing system, and an example of an intervention method according to each stage; FIG. 7 is a flowchart explaining processing performed by an information processing system when immersion is not desired from the start; FIG. 8 is a flowchart explaining processing performed by an information processing system when it is desired to maintain an appropriate immersion state; FIG. 9 is a flowchart explaining processing performed by an information processing system 1 when it is desired to bring the user back to reality from an immersive state; FIG. 10 is a diagram showing use cases of the information processing system; FIG. 11 is a diagram showing issues in each use case and examples of an intervention method of the information processing system; FIG. 12 is a diagram showing examples of use case scenes; FIG. 13 is a diagram showing another example of a use case scene; FIG. 14 is a block diagram showing an example configuration of computer hardware.

[0012] Hereinafter, embodiments of the present technology will be described in the following order: 1. Configuration of information processing system 2. Operation of information processing system 3. Use cases 4. Modifications

[0013] <1. Configuration of Information Processing System> There are technologies that allow users to experience virtual reality (VR) or augmented reality (AR) by presenting an image corresponding to a field of view in a virtual space or by superimposing and presenting a virtual object on reality.

[0014] As the development of VR and AR technologies progresses, the reality of virtual objects and virtual spaces increases, blurring the boundary between reality and virtual reality, which could lead to adverse effects on the user's mind and body. To mitigate the effects on the user's mind and body, it is necessary to reduce excessive immersion and realism.

[0015] However, with conventional technology, the visual presentation is controlled from the beginning, which may make it difficult for the user to achieve a sense of immersion. Also, with conventional technology, the visual presentation is controlled without taking into account human cognitive characteristics, so the visual presentation is suddenly controlled, which may make it difficult for the user to achieve a sense of immersion or may place a psychological burden on the user.

[0016] It is desirable to bring the user back from an immersive state to a non-immersive state by influencing their subconscious in order to minimize the psychological burden, but the mechanism behind immersion in virtual spaces has not been clarified, and there have been few effective methods.

[0017] One embodiment of the present technology was conceived with a focus on the above points, and proposes a technology that can prevent excessive immersion in a virtual space by reducing the "realism" of virtual objects without imposing a psychological burden on the user by taking into account human cognitive characteristics and causing the user to feel uncomfortable. This embodiment will be described in detail below.

[0018] FIG. 1 is a block diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present technology.

[0019] An information processing system 1 in FIG. 1 is a system that presents content including virtual objects to a user and allows the user to experience VR or AR, for example.

[0020] As shown in Figure 1, the information processing system 1 is composed of a user sensor 11, a user status acquisition unit 12, a user DB (Database) 13, a content sensor 14, a content status acquisition unit 15, a content DB 16, an interaction detection unit 17, an immersion level estimation unit 18, a presentation control unit 19, an immersion reduction stimulus DB 20, and a presentation unit 21.

[0021] The user sensor 11 is composed of a biometric sensor, a camera, a microphone, etc. The user sensor 11 detects the user's biometric information, facial expression, voice, etc., and supplies sensor data as the detection results to the user state acquisition unit 12.

[0022] The user state acquisition unit 12 acquires the user's state based on the sensor data supplied from the user sensor 11. The user state acquisition unit 12 also acquires user information indicating the user's attributes, user information related to the user's vision, the user's past game play history, etc. from the user DB 13. The user's attributes include the user's age and gender, and the user information related to the user's vision includes the user's eyesight.

[0023] The user state acquisition unit 12 supplies the information indicating the user state and the user information to the interaction detection unit 17 .

[0024] User information is registered in advance in the user DB 13 .

[0025] The content sensor 14 is configured with a camera, a microphone, etc. The content sensor 14 detects the video and audio of the content being presented to the user, and supplies sensor data as the detection result to the content state acquisition unit 15. Note that the content sensor 14 can also acquire the object ID of the virtual object being presented to the user as sensor data.

[0026] The content status acquisition unit 15 acquires the content scene and viewing status based on the sensor data supplied from the content sensor. The content viewing status includes the device used by the user, the playback volume of the audio of the content, and the playback speed of the content.

[0027] Furthermore, the content state acquisition unit 15 acquires content information including the immersion rate and intervention type information of the content from the content DB 16 .

[0028] The immersion rate is information relating to the ease of immersion, such as the realism of virtual objects included in the content and the ease of drawing attention to the virtual objects. The intervention type information is information indicating the target value of the immersion level (the degree to which immersion is undesirable), the timing of presenting stimuli to reduce the immersion level, and the method of presenting the stimuli. The immersion rate and intervention type information are set for each scene in the content. The content state acquisition unit 15 acquires content information corresponding to the scene being presented to the user.

[0029] The content state acquisition unit 15 supplies the immersion level estimation unit 18 with information indicating the viewing state of the content and content information corresponding to the scene of the content.

[0030] Content information is registered in advance in the content DB 16 .

[0031] The interaction detection unit 17 detects the user's interaction with the virtual object based on the information supplied from the user state acquisition unit 12 .

[0032] FIG. 2 is a diagram illustrating an example of a method for detecting an interaction.

[0033] 2 , the interaction detection unit 17 acquires the user's gaze behavior, such as the gaze fixing or following of the user on a virtual object, by, for example, eye tracking, and detects the user's interaction with the virtual object based on the user's gaze behavior. Furthermore, the interaction detection unit 17 acquires the user's muscle contraction and extension by, for example, an electromyograph, and detects the user's interaction with the virtual object based on the muscle contraction and extension.

[0034] The interaction detection unit 17 acquires, for example, an event-related potential or a readiness potential by measuring brain potentials, and detects the user's interaction with a virtual object based on these brain potentials. The interaction detection unit 17 acquires, for example, the activation state and concentration state of each part of the user's brain based on cerebral blood flow, and detects the user's interaction with a virtual object based on the state of each part of the brain.

[0035] The interaction detection unit 17 acquires the result of user emotion estimation based on electroencephalogram (EEG) measurement, such as a state of concentration, tension, or relaxation, and detects the user's interaction with a virtual object based on the emotion estimation result. The interaction detection unit 17 detects the user's interaction with a virtual object based on the user's heart rate, pulse wave, sweating, and gastric action potential, for example.

[0036] The interaction detection unit 17 acquires the user's center of gravity sway and head shaking by, for example, motion capture, and detects the user's interaction with the virtual object based on these shaking movements. The interaction detection unit 17 detects the user's interaction with the virtual object based on, for example, the user's facial expression, complexion, etc. The interaction detection unit 17 acquires the frequency and tone of the user's speech by, for example, a microphone, and detects the user's interaction with the virtual object based on this information related to the voice.

[0037] The information used to detect interactions, such as the user's gaze movements, muscle contraction and contraction, brain potential, state of each part of the brain, emotion estimation results, heart rate, pulse waves, sweating, gastric action potential, center of gravity and head movement, facial expressions, complexion, and voice, is information acquired by the user state acquisition unit 12 as information indicating the user's state.

[0038] Returning to FIG. 1 , the interaction detection unit 17 supplies the interaction detection result to the immersion level estimation unit 18 and the presentation control unit 19 .

[0039] The immersion degree estimation unit 18 estimates the degree of immersion of the user in the content based on the information supplied from the content state acquisition unit 15 and the interaction detection unit 17. Specifically, the immersion degree estimation unit 18 estimates the sense of immersion the user has in the content based on the details of the user's interaction with a virtual object, and rates the degree of immersion. When content is presented in which the information processing system 1 and the user interact, the immersion degree estimation unit 18 can also estimate the degree of immersion based on the user's reaction to the interactive interaction between the user and an avatar that embodies the information processing system 1.

[0040] The degree of immersion is scaled in advance based on subjective evaluation using evaluation scales related to immersion, such as a sense of realism, a sense of agency over the controls, and a sense of physical ownership.

[0041] FIG. 3 is a diagram showing an example of a scale of the degree of immersion.

[0042] In the example of Figure 3, an excessively immersive state is indicated by an immersion level of 100% or more, and a moderately immersive state is indicated by an immersion level of 80% or more but less than 100%. A state transitioning from a non-immersed state to an immersed state is indicated by an immersion level of 50% or more but less than 80%, and a non-immersed state is indicated by an immersion level of less than 50%. Note that the immersion level values ​​shown in Figure 3 are merely examples, and the immersion level may be a non-quantitative value such as a relative evaluation value or a feature defined by machine learning.

[0043] Returning to FIG. 1 , the immersion degree estimation unit 18 supplies the estimated immersion degree to the presentation control unit 19 .

[0044] The presentation control unit 19 controls the presentation unit 21 to present content to the user. The presentation control unit 19 also controls the presentation unit 21 to intervene in the interaction while taking into account human cognitive characteristics, thereby reducing the degree of immersion of the user.

[0045] FIG. 4 shows the pathways in the human brain that process visual information.

[0046] Visual information obtained through vision is processed in the human brain via the ventral pathway, ventrodorsal pathway, and dorsodorsal pathway, respectively indicated by arrows A1 to A3 in Figure 4. The ventral pathway has the function of analyzing the position and movement of the object of action and becoming aware of the object of action. The dorsal pathway, which includes the ventrodorsal pathway and the dorsodorsal pathway, has the function of processing information about the position, movement, and shape of the object of action in a manner that is not very conscious, and causing appropriate action.

[0047] First, in the human brain, visual information that reaches the starting point of the arrows in Figure 4 is sent forward sequentially along each pathway for processing. The ventral and ventrodorsal pathways process specific positional information within the visual field, such as whether a part of the visual field is gray or moving, as shown in the top row of the speech bubble. Next, the human brain processes visual properties, not limited to a specific visual field, such as whether the target object (a cup) is cylindrical and how many pieces it appears to have, via the ventral and ventrodorsal pathways, as shown in the middle row of the speech bubble. Next, the human brain determines knowledge of the object and its existence, such as whether it is cylindrical and used to hold beverages, as shown in the bottom row of the speech bubble. Here, judgments are made using information that transcends the types of senses, not just visual perception. This change is not clear in the dorsodorsal pathway, perhaps because the processing results do not need to be brought to consciousness.

[0048] The presentation control unit 19 controls the visual characteristics of virtual objects, which are processed using the dorsal pathway in the user's brain, and acts only unconsciously, thereby intervening in the interaction in a manner that does not reach the user's consciousness. By intervening in the interaction in a manner that does not reach the user's consciousness, the information processing system 1 can reduce the user's immersion in the content without imposing a psychological burden on the user.

[0049] Here, the visual characteristics of the virtual object are controlled in accordance with the stage of the interaction detected by the interaction detection unit 17 .

[0050] 5 is a diagram showing an example of information required at each stage of a grasping motion. In the following, it is assumed that a grasping motion is detected as an interaction.

[0051] As shown in the upper part of Figure 5, the action of a person grasping an object can be divided into five stages: recognition, flexion, reaching, pre-shaping, and grasping. The recognition stage indicates that a person fixates their gaze or pays attention to the object to be grasped, while the flexion stage indicates that a person bends their elbow or changes their center of gravity. The reaching stage indicates that a person moves their hand toward the object to be grasped, while the pre-shaping stage indicates that a person spreads their fingers apart to grasp the object to be grasped. The grasping stage indicates that a person grasps the object to be grasped with their hand.

[0052] The reaching movement stage requires information about the three-dimensional position of the object to be grasped. In the human brain, the dorsal pathway processes information about the three-dimensional position of the object to be grasped, including its position and depth across the entire visual field in the egocentric coordinate system.

[0053] In the preshaping stage, information about the size and shape of the object to be grasped must be acquired through object recognition. In the human brain, information about the size and shape of the object to be grasped is processed in the dorsal pathway, where position and size information in the egocentric coordinate system is processed, and in the ventral pathway, where visual information in the object-centered coordinate system is processed. Visual information processed in the ventral pathway includes shape, color, local size, and depth information.

[0054] During the grasping process, visual information is needed to predict how to hold an object without slipping and the grip force required to grasp the object. In the human brain, the visual information required for prediction is processed in the dorsal pathway, where external properties of the object to be grasped, such as its size, position, and center of gravity, and in the ventral pathway, where internal properties of the object to be grasped, such as its hardness and mass. While external properties of an object can be identified using visual information alone, internal properties of an object cannot be identified using visual information alone and therefore must be identified by accessing knowledge in addition to visual information. By mutually utilizing the external properties processed in the dorsal pathway and the internal properties processed in the ventral pathway, the method for holding an object without slipping and the grip force required to grasp the object can be predicted.

[0055] FIG. 6 is a diagram showing an example of a method for detecting each stage of a grasping motion in the information processing system 1 and an intervention method according to each stage.

[0056] As shown in Figure 6, in the recognition stage, a virtual object is recognized as a target for grasping, and preparation for interaction is made. At this stage, the user extracts features of the virtual object to be recognized, such as its position, size, shape, center of gravity, and movement, and perceives the virtual object as a target for grasping (action). The expression of recognition behavior is detected based on the user's gaze behavior and pupil diameter obtained by eye tracking, the user's speech obtained by a microphone, and so on.

[0057] At the timing of the recognition stage, the presentation control unit 19 performs visual presentation that makes the characteristics difficult to predict, such as changing the texture of the virtual object to a texture that is unfamiliar to the real world.

[0058] In the bending stage, the elbow is bent. At this time, the user bends the elbow toward the gripping position based on the extracted characteristics. The expression of the bending movement is detected based on changes in the contraction and extension of the muscles at the base of the user's body, which are acquired by an electromyograph, and changes in the user's center of gravity, which are acquired by motion capture.

[0059] At the timing of the bending stage, the presentation control unit 19 changes the position of the virtual object to be grasped, flattens the virtual object, erases the shadow of the virtual object, and so on.

[0060] During the reaching movement, the shoulder is flexed and the elbow is extended, performing preshaping. At this time, the user extends the elbow and flexes the shoulder toward the grasping position. The user also shapes their hand in accordance with the extracted characteristics. The expression of the reaching movement is detected based on the positional movements of the user's body joints, such as the elbow and shoulder, and the shape of the hand, which are acquired by electromyography and motion capture.

[0061] At the timing of the reaching motion stage, the presentation control unit 19 provides visual presentations such as the virtual object to be grasped slipping away, a change in the moving speed of the virtual object, and a sudden change in direction of the virtual object.

[0062] In the preshaping stage, the shoulders are flexed and the elbows are extended to perform preshaping. At this time, the user further extends the elbow and flexes the shoulder toward the grasping position. As the user approaches the virtual object to be grasped, the shape of the hand is made to more closely match the shape of the virtual object. The expression of the preshaping movement is detected based on the preparatory movement (hand shape) of the body part that will perform the grasping movement, acquired by an electromyography or motion capture.

[0063] At the timing of the pre-shaping stage, the presentation control unit 19 changes the shape, size, and contour of the virtual object to be grasped.

[0064] In the grasping motion stage, the user moves their hand to the desired location. At this time, the user reaches the virtual object to be grasped. The expression of the grasping motion is detected based on the number of hands reaching the virtual object, the direction of hand movement, the number of fingers used, the angle of the fingers, etc., acquired by motion capture.

[0065] At the timing of the stage of the gripping motion, the presentation control unit 19 provides feedback that differs from the appearance of the virtual object predicted by the user.

[0066] Although the above describes an example in which a grasping action is detected as an interaction, the interaction is not limited to a grasping action, and various user actions can be detected as interactions. In this case, the interaction is divided into five stages: recognition, preparatory action, reaching action, pre-preparatory action, and actual action, which correspond to the five stages of recognition, bending, reaching action, pre-shaping, and grasping action. The number of interaction stages is not limited to five, and the interaction can be divided into any number of stages.

[0067] Returning to FIG. 1 , the presentation control unit 19 selects a stimulus appropriate for the stage of the interaction described above from among the stimuli registered in the immersion reduction stimulus DB 20 or generates a new one and presents it to the user. The presentation control unit 19 intervenes in the interaction by presenting the stimulus. The timing for presenting the stimulus is determined based on the intervention type information. Specifically, the stimulus is presented at a timing according to the degree of immersion estimated by the immersion degree estimation unit 18 and the target value of the immersion degree indicated by the intervention type information. For example, when the degree of immersion is equal to or greater than the target value and a scene is presenting a virtual object whose visual characteristics can be changed to the user, the visual characteristics of the virtual object are changed.

[0068] In the immersion reduction stimulus DB 20, each stage of interaction and a stimulus appropriate for that stage are registered in association with each other.

[0069] The presentation unit 21 is configured with a head-mounted display, a display, a speaker, etc. Under the control of the presentation control unit 19, the presentation unit 21 presents content and presents stimuli for reducing the sense of immersion at each stage of interaction.

[0070] <2. Operation of the Information Processing System> In the information processing system 1, for example, a target value for the degree of immersion is set according to the following purposes (A) to (C), and a stimulus for reducing the degree of immersion is presented at a timing based on the target value.

[0071] (A) When immersion is not desired from the beginning Examples of when immersion in content should not be desired from the beginning include when excessive immersion makes it difficult to distinguish between virtual reality and reality, posing a risk of strain on the mind and body, or when there are concerns that visual information and brain stimulation may have adverse effects on the body.

[0072] For example, experiencing psychologically demanding content such as a VR guillotine can cause a strong psychological shock. Also, if the avatar controlled by the user receives damage that would be impossible for a human to survive, or if the user experiences the illusion of dying within the content, the user's brain may perceive this information as reality.

[0073] If immersion is not desired from the beginning of the content, the information processing system 1 presents stimuli from the beginning of the content so that the degree of immersion does not approach 100%, for example.

[0074] (B) When it is desired to maintain a moderate state of immersion An example of when it is desired to maintain a moderate state of immersion is communication that takes place in a virtual space.

[0075] To make communication more enjoyable and smoother, avatars and effects are used, but if users become accustomed to this state, they may feel a gap when returning to reality, which may hinder real-world communication. Furthermore, due to the effects of avatars and effects, users may continue communicating without noticing that the other person is tired or their mood is changing. Because virtual spaces allow ideal communication and communication as desired, users may perceive virtual reality as reality and end up escaping from reality.

[0076] When it is desired to maintain an appropriate level of immersion, the information processing system 1 presents stimuli, for example, in such a way that the immersion level is maintained at 100% as much as possible and does not exceed 100%.

[0077] (C) Wanting to return from an immersive state to reality (returning to a non-immersive state) An example of wanting to return from an immersive state to reality is when excessive immersion continues for a long period of time, causing actions in the virtual space to affect reality.

[0078] For example, if a user becomes too immersed in a game while wearing a head-mounted display, they may bump into a wall or other person around them, potentially injuring themselves or causing others to get hurt. Also, users may be subject to broad-based mind control through the arbitrary guidance of content providers, such as wasting time and money while staying in a virtual space where VR shopping is available or in a pay-to-play game system for a long period of time.

[0079] When it is desired to bring the user back to reality from the immersive state, the information processing system 1 presents a stimulus to induce the level of immersion from 100% to less than 50%, for example.

[0080] The following describes the operations of the information processing system 1 according to the above-mentioned purposes (A) to (C).

[0081] The process performed by the information processing system 1 when immersion is not desired from the beginning (purpose (A)) will be described with reference to the flowchart in Fig. 7. The process in Fig. 7 is started, for example, when the presentation unit 21 starts presenting the content.

[0082] In step S1 , the content status acquisition unit 15 acquires content information from the content DB 16 .

[0083] In step S2, the user state acquisition unit 12 acquires user information from the user DB 13. For example, the user state acquisition unit 12 acquires, as the user information, user attributes, user information related to vision, and information indicating the user's past game play history.

[0084] In step S3, the content status acquisition unit 15 acquires the status of the content based on the sensor data of the content sensor 14. For example, the content status acquisition unit 15 acquires the scene and viewing status of the content being presented to the user as the content status.

[0085] In step S4, the user state acquisition unit 12 acquires the user's state based on the sensor data of the user sensor 11. For example, the user state acquisition unit 12 acquires, as the user's state, the user's gaze fixation position and gaze duration by eye tracking, or the user's speech frequency by a microphone. In addition, the user state acquisition unit 12 acquires, as the user's state, time history such as the user's content viewing time, operation start time, operation time, and response time, as well as peripheral devices and connected devices such as a head-mounted display or data glove worn by the user and a dome-shaped display used by the user to view content.

[0086] The interaction detection unit 17 detects a user's interaction with an action target based on the user's state acquired by the user state acquisition unit 12. For example, the interaction detection unit 17 determines whether the user recognizes or gazes at the action target virtual object based on gaze fixation time. When multiple virtual objects are simultaneously presented to the user, the interaction detection unit 17 detects which virtual object the user is paying attention to and which part of the virtual object the user is paying attention to. In step S5, the interaction detection unit 17 determines whether an interaction has been detected.

[0087] If it is determined in step S5 that no interaction has been detected, the process returns to step S4, and acquisition of the user's state is repeated.

[0088] On the other hand, if it is determined in step S5 that an interaction has been detected, in step S6, the immersion degree estimation unit 18 estimates the immersion degree based on the interaction detected by the interaction detection unit 17.

[0089] In step S7, the presentation control unit 19 determines whether the degree of immersion is equal to or greater than 50%, which is set as a target value.

[0090] If it is determined in step S7 that the degree of immersion is less than 50%, the process returns to step S6, and the estimation of the degree of immersion is repeated.

[0091] On the other hand, if it is determined in step S7 that the immersion level is 50% or higher, the process proceeds to step S8. For example, if the gaze stays or follows an area where a virtual object that is likely to attract negative attention is presented for 3000 ms or more, the immersion level estimation unit 18 estimates that the immersion level is 50%, and the presentation control unit 19 starts presenting a stimulus to reduce the immersion level. Note that the accuracy of estimating the immersion level may vary based on the relationship between the characteristics of the virtual object and the interaction.

[0092] In step S8, the presentation control unit 19 selects or newly generates a stimulus appropriate for the interaction stage from among the stimuli registered in the immersion reduction stimulus DB 20. For example, the presentation control unit 19 causes the presentation unit 21 to present a stimulus that moves the position of a virtual object or changes its size or shape. Specifically, the presentation control unit 19 causes the presentation unit 21 to present a stimulus that controls the visual characteristics processed in the dorsal pathway of the user's brain, such as by changing the three-dimensional representation of a virtual object to a flat one or by changing the movement of a virtual object from smooth to jerky.

[0093] In addition, in order to present stimuli around the motion object to which the user is paying attention, the presentation control unit 19 selects stimuli to be presented near the area where the line of sight is detected or stimuli to be presented in a prominent location in the scene presented to the user, and performs visual presentation that makes the user feel uncomfortable with the motion object itself. Since the stimuli need to be presented in a manner that is relatively easy to understand, the changes in the virtual object are presented in a relatively large and easy-to-understand manner.

[0094] In step S9, the presentation unit 21, under the control of the presentation control unit 19, presents a stimulus appropriate for the interaction stage to the user.

[0095] Thereafter, the process returns to step S3, and the subsequent processes are repeated.

[0096] Next, a process performed by the information processing system 1 when maintaining an appropriate immersive state (purpose (B)) will be described with reference to the flowchart in Fig. 8. The process in Fig. 8 is started, for example, when the presentation unit 21 starts presenting content that allows communication with avatars of other users.

[0097] In step S21, the content status acquisition unit 15 acquires content information from the content DB 16.

[0098] In step S22, the user state acquisition unit 12 acquires user information from the user DB 13. For example, the user state acquisition unit 12 acquires user attributes and user information related to vision as the user information.

[0099] In step S23, the content status acquisition unit 15 acquires the status of the content based on the sensor data of the content sensor 14. For example, the content status acquisition unit 15 acquires the scene and viewing status of the content being presented to the user as the content status. The content status acquisition unit 15 also acquires the speech frequency and speech tone of other users who are communication partners as the content status.

[0100] In step S24, the user state acquisition unit 12 acquires the user's state based on the sensor data of the user sensor 11. For example, the user state acquisition unit 12 acquires, as the user's state, the user's gaze fixation position and gaze duration by eye tracking, or the user's speech frequency and speech tone by a microphone. The user state acquisition unit 12 also acquires, as the user's state, time history such as the user's content viewing time, operation start time, operation time, and response time.

[0101] The interaction detection unit 17 detects user interaction with an action target based on the user state acquired by the user state acquisition unit 12. For example, the interaction detection unit 17 determines whether the user recognizes another user's avatar or whether the user is paying attention to another user's avatar based on the user's gaze fixation time and gaze tracking. When communication with multiple avatars is taking place, the interaction detection unit 17 detects which avatar the user is paying attention to and what part of the avatar the user is paying attention to. Note that this interaction detection is performed in order to detect a stable state of interaction in step S26. In step S25, the interaction detection unit 17 determines whether an interaction has been detected.

[0102] If it is determined in step S25 that an interaction has been detected, the interaction detection unit 17 determines whether the interaction is stable in step S26. For example, if the interaction has continued for a period of time that is predetermined for each content, the interaction detection unit 17 determines that the interaction is stable.

[0103] If it is determined in step S26 that the interaction is not stable, or if it is determined in step S25 that no interaction has been detected, the process returns to step S24, and acquisition of the user's state is repeated.

[0104] On the other hand, if it is determined in step S26 that the interaction is stable, the immersion degree estimation unit 18 estimates the immersion degree based on the interaction detected by the interaction detection unit 17 in step S27.

[0105] In step S28, the presentation control unit 19 determines whether the degree of immersion is equal to or greater than 90%, which is set as a target value.

[0106] If it is determined in step S28 that the degree of immersion is less than 90%, the process returns to step S27, and the estimation of the degree of immersion is repeated.

[0107] On the other hand, if it is determined in step S28 that the immersion level is 90% or higher, the interaction detection unit 17 detects the user's interaction with the action target. Note that this interaction detection is performed in order to select a stimulus appropriate for the interaction stage in steps S29 to S32.

[0108] Then, in step S29, the presentation control unit 19 determines which stage the interaction is in. An example in which a grasping action is detected as the interaction will be described below. As described above, the interaction stages are divided into five stages: recognition, bending, reaching, preshaping, and grasping, which will be referred to as the first to fifth stages below.

[0109] If it is determined in step S29 that the interaction is at the first stage (cognition), then in step S30, the presentation control unit 19 determines to change at least the visual characteristics that provide a clue to the position or depth of the virtual object.

[0110] If it is determined in step S29 that the interaction is in the second stage (bending) or the third stage (reaching movement), the presentation control unit 19 determines to change at least one of the movement, size, and shape of the virtual object in step S31. For example, if content that allows communication with a virtual idol is presented to the user, when the user reaches out toward the virtual idol avatar, the presentation control unit 19 presents a stimulus in which the virtual idol avatar runs away or becomes flat.

[0111] If the interaction is determined to be at the fourth stage (preshaping) or the fifth stage (grasping action) in step S29, the presentation control unit 19 determines in step S32 to gradually remove at least the effects of the virtual object. For example, when the user tries to grab the hand of a virtual idol avatar, the size and shape of part or all of the virtual idol avatar changes, thereby presenting a stimulus that reminds the user that the virtual idol avatar is not real. Furthermore, when content is presented to the user that superimposes avatars on other users who are actually present around the user, the presentation control unit 19 presents a stimulus that reminds the user of reality by occasionally removing the effects, such as by partially not presenting the avatars of other users and the effects on other users.

[0112] After any of the processes in steps S30 to S32 has been performed, in step S33, the presentation control unit 19 selects the stimulus determined as described above from among the stimuli registered in the immersion reduction stimulus DB 20, or generates a new one.

[0113] The presentation control unit 19 presents a stimulus near the area where the line of sight is detected in order to present a stimulus around the action target to which the user is paying attention. For example, the presentation control unit 19 presents a stimulus that changes the visual characteristics of a part of the avatar of another user as the action target itself or a virtual object presented around the avatar (another user).

[0114] In step S34, the presentation unit 21, under the control of the presentation control unit 19, presents a stimulus appropriate for the stage of the interaction to the user.

[0115] In step S35, the immersion degree estimation unit 18 estimates the immersion degree again.

[0116] In step S36, the presentation control unit 19 determines whether the immersion level is less than 90%.

[0117] If it is determined in step S36 that the immersion level is 90% or higher, the process returns to step S35, and the estimation of the immersion level is repeated while continuing to present stimuli to reduce the immersion level until the immersion level becomes less than 90%.

[0118] If it is determined in step S36 that the degree of immersion is less than 90%, the process returns to step S23, and the subsequent processes are repeated.

[0119] In the process of FIG. 8, steps S25 and S26 may be skipped, and the immersion level may be estimated and stimuli may be presented regardless of whether the interaction is in a stable state.

[0120] Next, the process performed by the information processing system 1 when it is desired to bring the user back to reality from an immersive state (purpose (C)) will be described with reference to the flowchart in Fig. 9. The process in Fig. 9 is initiated when the presentation of content begins in a situation where, for example, a user becomes so engrossed in the virtual space that their actions in the virtual space may affect reality, resulting in an accident.

[0121] In step S 51 , the content status acquisition unit 15 acquires content information from the content DB 16 .

[0122] In step S52, the user state acquisition unit 12 acquires user information from the user DB 13. For example, the user state acquisition unit 12 acquires user attributes and user information related to vision as the user information.

[0123] In step S53, the content status acquisition unit 15 acquires the status of the content based on the sensor data of the content sensor 14. For example, the content status acquisition unit 15 acquires the scene and viewing status of the content being presented to the user as the content status.

[0124] In step S54, the user state acquisition unit 12 acquires the user's state based on the sensor data of the user sensor 11. For example, the user state acquisition unit 12 acquires, as the user's state, the user's gaze fixation position and gaze duration by eye tracking, or the user's speech frequency and speech tone by a microphone. The user state acquisition unit 12 also acquires, as the user's state, time history such as the user's content viewing time, operation start time, operation time, and response time.

[0125] The interaction detection unit 17 detects user interaction with an action target based on the user state acquired by the user state acquisition unit 12. For example, the interaction detection unit 17 determines whether the user recognizes a virtual object or whether the user is paying attention to the virtual object based on the user's gaze fixation time and gaze tracking. When multiple virtual objects are simultaneously presented to the user, the interaction detection unit 17 detects which virtual object the user is paying attention to and where on the virtual object the user is paying attention. Note that this interaction detection is performed in order to detect a stable state of interaction in step S56. In step S55, it is determined whether an interaction has been detected.

[0126] If it is determined in step S55 that an interaction has been detected, the interaction detection unit 17 determines whether the interaction is stable in step S56. For example, if the interaction has continued for a period of time that is predetermined for each content, the interaction detection unit 17 determines that the interaction is stable.

[0127] If it is determined in step S56 that the interaction is not stable, or if it is determined in step S55 that no interaction has been detected, the process returns to step S54, and acquisition of the user's state is repeated.

[0128] On the other hand, if it is determined in step S56 that the interaction is stable, the immersion degree estimation unit 18 estimates the immersion degree based on the interaction detected by the interaction detection unit 17 in step S57.

[0129] For example, the immersion degree estimation unit 18 estimates the immersion degree so that the value increases in the following cases (a) to (d): (a) when none of the following (b) to (d) applies; (b) when the gaze stays or follows a virtual object that is likely to attract attention, regardless of whether it is positive or negative, for 3000 ms or more; (c) when second-stage or third-stage interactions are observed two or more times; (d) when fourth-stage or fifth-stage interactions are observed two or more times.

[0130] In step S58, the presentation control unit 19 determines whether the immersion level is equal to or greater than 100%, which is set as the first target value.

[0131] If it is determined in step S58 that the immersion level is less than 100%, the process returns to step S57, and the subsequent processes are carried out.

[0132] On the other hand, if it is determined in step S58 that the immersion level is 100% or higher, the interaction detection unit 17 detects the user's interaction with the action target. Note that this interaction detection is performed in order to select a stimulus appropriate for the interaction stage in steps S59 to S62.

[0133] After that, in step S59, the presentation control unit 19 determines which stage the interaction is in. In the following, an example in which a gripping motion is detected as an interaction will be described.

[0134] If it is determined in step S59 that the interaction is in the first stage, in step S60, the presentation control unit 19 determines to change at least the visual characteristics that provide a clue to the position or depth of the virtual object.

[0135] If it is determined in step S59 that the interaction is in the second or third stage, the presentation control unit 19 determines to change at least one of the movement, size, and shape of the virtual object in step S61. For example, when the user reaches out their hand toward the virtual object, the presentation control unit 19 presents a stimulus that causes the virtual object to move away or become flat.

[0136] If it is determined in step S59 that the interaction is at the fourth or fifth stage, the presentation control unit 19 determines to change at least the surface attributes of the virtual object in step S62. For example, when the user attempts to grab the virtual object, the presentation control unit 19 presents a stimulus that changes the visual characteristics of the virtual object, which are processed by the reflexive dorsal pathway, according to the stage of the interaction, such as changing the size or shape of part or all of the virtual object.

[0137] After any of the processes in steps S60 to S62 has been performed, in step S63, the presentation control unit 19 selects the stimulus determined as described above from among the stimuli registered in the immersion reduction stimulus DB 20, or generates a new one.

[0138] In step S64, the presentation unit 21, under the control of the presentation control unit 19, presents to the user a stimulus appropriate for the stage of the interaction.

[0139] In step S65, the immersion degree estimation unit 18 estimates the immersion degree again.

[0140] In step S66, the presentation control unit 19 determines whether the degree of immersion is less than 50%, which is the second target value.

[0141] If it is determined in step S66 that the immersion level is 50% or higher, the process returns to step S59, where the presentation unit 21 continues to present stimuli to reduce the immersion level until the immersion level falls below 50%. Note that, in order to gradually bring the immersion state back to reality, the presentation unit 21 continues to present unnatural stimuli that reduce the realism of the virtual object in accordance with the stage of interaction.

[0142] If it is determined in step S66 that the degree of immersion is less than 50%, the process returns to step S53, and the subsequent processes are repeated.

[0143] As described above, in the information processing system 1, a stimulus that changes a virtual object according to the stage of the user's interaction with the virtual object is presented to the user at a timing according to the user's level of immersion in the content.

[0144] By changing the visual characteristics of virtual objects that are processed in the dorsal pathway of the user's brain at each stage of interaction, it is possible to reduce the level of immersion in the content without placing a psychological burden on the user.

[0145] Changing the visual characteristics of the virtual object at the timing of the premonition stage, which includes the recognition stage and the preparatory movement stage, can have a more unconscious effect. When the level of immersion is equal to or greater than a predetermined threshold, the information processing system 1 can change the visual characteristics of the virtual object at the timing of the premonition stage to make the user recognize that the virtual object is a virtual object without the user being very conscious of it. This makes it possible to prevent a sudden drop in the level of immersion while the user is immersed in content, due to the user being made to recognize that the virtual object is a virtual object.

[0146] When the level of immersion is lower than a predetermined threshold, it is not necessary to consider a sudden drop in the level of immersion, and therefore the information processing system 1 can change the visual characteristics of the virtual object at the timing of the precursor stage and the main stage, which includes the reaching action stage, the pre-action action stage, and the action action stage. Furthermore, when the level of immersion is lower than a predetermined threshold, the information processing system 1 can change the virtual object before an interaction occurs.

[0147] Furthermore, the information processing system 1 estimates the user's immersion level based on the content of the interaction. Since the user's state can be accurately estimated based on the content of the interaction, the user's immersion level can also be accurately estimated. Furthermore, the information processing system 1 can evaluate the stability of the interaction or the content itself based on the content of the interaction.

[0148] 3. Use Cases FIG. 10 is a diagram showing use cases of the information processing system 1. As shown in FIG.

[0149] As shown on the left side of Figure 10, stakeholders who use the information processing system 1 can be three parties: the provider of the information processing system 1, the user, and others (other users) who use the information processing system 1 together with the user.

[0150] In use case 1, a virtual object is an example of a target with which the stakeholder will form an interaction via the information processing system 1, as shown on the right side of Fig. 10. In use case 2, a human-like character is an example of a target with which the stakeholder will form an interaction via the information processing system 1. In use case 3, another person is an example of a target with which the stakeholder will form an interaction via the information processing system 1.

[0151] The virtual objects that are the targets of interaction, the character avatars, and the avatars of other people are all virtual objects within the virtual space.

[0152] FIG. 11 is a diagram showing examples of problems in each use case (UC) and intervention methods of the information processing system 1.

[0153] As shown in FIG. 11, in use case 1, the actors in the content are the user, the virtual object, and the provider.

[0154] An example of a scenario in Use Case 1 is a scenario in which entertainment content and non-entertainment content are experienced. In such a scenario, the first issue is that virtual reality and reality may be confused.

[0155] For example, when experiencing content using a head-mounted display or dome-shaped display, a user may wave their hand around, thinking they are moving in a virtual space, and end up hitting a real wall and injuring themselves. If a user mistakenly believes a virtual object to be part of their own hand, as in the rubber hand illusion, they may be fooled into thinking their hand is hitting an object when in fact it is not, and may pull their hand back forcefully, causing it to hit something. Furthermore, if a user mistakenly believes a virtual object to be part of their own body, they may get the feeling that their body is an extension of their own body, stretching their foot out of reach of an object and tripping, leading to a real-world accident.

[0156] As shown in Figure 12, when a user wearing a head-mounted display 51 is actually walking along an elliptical wall, content is presented that misleads the user into thinking that they are walking along a straight path in the virtual space, which can lead the user to continue walking indefinitely.

[0157] In this way, people may become so immersed in highly realistic games that their actions in virtual reality extend to reality.

[0158] Therefore, the information processing system 1 presents the user with a stimulus that transitions the user from a highly immersive state to a non-immersive state, with the objective of bringing the user back from the immersive state to reality (objective (C)). For example, the information processing system 1 presents the user with a stimulus that induces the user to reduce the immersion level from 100% to 80% or less.

[0159] Specifically, the information processing system 1 senses the movement of the hand or other device using a data glove when the user uses a head-mounted display, and senses the movement of the hand or other device using a marker or the like when the user uses a dome-shaped display. If interaction from the second stage onwards is detected several times, the information processing system 1 presents a stimulus appropriate for the interaction stage. For example, if the user tries to reach out their hand towards a virtual object, the information processing system 1 may deform part of the virtual object or make the virtual object move in an unpredictable manner, giving the user a sense of discomfort.

[0160] It should be noted that the visual characteristics of all virtual objects included in the content are not made changeable, but rather virtual objects with changeable visual characteristics are presented in a scene where a stimulus presentation is anticipated in advance. The virtual objects with changeable visual characteristics are presented in positions that naturally attract the user's attention or where the user is likely to pause.

[0161] Presenting stimuli to reduce immersion in a highlight scene of the content may reduce the enjoyment, so the information processing system 1 presents stimuli in scenes that do not reduce the enjoyment, based on intervention type information registered in the content DB 16 indicating whether intervention is possible for each scene.

[0162] If it is predicted that the level of immersion will be high based on the user information and content information, the information processing system 1 starts presenting stimuli at the precursory stage of the interaction to reduce the psychological burden on the user.

[0163] The information processing system 1 may generate an event that causes a user to interact with a virtual object based on the difference between the immersion level and the target value. For example, when no interaction is occurring but the user wants to reduce the immersion level, the information processing system 1 may present a stimulus that corresponds to the stage of the interaction that has occurred in response to the event.

[0164] Returning to Figure 11, in the scene of Use Case 1, where entertainment content and non-entertainment content are experienced, there is a second issue in that the emphasis on reality in the virtual space places a strain on the mind and body.

[0165] For example, experiencing psychologically demanding content such as a VR guillotine can cause a strong psychological shock. Also, if the avatar controlled by the user receives damage that would be impossible for a human to survive, or if the user experiences the illusion of dying within the content, the user's brain may perceive this information as reality.

[0166] In this way, spending a long time in a virtual space makes the user become more integrated with the avatar, and the visual information and brain stimulation can have adverse effects on the body.

[0167] Therefore, the information processing system 1 presents stimuli to maintain a moderate level of immersion and to balance immersion with objective content viewing, with the objective of preventing immersion from the beginning (objective (A)). For example, the information processing system 1 presents stimuli from the beginning of the content so that the level of immersion does not approach 100%.

[0168] In this case, the information processing system 1 must allow the user to view the content from the start to the end of the content viewing without immersing the user to the point where it feels like a real experience, but with an objectivity similar to that of watching a realistic video on a television receiver. To achieve this, the information processing system 1 preferably presents a stimulus before an interaction occurs. However, if a stimulus is presented when the user is not attempting to perform any action on the virtual object, there is a concern that it may hinder the user's sense of immersion and realism. Therefore, it is preferable to present a stimulus at a timing preceding the interaction.

[0169] Furthermore, when an interaction is detected, even if the interaction is in the precursor stage, the information processing system 1 presents a stimulus that is likely to make the user feel uncomfortable as soon as they see it, such as changing the surface attributes of the virtual object to surface attributes that give the impression of properties that do not exist in reality (such as soft glass).When an interaction at this stage is detected, the information processing system 1 presents a stimulus that further makes the user feel uncomfortable.

[0170] Next, in the scenario of Use Case 1, where entertainment content and non-entertainment content are experienced, there is a third challenge, which is different from the second challenge, in that the emphasis on reality in the virtual space places a strain on the mind and body.

[0171] 13, a user can use a head-mounted display 51 and a controller 52 to experience shopping in a virtual space, just as if they were looking at and purchasing clothes in a store. If the user goes overboard with purchasing behavior or watching videos in a virtual space, they may waste time and money and experience psychological stress.

[0172] Even if a user wants to stop an experience, they may continue due to mechanisms on the part of the content provider that make it difficult to stop, such as advertising functions or recommendations based on AI (Artificial Intelligence).In this way, users may fall prey to a broad form of mind control, which is the arbitrary guidance of content providers.

[0173] Since it is problematic for the user to remain in the virtual space where the content is experienced, the information processing system 1 presents stimuli to the user, such as reminding the user that he or she can decide whether or not to continue the experience of his or her own volition, or forcing the user to leave the virtual space where the content is experienced, with the aim of bringing the user back to reality from the immersive state (aim (C)). For example, the information processing system 1 presents the user with stimuli that induce the level of immersion to decrease from 100% to 80% or less.

[0174] Specifically, when the level of immersion is 100% or higher, the information processing system 1 presents a stimulus at the first stage of the interaction, and subsequently presents stimuli according to the stage of the interaction. To present stimuli multiple times, the information processing system 1 needs to repeatedly trigger an event (situation) in which the user attempts to grasp a virtual object. The information processing system 1 triggers events such as selecting a product in shopping content or picking up a virtual object in game content until the level of immersion decreases.

[0175] Furthermore, if the information processing system 1 detects multiple interactions from the second stage onward, it presents stimuli according to the detected interaction stage. For example, if the user tries to reach out for a virtual object, the information processing system 1 presents a stimulus to move the virtual object away from the user, reminding the user that they can interrupt the experience or decide whether or not to purchase it at their own will.

[0176] When the immersion level is 50% or less, the information processing system 1 may present a stimulus that reduces the image quality of all or part of the content, regardless of the stage of interaction.When the immersion level is 50% or less, a greater number of types of visual characteristics may be combined and changed.

[0177] 11 , examples of use case 1 include remote medical care and driving simulations. In such cases, there is a problem that the emphasis on reality in the virtual space places a strain on the mind and body.

[0178] For example, in a virtual space, it is possible to create an environment that makes it easier to concentrate on a task by blocking out external disturbances and emphasizing the stimuli one experiences. However, as one continues working, fatigue may accumulate without one realizing it.

[0179] Since it becomes a problem if the user continues to stay in the virtual space where the content is experienced, the information processing system 1 presents the user with a stimulus that forcibly urges the user to leave the virtual space where the content is experienced, with the objective of bringing the user back from the immersive state to reality (objective (C)). For example, the information processing system 1 presents the user with a stimulus that induces the level of immersion to decrease from 100% to 80% or less.

[0180] Specifically, the information processing system 1 presents stimuli that gradually make the user aware that they are in a virtual space by deforming or shrinking the virtual objects presented in the user's field of view so as not to disturb the user's concentration, thereby preventing the user from becoming too immersed.

[0181] Next, in Use Case 2, the actors in the content are users, characters, and providers. Characters include virtual personalities such as virtual idols, virtual family members, humans created by AI, and personalities created by AI.

[0182] Use case 2 scenarios include communicating with virtual idols, virtual families, people created by AI, and personalities created by AI. In these scenarios, there is the issue of confusing virtual reality with reality.

[0183] For example, by communicating with a non-existent character, a user may believe that the character is real, and when they find out that the character is not real, they may feel a sense of loss or disappointment, making it impossible for them to accept reality. Also, in a virtual space, it may be possible to make the virtual personality of a deceased person speak or move as if they were alive, which could cause problems.

[0184] When communicating in a virtual space instead of meeting in real life to negotiate business or find a partner, there is a possibility that the other person may use a virtual personality to participate in the communication in place of the other person. If this communication is successful, there is a possibility that when the two people actually meet, the users will be disappointed due to the gap between their image of the other person and the reality.

[0185] In this way, when communicating with a non-existent personality (such as a reproduction of a deceased person or a virtual idol), the virtual space allows the user to experience communication that is ideal for them or that they desire, which makes the user feel as if the virtual reality is real.

[0186] Therefore, the information processing system 1 presents stimuli that remind the user that the experience is a virtual event, with the objective of maintaining a moderate state of immersion (objective (B)). For example, the information processing system 1 presents stimuli so as to maintain the level of immersion at 100% as much as possible and prevent the level of immersion from exceeding 100%.

[0187] Specifically, when the information processing system 1 detects multiple second- or third-stage interactions, it changes the visual characteristics of the avatar as a character to accurately mimic the shape and movements of a human while still creating a sense of incongruity, or it makes the avatar behave in a way that deviates from reality. By presenting such stimuli without stress (for example, without creating an eerie feeling), the information processing system 1 subconsciously reminds the user that the person with whom they are communicating is a virtual persona.

[0188] The information processing system 1 can also detect user interactions with an avatar, such as shaking hands (making contact) with the avatar, handing over an object to the avatar, or showing something to the avatar. In this case, the information processing system 1 presents a stimulus to reduce the level of immersion at the timing of these interactions. The timing of presenting the stimulus may be controlled by generating an event that causes an interaction based on the difference between the level of immersion and a target value. For example, when the level of immersion is 100% or higher, an event to hand over an object to the avatar is generated.

[0189] Since changing the visual characteristics of the avatar itself may increase stress for the user, it is preferable to change the visual characteristics of the virtual objects presented around the avatar. The visual characteristics of the virtual objects presented around the avatar can be changed at the timing of the premonition stage. For example, at least one of the position and depth of the virtual objects can be changed.

[0190] When an interaction at this stage is detected, it is possible to increase the frequency of presenting stimuli to reduce the level of immersion, but since presenting stimuli too frequently may prevent the user from achieving a sense of immersion, it is preferable to present the stimuli at a frequency that allows the user to achieve a moderate level of immersion.

[0191] In addition, the information processing system 1 can also select whether to present stimuli with the aim of preventing immersion from the start (goal (A)) or with the aim of maintaining a moderate state of immersion (goal (B)), depending on the user's level of belief in virtual reality, which is determined based on user information.

[0192] Next, in use case 3, the actors in the content are the user and others.

[0193] Use case 3 scenarios include communication with others in a virtual space, business, entertainment, and communication with video streamers in a virtual space. In such scenarios, there is a problem of confusing virtual reality with reality.

[0194] For example, in virtual spaces, avatars and effects are superimposed on other people, so even if the other person is actually exhausted, the effect makes it difficult for the user to accurately read the other person's facial expression, which could lead to an endless meeting. Also, by becoming accustomed to the use of cheerful character avatars and smiling effects, which are used to facilitate communication, users may feel a gap in the personality of the other person when they meet in real life, which could hinder real-life communication.

[0195] Therefore, the information processing system 1 aims to maintain a moderate level of immersion (purpose (B)) by presenting stimuli that remind the user that the expressions of others in the virtual space are merely staged. For example, the information processing system 1 presents stimuli to keep the level of immersion at 100% as much as possible and to prevent the level of immersion from exceeding 100%.

[0196] Specifically, the information processing system 1 does not present stimuli immediately after the start of communication, but begins presenting them after communication has stabilized. Immediately after communication begins, effects aimed at smooth communication with others are presented. To remind the user that the representation of others in the virtual space is merely intrusive, the information processing system 1 reduces the effects by partially removing the effects superimposed on the other person's body once multiple second- or third-stage interactions are detected. In addition, the information processing system 1 changes the position, depth, movement, shape, etc. of virtual objects presented around the other person's avatar.

[0197] The information processing system 1 changes the visual characteristics of virtual objects such as another person's avatar, an effect superimposed on another person, and a virtual object presented around the other person's avatar.

[0198] As the level of immersion increases and interaction at this stage is detected, it may be possible to present stimuli more frequently, but frequent presentation of stimuli such as removing effects may interfere with communication, so it is preferable to present stimuli at a frequency that provides a moderate sense of immersion.

[0199] The information processing system 1 gradually presents stimuli from the premonition stage onward, thereby imprinting a sense of discomfort on the unconscious and imprinting the idea that the interaction with another person is not a real event.

[0200] 4. Modifications In some cases, it may be more effective to gradually change multiple types of visual characteristics (e.g., size and shape) of a virtual object rather than making a large or sudden change to one type of visual characteristic (e.g., position, depth, movement, size, or shape).

[0201] A plurality of combinations are prepared so that the degree of change in visual characteristics and the frequency at which the visual characteristics are changed can be changed.

[0202] When a virtual object that is likely to attract attention is presented, interaction at the cognitive stage may be detected, or a high value may be estimated as the level of immersion in a scene where immersion is likely to be high. For example, interaction detection and immersion estimation may be performed based solely on the characteristics of the virtual object or the characteristics of each scene.

[0203] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0204] 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. A portion of the information processing system 1 is configured, for example, by a PC having a configuration similar to that shown in FIG.

[0205] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0206] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0207] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0208] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0209] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0210] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0211] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0212] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0213] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0214] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0215] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0216] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0217] (1) An information processing system including a presentation control unit that presents content including an object to a user and changes the object in accordance with a stage of the user's interaction with the object at a timing according to the user's level of immersion in the content. (2) The information processing system described in (1), wherein the presentation control unit changes multiple types of visual characteristics of the object. (3) The information processing system described in (1) or (2), wherein the presentation control unit changes visual characteristics including at least one of the position, depth, movement, size, and shape of the object. (4) The information processing system described in (3), wherein the presentation control unit changes the object at a timing corresponding to each stage of the interaction. (5) The information processing system described in (3) or (4), wherein the stages of the interaction include a recognition stage, a preparatory movement stage, a reaching movement stage, a last-minute preparatory movement stage, and a final movement stage. (6) The information processing system according to (5), wherein the presentation control unit changes at least one of a position and a depth of the object at the timing of the recognition stage, changes at least one of a movement, a size, and a shape of the object at the timing of the preparatory movement stage or the reaching movement stage, changes at least one of a shape, a size, and a contour of the object at the timing of the immediately preceding preparatory movement stage, and does not present a part of the object or changes a surface attribute of the object at the timing of the main movement stage. (7) The information processing system according to (5) or (6), wherein the presentation control unit changes the object at the timing of a precursor stage including the recognition stage and the preparatory movement stage when the degree of immersion is equal to or greater than a predetermined threshold. (8) The information processing system according to any one of (5) to (7), wherein the presentation control unit, when the level of immersion is lower than a predetermined threshold, changes the object at a timing of a premonition stage including the recognition stage and the preparatory movement stage, and at a timing of a main stage including the reaching movement stage, the last-minute preparatory movement stage, and the main movement stage.(9) The information processing system described in (8), wherein the presentation control unit changes the object at a timing before the interaction occurs when the immersion level is lower than a predetermined threshold. (10) The information processing system described in (1), wherein the presentation control unit changes the object at a timing based on a target value of the immersion level. (11) The information processing system described in (10), wherein the presentation control unit generates an event that causes the interaction based on a difference between the immersion level and the target value. (12) The information processing system described in (11), wherein the content provides the user with communication with a predetermined other party, and the presentation control unit changes the object according to a stage of the user's interaction with the other party. (13) The information processing system described in (12), wherein the presentation control unit changes at least one of an avatar of the other party of communication and an effect on the other party of communication as the object. (14) The information processing system according to (12), wherein the presentation control unit changes the objects presented around the communication partner, other than the avatar of the communication partner and the effect on the communication partner. (15) The information processing system according to any of (12) to (14), wherein the communication partner is another user. (16) The information processing system according to any of (12) to (14), wherein the communication partner is a virtual personality. (17) The information processing system according to (16), wherein at least one of a position and a depth of the objects presented around the communication partner can be changed. (18) The information processing system according to any of (12) to (17), wherein the interaction includes the user contacting the communication partner or the user handing over an object to the communication partner.(19) The information processing system according to any one of (1) to (17), wherein the presentation control unit generates the event in which the user hands over an object to the communication partner based on a difference between the degree of immersion and the target value. (20) An information processing method, wherein an information processing system presents content including an object to a user, and changes the object in accordance with a stage of the user's interaction with the object at a timing according to the degree of immersion of the user in the content. (21) The information processing system according to any one of (1) to (17), wherein the interaction includes the user grasping the object.

[0218] REFERENCE SIGNS LIST 1 Information processing system, 11 User sensor, 12 User state acquisition unit, 13 User DB, 14 Content sensor, 15 Content state acquisition unit, 16 Content DB, 17 Interaction detection unit, 18 Immersion degree estimation unit, 19 Presentation control unit, 20 Immersion reduction stimulus DB, 21 Presentation unit

Claims

1. Presenting content including an object to a user and changing the object in response to the user's immersion in the content and the user's interaction with the object. An information processing method including:

2. Changing multiple types of visual characteristics of the object. The information processing method according to claim 1 .

3. Changing visual characteristics of the object, including at least one of position, depth, movement, size, and shape. The information processing method according to claim 1 .

4. The object is changed at the timing of each stage of the interaction. The information processing method according to claim 3 .

5. The interaction stages include a recognition stage, a preparatory movement stage, a reaching movement stage, a preparatory movement stage, and a main movement stage. The information processing method according to claim 3 .

6. At the timing of the recognition stage, at least one of the position and depth of the object is changed; at the timing of the preparatory movement stage or the reaching movement stage, at least one of the movement, size, and shape of the object is changed; at the timing of the immediately preceding preparatory movement stage, at least one of the shape, size, and contour of the object is changed; and at the timing of the main movement stage, a part of the object is not presented or a surface attribute of the object is changed. The information processing method according to claim 5 .

7. When the degree of immersion is equal to or greater than a predetermined threshold, the object is changed at a timing of a premonition stage including the recognition stage and the preparatory movement stage. The information processing method according to claim 5 .

8. When the degree of immersion is lower than a predetermined threshold, the object is changed at a timing of a premonition stage including the recognition stage and the preparatory movement stage, and at a timing of a main stage including the reaching movement stage, the immediately preceding preparatory movement stage, and the main movement stage. The information processing method according to claim 5 .

9. When the degree of immersion is lower than a predetermined threshold, the object is changed at a timing before the interaction occurs. The information processing method according to claim 8.

10. The object is changed at a timing based on a target value of the immersion level. The information processing method according to claim 1 .

11. The method further includes generating an event that causes the interaction based on a difference between the degree of immersion and the target value. The information processing method according to claim 10.

12. the content is content that provides the user with communication with a predetermined person, Changing the object according to the stage of the user's interaction with the communication partner. The information processing method according to claim 11.

13. At least one of the avatar of the communication partner and the effect on the communication partner is changed as the object. The information processing method according to claim 12.

14. Changing the avatar of the communication partner and the objects presented around the communication partner other than the effect on the communication partner. The information processing method according to claim 12.

15. The communication partner is another user. The information processing method according to claim 12.

16. The communication partner is a virtual personality. The information processing method according to claim 12.

17. The object presented around the communication partner can be changed in at least one of its position and depth.

17. The information processing method according to claim 16.

18. The interaction includes the user contacting the other party of the communication or the user handing over an object to the other party of the communication. The information processing method according to claim 12.

19. The event in which the user hands over an object to the other party of the communication is generated based on a difference between the immersion level and the target value.

19. The information processing method according to claim 18.

20. A presentation control unit that presents content including an object to a user and changes the object according to the user's immersion in the content and the user's interaction with the object. An information processing system comprising:

21. Presenting content including an object to a user, and changing the object according to the user's immersion in the content and the user's interaction with the object. A program for causing a computer to execute a process including the above.