Application processing system, application processing method, and application processing program

Abstract

To provide an interaction with a virtual character to a user by outputting an image of a virtual space including the virtual character to a head-mounted display attached to the head of the user.SOLUTION: An application processing system specifies a first user action when it is detected that a first gazing position set in a virtual character is gazed, and specifies a second user action when it is detected that a second gazing position set in the virtual character is gazed. The first user action is the action of the first type and executed to a portion corresponding to the first gazing position. The second user action is the action of the second type different from the action of the first type and executed to a portion corresponding to the second gazing position different from the portion corresponding to the first gazing position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosure herein relates to an application processing system, an application processing method, and an application processing program that provide interaction with a virtual character. More specifically, the disclosure herein relates to an application processing system, an application processing method, and an application processing program that provide interaction with a virtual character according to detection information detected by a head mounted display. [Background technology]

[0002] As an application that provides interaction with a virtual character, a simulation game that allows interaction with the virtual character is known. Such a simulation game is disclosed in, for example, Japanese Patent Application Laid-Open No. 2012-055644 (Patent Document 1).

[0003] In games that provide interactions with such virtual characters, a head mounted display (hereinafter sometimes referred to as "HMD") is sometimes used to enhance the sense of immersion in the game. For example, a VR game is known in which a virtual space is displayed on an HMD and the player can use the HMD to interact with virtual characters appearing in this virtual space. One such VR game is "Summer Lesson" by Bandai Namco Entertainment Inc. (see Non-Patent Document 1).

[0004] As described in JP 2017-058969 A (Patent Document 2), the HMD is equipped with a tracking sensor for detecting the position and orientation of the user's head. In a VR game played using the HMD, a viewpoint in a virtual space is determined based on detection information detected by the tracking sensor, and an image of the virtual space seen from the viewpoint thus determined is displayed on the HMD.

[0005] In the above-mentioned "Summer Lesson," an action to be taken with respect to a virtual character is specified according to detection information detected by a tracking sensor in the HMD, and the virtual character is made to react to this specified action, thereby realizing interaction between the user and the virtual character. For example, when a user wearing an HMD nods in response to a question from a virtual character displayed on the HMD, the movement of the user's head corresponding to this nod is detected, and an action from the user to the virtual character in the virtual space is specified based on this detected head movement. For example, when the movement of the user's head corresponding to a nod is detected, an action of making an affirmative reply to the question from the virtual character is specified as an action from the user to the virtual character in the virtual space. When the HMD detects a movement of the user's head corresponding to a head shaking movement, an action to be taken in the virtual space, such as making a negative reply to a question from the virtual character, is specified as an action to be taken by the virtual character. The reaction of the virtual character to the user's action thus specified is determined, and an image of the virtual character making the determined reaction is displayed on the HMD.

[0006] The actions of the user in the virtual space that are identified based on the detection information of the HMD are limited to those associated with the user's head movements, such as nodding or shaking the head, and the user's gaze movements, such as gazing at a specific part of the virtual character. Thus, the variety of actions that the user can take on the virtual character that are identified by the detection signal of the HMD is limited.

[0007] If an action associated with a part of the user's body other than the head (e.g., a hand) can be specified as an action of the user in a virtual space, the variation of the interaction between the virtual character and the user will be enriched. Conventionally, an action associated with a part of the user's body other than the head has been specified based on an input from an operation controller separate from the HMD. A typical conventional operation controller is configured to be held by a user. For example, the specification of Japanese Patent No. 6189497 discloses that the position of the user's virtual hand is specified in a virtual space based on an operation on an operation controller held by the user, and an action of touching an object in the virtual space is performed with the virtual hand. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2012-055644 A [Patent Document 2] JP 2017-058969 A [Patent Document 3] Patent No. 6189497 specification [Non-patent literature]

[0009] [Non-Patent Document 1] "Summer Lesson", [online], searched on December 12, 2017, Internet (URL: HYPERLINK "http: / / summer-lesson.bn-ent.net / " http: / / summer-lesson.bn-ent.net / ) Summary of the Invention [Problem to be solved by the invention]

[0010] As described above, conventionally, a control controller separate from the HMD is used to identify actions of a user in a virtual space that are not associated with the movement of the user's head (for example, an action of touching with a hand in the virtual space). The use of such a control controller can impair the immersive feeling in the application. In addition, when wearing an HMD, the view in the real space is obstructed, making it difficult to operate the control controller.

[0011] There is a demand for an application processing system that allows users to perform actions on virtual characters in a virtual space that are not associated with head movements, without using an operation controller separate from the HMD.

[0012] It is an object of the present disclosure to provide technical improvements that solve or mitigate at least some of the problems in the prior art discussed above.

[0013] One specific objective of the present disclosure is to provide technical improvements in application processing systems that are capable of performing actions on virtual characters that are not associated with a user's head movements.

[0014] A more specific objective of the present disclosure is to provide an application processing system capable of performing actions on a virtual character that are not associated with the movement of the user's head, based on information detected by a head-mounted display.

[0015] One of the more specific objectives of the present disclosure is to increase the variety of actions that can be performed on a virtual character that are not associated with the user's head movement.

[0016] These and other objects of the present disclosure will become apparent upon reference to the entire specification. [Means for solving the problem]

[0017] According to one aspect of the present invention, an application processing system includes one or more computer processors, and provides a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head-mounted display worn on the user's head. In one aspect, the one or more computer processors execute computer-readable instructions to identify a first action to be performed on the virtual character and associated with a movement of the user's head in the virtual space based on one or more pieces of detection information detected by the head-mounted display, and to perform an action on the virtual character based on the one or more pieces of detection information in response to a selection of a specific object included in the virtual space. A second action is identified that is performed by the user and is not associated with a movement of the user's head in the virtual space.

[0018] In an application processing system according to one aspect of the present invention, the second action is an action associated with a hand movement of the user in the virtual space.

[0019] In an application processing system according to one aspect of the present invention, the second action is an action associated with the user's use of a tool in the virtual space.

[0020] In an application processing system according to one embodiment of the present invention, one or more character gaze positions are set on the virtual character in response to selection of the specific object, and the second action is identified in response to gaze being placed on at least one of the one or more character gaze positions set on the virtual character.

[0021] In an application processing system according to one embodiment of the present invention, the one or more character gaze positions include a first character gaze position and a second character gaze position, the second action includes a first gaze position action and a second gaze position action, and by gazing at the first character gaze position, a first gaze position action is identified as the second action, and by gazing at the second character gaze position, a second gaze position action is identified as the second action.

[0022] In an application processing system according to one embodiment of the present invention, the specific object is associated with a first menu item and a second menu item, and the second action includes a first menu item action associated with the first menu item and a second menu item action associated with the second menu item, and the first menu item action is identified as the second action by gazing at at least one of the one or more character gaze positions after gazing at the first menu item, and the second menu item action is identified as the second action by gazing at at least one of the one or more character gaze positions after gazing at the second menu item.

[0023] In an application processing system according to one aspect of the present invention, the one or more computer processors identify a first main display area in the virtual space in a first scene, and the specific object is positioned outside the first main display area in the first scene.

[0024] In an application processing system according to one aspect of the present invention, the one or more computer processors identify a second main display area in the virtual space in a second scene, and the identified object is positioned outside the second main display area in the second scene.

[0025] In an application processing system according to one aspect of the present invention, the one or more computer processors configure the virtual space so that the virtual space includes the virtual character that reacts in response to the first action or the second action.

[0026] According to one aspect, an application processing method provides a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head-mounted display worn on a user's head by executing computer-readable instructions by one or more computer processors. The application processing method includes: identifying a first action to be performed on the virtual character and associated with a movement of the user's head in the virtual space based on one or more pieces of detection information detected by the head-mounted display; and identifying a second action to be performed on the virtual character and not associated with a movement of the user's head in the virtual space based on the one or more pieces of detection information in response to a selection of a specific object included in the virtual space.

[0027] An application processing program according to one aspect of the present invention, when executed by one or more computer processors, provides a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head-mounted display worn on the user's head. The application processing program causes the one or more computer processors to execute the following steps: specifying a first action to be performed on the virtual character and associated with a movement of the user's head in the virtual space, based on one or more pieces of detection information detected by the head-mounted display; and specifying a second action to be performed on the virtual character and not associated with a movement of the user's head in the virtual space, based on the one or more pieces of detection information in response to a selection of a specific object included in the virtual space. Effect of the Invention

[0028] SUMMARY OF THE DISCLOSURE In accordance with embodiments disclosed herein, technical improvements are provided in application processing systems that allow actions to be performed on virtual characters that are not associated with a user's head movements. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a block diagram illustrating an application processing system according to one embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a head mounted display that can be used in the application processing system of FIG. 1. [Diagram 3] FIG. 1 is a diagram illustrating a schematic diagram of a head mounted display worn by a user. [Figure 4a] 4A and 4B are schematic diagrams illustrating a virtual space in an application processing system according to an embodiment, in which a user is in a standing position in the virtual space; [Figure 4b]4a and 4b are schematic diagrams illustrating a virtual space in an application processing system according to an embodiment, in which a user is in a supine position in the virtual space; [Diagram 5] FIG. 2 is a diagram illustrating mode transitions of a game processed by an application processing system according to one embodiment. [Figure 6] FIG. 11 is a flow diagram showing a process flow in a chat mode in one embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a display image in a chat mode according to an embodiment. [Figure 8] FIG. 11 is a flowchart showing a process flow in a VR mode in one embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of a display image in a VR mode according to an embodiment. [Figure 10] FIG. 11 is a flowchart showing a process for identifying a second action in one embodiment. [Figure 11a] 11A and 11B are diagrams illustrating examples of display images in VR mode according to one embodiment.The display image in Fig. 11A includes an image showing a menu item icon. [Figure 11b] 11A is a diagram showing an example of a display image in VR mode according to an embodiment, in which a plurality of character gaze positions are set for the virtual character in FIG. [Figure 11c] 11C is a diagram showing an example of a display image in a VR mode according to an embodiment. The display image in FIG. 11C includes an image showing a second action of the user in the virtual space. [Figure 11d] 11d is a diagram illustrating an example of a display image in a VR mode according to an embodiment. The display image in FIG. 11d includes an image showing a second action of the user in the virtual space. [Figure 12] FIG. 2 is a block diagram illustrating an application processing system according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Various embodiments of the present invention will now be described with reference to the accompanying drawings, in which the same reference numerals are used to designate the same or similar components throughout the drawings.

[0031] An application processing system according to an embodiment will be described with reference to Fig. 1 to Fig. 3. Fig. 1 is a block diagram showing an application processing system 1 according to an embodiment, Fig. 2 is a schematic diagram of a head mounted display 10 (hereinafter referred to as "HMD 10") used in the application processing system 1, and Fig. 3 is a schematic diagram showing the HMD 10 worn by a user 5.

[0032] The application processing system 1 according to one embodiment provides various applications by executing an application processing program. An example of this application is a video game. A user can use the application processing system 1 to play various games that use game media. The game media is electronic data used in a game realized by the application processing system 1, and can be acquired, owned, used, managed, exchanged, synthesized, enhanced, sold, discarded, donated, or used in other ways by a user in the game. Game media includes, for example, cards, items, virtual currency, tickets, characters, avatars, level information, status information, and parameter information (such as stamina value and attack power), ability information skills, abilities, spells, jobs, and other game information. The game media available in the application processing system 1 and the manner of use thereof are not limited to those specified in this specification. The application processing system 1 can realize, for example, a game in which a virtual character and a user interact in a virtual space. The game realized by the application processing system 1 has a first mode and a second mode. An example of the first mode is a chat mode, and an example of the second mode is a VR mode. The first mode and the second mode will be described later.

[0033] The application processing system 1 may be configured to process applications other than games. For example, the application processing system 1 may be configured to provide a simulation in driving or other fields, a training application in medical or other fields, an application for experiencing products or services, an application for watching movies or other various entertainment, and various other applications.

[0034] The application processing system 1 includes an HMD 10 and a server 50. The HMD 10 and the server 50 are connected via a network 40 so as to be able to communicate with each other.

[0035] As shown in FIG. 2, the HMD 10 includes a wearable device 11 that is worn on a person's head, and an information processing device 20 that is attached to the wearable device 11.

[0036] The wearing tool 11 has a goggle-shaped housing 11a with an opening 11d formed therein, a first belt 11b and a second belt 11c attached to the housing 11a, and a lid 11e. The lid 11e is attached to the housing 11a so as to be able to be opened and closed. The opening 11d is formed in the front of the housing 11a. In FIG. 2, the lid 11e is open. When the HME 10 is in use, the lid 11e is closed to close the opening 11d.

[0037] The information processing device 20 is detachably provided inside the lid 11e of the HMD 10. The information processing device 20 includes a display 24. The information processing device 20 is attached to the lid 11e such that the display 24 faces the inside of the housing 11a when the lid 11e is closed. The information processing device 20 is attached to the wearing device 11 when the HMD 10 is used. The information processing device 20 is detached from the wearing device 11 when the HMD 10 is not used.

[0038] In the illustrated embodiment, the information processing device 20 is a smartphone. The information processing device 20 may be a mobile phone, a tablet terminal, a personal computer, an e-book reader, a wearable computer, a game console, or any other information processing device other than a smartphone. The information processing device 20 removed from the HMD 10 is used according to its intended purpose.

[0039] As described above, the information processing device 20 includes the display 24. In the illustrated embodiment, when the information processing device 20 is attached to the wearing tool 11, the display 24 functions as a device for displaying images in the HMD 10. Therefore, when a game is played using the HMD 10, the virtual space and virtual characters of the game, as well as other images related to the game, are displayed on the display 24.

[0040] The shape of the wearing equipment 11 is not limited to the goggle type shown in the figure. The wearing equipment 11 may include a structure of any shape that moves following the movement of the head of the user wearing it and can place the display 24 in front of the eyes of the user when worn. The wearing equipment 11 may be shaped like glasses, a hat, or a helmet, for example. In order to improve the user's sense of immersion, it is desirable that the HMD 10 is configured so that the display 24 covers both eyes of the user when the wearing equipment 11 is worn on the user's head.

[0041] 3, the HMD 10 is attached to the head of a user 5 via a wearing device 11. An information processing device 20 is attached to the wearing device 11 that is worn on the head of the user 5.

[0042] 1, the information processing device 20 will be further described. As shown in the figure, in one embodiment, the information processing device 20 includes a computer processor 21, a memory 22, a communication I / F 23, a display 24, a sensor unit 25, a sound collecting device 26, and a storage 27.

[0043] The computer processor 21 is an arithmetic device that loads various programs that realize an operating system and game logic from the storage 27 or other storage into the main memory 22 and executes instructions included in the loaded programs. The computer processor 21 is, for example, a CPU, an MPU, a DSP, a GPU, or various other arithmetic devices. The processor 21 may be realized by an integrated circuit such as an ASIC, a PLD, an FPGA, or an MCU. Although the computer processor 21 is illustrated as a single component in FIG. 1, the computer processor 21 may be a collection of multiple physically separate computer processors. In this specification, a program described as being executed by the computer processor 21 or instructions included in the program may be executed by a single computer processor or may be executed in a distributed manner by multiple computer processors. In addition, a program executed by the computer processor 21 or instructions included in the program may be executed in a distributed manner by multiple virtual computer processors.

[0044] The memory 22 is used to store instructions executed by the computer processor 21 and various other data. At least a part of the application processing program according to one embodiment is loaded into the memory 22 as the game progresses. The memory 22 is, for example, a main memory that can be accessed at high speed by the computer processor 21. The memory 22 is, for example, a RAM such as a DRAM or an SRAM.

[0045] The communication I / F 23 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these. The information processing device 20 can transmit and receive data to and from other devices via the communication I / F 23.

[0046] The display 24 has a display panel 24a and a touch panel 24b. For example, the touch panel 24b is laminated on the upper or lower surface of the display panel 24a. The display panel 24a is a liquid crystal panel, an organic EL panel, an inorganic EL panel, or any other display panel capable of displaying images. The touch panel 24b is configured to be able to detect a touch operation by the user. The touch panel 24b can detect various touch operations by the user, such as tapping, double tapping, and dragging.

[0047] The sensor unit 25 includes one or more sensors. The sensor unit 25 includes at least one of a gyro sensor, an acceleration sensor, and a geomagnetic sensor. The sensor unit 25 may include an eye tracking sensor that directly detects the movement of the user's eyeball. The eye tracking sensor is, for example, a gaze detection sensor that irradiates near-infrared light into the iris and detects the reflected light. Based on the detection information detected by the sensor unit 25, the position and orientation of the HMD 10 worn on the head of the user 5 are specified. The position and orientation of the HMD 10 are calculated, for example, as a position in a three-dimensional orthogonal coordinate system and angles around each axis. This three-dimensional orthogonal coordinate system is, for example, an orthogonal coordinate system consisting of a roll axis along the front-rear direction, a yaw axis along the up-down direction, and a pitch axis along the left-right direction. At least a part of the various sensors that constitute the sensor unit 25 may be provided in a member different from the wearing equipment 11 and / or the information processing device 20 other than the wearing equipment 11.

[0048] The sound collector 26 is configured to collect sound. The sound collector 26 is, for example, a microphone. Based on the sound information collected by the sound collector 26, the voice of the user 5 is detected.

[0049] The storage 27 is an external storage device accessed by the computer processor 21. The storage 27 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or any other storage device capable of storing data. Various programs such as application processing programs are stored in the storage 27. In addition, various data used in the game is also stored in the storage 27. At least a part of the programs and various data that can be stored in the storage 27 may be stored in a storage that is physically separate from the information processing device 20.

[0050] In the illustrated embodiment, storage 27 stores image data 28a, chat data 28b, scenario data 28c, event data 28d, game progress data 28e, and various other data necessary for the progress of the game.

[0051] The image data 28a includes data for drawing the background of the virtual space in which the game is played, data for drawing virtual characters, and data for drawing objects other than the virtual characters used in the game. The image data 28a may also include position information of objects in the virtual space. The image data 28a may store each of the above data in association with each scenario, event, or scene.

[0052] Chat data 28b includes data for drawing an icon of a virtual character, data for drawing an icon of user 5, data for identifying a plurality of messages of a virtual character, data indicating options of reply messages to each of the plurality of messages of the virtual character, and other data used in chat modes. The plurality of messages of a virtual character may have a node corresponding to each message, and may be defined in a tree structure in which the nodes are connected to each other by arcs. In this tree structure, for example, a plurality of arcs extend from a start message which is the top root node, and each arc is connected to a lower node. Arcs extend from these lower nodes, and the arcs are further connected to lower nodes. Each of the lower nodes may be displayed after the start message. The chat data 28b may include a mode transition condition that is a condition for starting a transition from the chat mode to the VR mode. The mode transition condition may include, for example, a condition that a predetermined time or more has elapsed since the game was started in the chat mode, that the chat has progressed to the end node in the message tree structure, and other conditions. The chat data 28b may include data indicating a message to be displayed in the chat mode resumed after the selection of the transition object has not been completed.

[0053] The scenario data 28c includes data that defines a scenario experienced by the user 5 in the game. When there are multiple scenarios experienced by the user 5 in the game, the scenario data 28c may define scenario data for each of the multiple scenarios. The scenario data 28c may be defined as a collection of multiple events. The multiple events included in the scenario data 28c may have a node corresponding to each event, and may be defined in a tree structure in which the nodes are connected to each other by arcs. In this tree structure, for example, multiple arcs extend from an opening event that is the top root node, and each arc is connected to a lower node. Arcs also extend from these lower nodes, and the arcs are connected to further lower nodes. Each of the lower nodes in a certain scenario is a set of events that the user 5 experiences. The candidate events correspond to candidates for events that may occur after the opening event in the scenario. The scenario data 28c may store the above data in association with each scenario.

[0054] The event data 28d is data for defining each of the events constituting the scenario data 28c. The event data 28d may be defined as a collection of data for specifying a plurality of actions of a virtual character in a game. The plurality of actions of a virtual character may be defined in a tree structure having a node corresponding to each action, and the nodes are connected to each other by arcs. In this tree structure, for example, a plurality of arcs extend from the top root node, which is the opening action, and each arc is connected to a lower node. Arcs also extend from these lower nodes, and the arcs are connected to further lower nodes. Each of the lower nodes in a certain event corresponds to a candidate action that may occur after the opening action in the event. The event data 28d is a node. Each node may include video data corresponding to the action corresponding to that node, data regarding actions the user can take with respect to the virtual character, and data regarding other actions of the virtual character.

[0055] The game progress data 28e includes data used to manage the progress of the game. The game progress data 28e may be updated according to the progress of the game. The game progress data 28e may include, for example, data on the likeability of the virtual character to the user, data on points acquired by the user in the game, and various other data that may change according to the progress of the game.

[0056] 1 are merely illustrative. The information processing device 20 applicable to the present invention may include various components not shown. For example, the information processing device 20 may include a speaker that outputs game sound effects and the voice of a virtual character.

[0057] 1 are merely illustrative. The information processing device 20 applicable to the present invention may include various components not shown. For example, the information processing device 20 may include a speaker that outputs game sound effects and the voice of a virtual character.

[0058] Next, a description will be given of the functions of the HMD 10. In the illustrated embodiment, various functions of the HMD 10 are realized by executing computer-readable instructions in a computer processor 21 of an information processing device 20. The instructions executed in the computer processor 21 include instructions included in an application processing program according to one embodiment.

[0059] An application processing program according to one embodiment is executed by the computer processor 21, thereby realizing a game having a first mode and a second mode different from the first mode in the application processing system 1. The game realized in the application processing system 1 may have a mode other than the first mode and the second mode.

[0060] In a first mode of a game realized in the application processing system 1, when the HMD 10 is not worn by the user 5, that is, when the information processing device 20 is detached from the wearing equipment 11, a process related to the game is performed according to first detection information detected by the information processing device 20. This first detection information may include information related to a touch operation of the user 5 detected by the touch panel 24b of the information processing device 20, information related to the user's voice detected by the sound collection device 26, and other detection information detectable by the information processing device 20 when the HMD 10 is not worn by the user 5. In the first mode, the user 5 can perform an operation related to the game using the information processing device 20 detached from the wearing equipment 11. In the first mode, it is assumed that the game will progress when the HMD 10 is not worn by the user 5, and therefore, in one embodiment, an image other than a stereoscopic image is displayed on the display 24.

[0061] In the second mode of the game realized in the application processing system 1, the game progresses using second detection information detected by the HMD 10 worn on the head of the user 5. This second detection information is, for example, detection information detected by the sensor unit 25. Based on this second detection information, head tracking information for identifying the position and orientation of the head of the user 5 is calculated. The process for progressing the game in the second mode is executed, for example, based on the head tracking information calculated based on the second detection information. The process for progressing the game in the second mode may be executed using information other than the head tracking information in addition to the head tracking information. In preparation for playing the game in the second mode, the user 5 attaches the information processing device 20 to the wearing equipment 11 and wears the wearing equipment 11 to which the information processing device 20 is attached on the head. In this way, in the second mode, it is assumed that the game progresses while the HMD 10 is worn by the user 5, so in one embodiment, in the second mode, a stereoscopic image stereoscopically viewed by the user 5 is displayed on the display 24. The stereoscopic image is displayed on the display 24 by, for example, a parallax barrier method. In the parallax barrier method, a left-eye image and a right-eye image are displayed on the display 24. In this way, a stereoscopic image is a set of images including a left-eye image and a right-eye image configured to be displayed on the display to be viewed stereoscopically using the parallax between the left and right eyes.

[0062] The first mode is, for example, a chat mode. The chat mode is an example of the first mode. In the chat mode, a function is realized in which the user chats with a virtual character via text messages. In the first mode, the user can experience an interaction with the virtual character by chatting with the virtual character. The specific content of the processing in the chat mode will be described later. In a broad sense, the interaction means that the virtual character reacts to an action performed by the player. The interaction with the virtual character includes an interaction performed as communication through a conversation with the virtual character. In this specification, an interaction performed as communication with the virtual character may be called a communicative interaction. In addition to the communicative interaction, the interaction with the virtual character may include a battle with the virtual character, a cooperative play in which the user cooperates with the virtual character to play a game, and other interactions with the virtual character. In this specification, an interaction performed as a battle with the virtual character may be called a battle interaction. In this specification, an interaction carried out as cooperative play with a virtual character may be referred to as a cooperative interaction.

[0063] The second mode is, for example, a VR mode. In the VR mode, a function for performing various interactions with a virtual character appearing in a virtual space displayed on the display of the information processing device 20 is realized. Specific contents of the processing in the VR mode will be described later. The VR mode is an example of the second mode, and the second mode may include any game mode in which processing for game progress is performed using head tracking information.

[0064] In one embodiment, a game having a first mode and a second mode realized by the application processing system 1 may be a game in which interactions other than communicative interaction with a virtual character are performed. A game realized by the application processing system 1 does not have to involve communicative interaction. In one embodiment, a game is played in a two-dimensional space in the first mode, and in a three-dimensional space in the second mode. In one embodiment, a game is played in a three-dimensional space in the first mode, and in a three-dimensional space in the second mode that is displayed in a different display manner from the three-dimensional space in the first mode (or configured in a different manner from the three-dimensional space in the first mode). In one embodiment, a game realized by the application processing system 1 is The first mode and the second mode may use a common game medium. The parameters associated with this game medium may be carried over between the first mode and the second mode. For example, if the parameters of a specific game medium are changed during play in the first mode and then the mode is switched to the second mode, the specific game medium with the changed parameters set is used in the second mode. In one embodiment, the parameters of the specific game medium may be changed at least one of when the mode is switched from the first mode to the second mode and when the mode is switched from the second mode to the first mode. The change in the parameters of this game medium may be an advantageous change or a disadvantageous change for the player 5 in the progress of the game. In one embodiment, the results of play in the first mode are reflected in the game in the second mode. Alternatively, the results of play in the second mode may be reflected in the game in the first mode. For example, the experience points acquired by the player 5 in the first mode may be carried over to the second mode.

[0065] As described above, the first mode and the second mode of the game realized in the application processing system 1 are modes that are distinguished from each other. That is, the first mode is a mode different from the second mode. In one embodiment, when the game realized in the application processing system 1 is played in the first mode, the HMD 10 is not worn on the head of the user 5, whereas when the game is played in the second mode, the HMD 10 is worn on the head of the user 5. In this case, in the first mode, the game is processed according to the first detection information detected by the information processing device 20 that is not worn on the head of the user 5, and in the second mode, the game is processed according to the second detection information detected by the HMD 10 worn on the head of the user 5.

[0066] In one embodiment, in the second mode, a stereoscopic image is displayed on the display 24, whereas, as described above, in the first mode, an image other than a stereoscopic image is displayed on the display 24. In the second mode, at least an image of a virtual character among the images for the game is displayed as a stereoscopic image, whereas in the first mode, an image of a virtual character is displayed as an image other than a stereoscopic image.

[0067] In one embodiment, in the first mode, a process is performed to progress the game without using head tracking information calculated based on the second detection information, whereas in the second mode, a process is performed to progress the game based on the head tracking information.

[0068] In one embodiment, in the first mode, a process for progressing the game is performed in response to a touch operation detected by the touch panel 24b, whereas in the second mode, no process is performed in response to a touch operation on the touch panel 24b.

[0069] In one embodiment, when an interaction with a virtual character is provided in a game realized in the application processing system 1, in a first mode, an interaction with the virtual character is provided according to first detection information detected by an information processing device 20 that is not worn on the head of the user 5, and in a second mode, an interaction with the virtual character is provided according to second detection information detected by an HMD 10 that is worn on the head of the user 5.

[0070] After the game is started, it is possible to switch between the first mode and the second mode, which are distinguished as described above. With reference to FIG. 5, an overview of the mode transition of the game processed by the application processing system 1 will be described. As shown in the figure, when the game is started in step S1, the chat mode S2, which is the first mode, is started in step S2. When a transition process for transitioning to the VR mode, which is the second mode, is started in this chat mode, the process proceeds to step S3, where a transition process to the second mode is performed. When the transition process is completed, the VR mode, which is the second mode, is started in step S4. When the VR mode is ended or interrupted, a return process to the chat mode is performed.

[0071] Next, the functions realized by the computer processor 21 will be described in more detail. The computer processor 21 functions as a chat mode execution unit 21a, a transition processing unit 21b, and a VR mode execution unit 21c by executing computer-readable instructions. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the game system 1. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor mounted on the server 50, for example.

[0072] The chat mode execution unit 21a executes an application processing program according to an embodiment, thereby performing processing for providing a game in a chat mode, which is a first mode. The chat mode execution unit 21a realizes a function for chatting between the user 5 and a virtual character. After logging in to the chat mode, the chat mode execution unit 21a displays a message of the virtual character and a message input or selected by the user 5 on the display 24, and allows the virtual character and the user 5 to chat. The chat mode execution unit 21a may display a plurality of options regarding the response of the user 5 to the message on the display 24, following the message of the virtual character. A message of the user 5 is specified from the plurality of options in response to the operation of the user 5, and the specified message is displayed as the message of the user 5, following the message of the virtual character. The user 5 can select a desired one of the displayed options by performing a touch operation on the touch panel 24b. The message of the virtual character and the user's option for the message can be specified by referring to the chat data 28b according to the application processing program. The message of the virtual character may be displayed together with an image of the virtual character, and the message of the user 5 may be displayed together with an image of the user 5. The message of the virtual character may include a message encouraging the user 5 to transition to the second mode, which is the VR mode, a message for causing the user 5 to select settings in the VR mode, and other messages associated with the VR mode.

[0073] In the chat mode, a transition start object for starting a transition to the second mode, VR mode, is displayed according to the progress of the chat. The transition start object is displayed when a mode transition condition is satisfied according to the application processing program. When it is detected that the transition start object has been selected based on a detection signal from the touch panel 24b or another user interface, a transition process to the second mode, VR mode, is started.

[0074] The transition processing unit 21b performs a transition process for transitioning from the first mode to the second mode. This transition process may include displaying on the display 24 a guidance for encouraging the user to wear the information processing device 20 on the wearing equipment 11, and displaying on the display 24 a transition object selectable by the user's gaze. After displaying this transition object, the transition processing unit 21b receives a detection signal from the sensor unit 25 or another sensor, and can determine whether or not the transition object has been selected by the user's gaze based on this detection signal. For this determination, the transition processing unit 21b calculates the position and orientation of the HMD 10 based on the detection signal from the sensor unit 25, and identifies the gaze position of the user 5 based on the calculated position and orientation of the HMD 10. Identification of the gaze position can be performed using various known methods. The gaze position may be determined by a method such as the above. For example, the gaze position may be determined based on a detection signal from an eye tracking sensor. The transition processing unit 21b measures, for example, the time (gaze time) during which the gaze position is on the transition start object, and when this gaze time reaches a predetermined time, determines that the selection of the transition start object is complete. When the transition processing unit 21b determines that the selection of the transition object is complete, it starts the VR mode, which is the second mode.

[0075] The VR mode execution unit 21c configures a virtual space including virtual characters and other objects, and outputs image information for displaying this virtual space to the display 24. The VR mode execution unit 21c can generate image information of the virtual space 60 so as to correspond to a scene determined according to the progress of the game.

[0076] The virtual space configured by the VR mode execution unit 21c will be described with further reference to FIG. 4a. FIG. 4a is a diagram that shows a schematic representation of a virtual space 60 configured by the VR mode execution unit 21c. FIG. 4a shows the virtual space 60 at the time of initial setting in the first scene. The VR mode execution unit 21c specifies the position of the user 61 and the reference line of sight 62 in the virtual space 60 based on the initial setting of the first scene, and specifies an area in a predetermined range centered on the reference line of sight 62 in the entire virtual space 60 as an initial display area 63. This initial display area 63 is, for example, a range within a predetermined angle of view centered on the reference line of sight 62. The initial display area 63 is partitioned by a non-display area and a boundary line 63a. When the processing of the first scene is started, the VR mode execution unit 21c generates image information showing an image of the virtual space in the initial display area 63 of the entire virtual space 60, and outputs the generated image information to the display 24. Based on the image information generated in this way, an image of the virtual space 60 corresponding to the initial display area 63 is displayed on the display 24. To generate the image information, for example, image data 28a, scenario data 28c, event data 28d, game progress data 28e, and other data stored in the storage 27 may be used.

[0077] When the HMD 10 detects the head movement of the user 5 after starting the processing of the first scene in the VR mode, the VR mode execution unit 21c moves the reference line of sight 62 so as to follow the detected head movement of the user 5, and specifies a range within a predetermined angle of view centered on the reference line of sight 62 after the movement as a new display area. In this way, the VR mode execution unit 21c updates the display area at any time according to the head movement of the user 5 by the HMD 10, and outputs an image of the virtual space 60 corresponding to the updated display area to the display 24. As a result, the image of the virtual space 60 displayed on the display 24 also changes according to the display area. In this way, the user 5 can see the image of the virtual space 60 that changes according to the movement of his / her head. When the user 61 moves in the virtual space 60, the direction of the reference line of sight 62 changes according to the position of the user 61 in the virtual space 60.

[0078] A main display area 64 is provided in the virtual space 60. The main display area 64 is an example of a first main display area. This main display area 64 is provided at a predetermined position in the virtual space 60 that is determined for each scene. The main display area 64 is divided from other areas by a boundary line 64a. For example, the main display area 64 is an area including a position to which the gaze of the user 5 is likely to be directed in each scene. The main display area 64 is arranged, for example, inside an initial display area 63 that is determined based on the initial settings of each scene. The main display area 64 in a certain scene is set so that at least a part of an object representing a virtual character that will perform an interaction is included in the initial display area 63 that is determined based on the initial settings of the scene. The main display area 64 may be an area that forms a part of the initial display area 63, or may be an area that coincides with the initial display area 63. In the first scene shown in FIG. 4a, the main display area 64 is provided around the reference line of sight 62 of the user 61 in a standing posture.

[0079] Virtual character objects and various other objects are placed in the virtual space 60. The objects included in the virtual space 60 may include a specific object that can be selected by the gaze of the user 5. In Fig. 4a, a menu object 66 is shown as an example of such a specific object.

[0080] The menu object 66 may be associated with one or more menu items that can be selected by the gaze of the user 5. Each of the one or more menu items may be associated with an action of the user 61 in the virtual space 60. The menu items included in the menu object 66 may be associated with an action that is not associated with the movement of the head of the user 61 in the virtual space 60, for example, a hand action associated with the movement of the hand of the user 61 in the virtual space 60, or a tool action related to the use of a tool by the user 61 in the virtual space 60. The tool action is, for example, a photo action in which the user 61 takes a photo using a camera in the virtual space 60. When multiple menu items are associated with the menu object 66, the game Depending on the progress of the program, only some of them may be selectable.

[0081] 4a, the menu object 66 is placed at a position in the virtual space 60 that is not included in the main display area 64. In other words, the menu object 66 is placed outside the main display area 64. The menu object 66 may be placed inside or outside the initial display area 63.

[0082] FIG. 4b shows the virtual space 60 in the second scene. The second scene is a game scene different from the first scene. The second scene is executed before the first scene or after the second scene. As shown in the figure, in the second scene, a main display area 64 is provided around a reference line of sight 62 of a user 61 in a supine position. In this second scene, a menu object 66 is also arranged outside the main display area 64.

[0083] The VR mode execution unit 21c displays an image of a virtual character in a virtual space in accordance with an application processing program, and realizes an interaction between this virtual character and the user 5. The VR mode execution unit 21c determines the action of the virtual character in the virtual space in accordance with the application processing program, and generates image information and audio information corresponding to the action of this virtual character. The image information is output to the display 24, and the audio information is output to the speaker of the HMD 10. As a result, an image of the virtual character moving in the virtual space is displayed on the display 24, and audio corresponding to the movement of the virtual character is output from the speaker. For example, when the virtual character speaks to the user 5, a moving image of the virtual character's head and mouth moving is displayed on the display 24, and the virtual character's voice is output. A sound corresponding to what the character says is output from the speaker. For example, image data 28a, scenario data 28c, event data 28d, game progress data 28e, and other data stored in the storage 27 may be used to generate image information and sound information corresponding to the actions of the virtual character. In one embodiment, the virtual character that interacts with the user 5 in each of the VR mode and chat mode is the same virtual character. For example, a virtual character that appears in the VR mode and a character that appears in the chat mode have the same name and have a commonality in appearance to the extent that they can be recognized as the same character. However, while an image of a virtual character is displayed as a stereoscopic image in the VR mode, an image of a virtual character is displayed as an image other than a stereoscopic image in the chat mode. In the respects shown, the images of the virtual characters in both modes differ from each other, but such differences in presentation do not preclude the identity of the virtual characters.

[0084] In addition to the above functions, the VR mode execution unit 21c has a function of identifying an action of the user 61 in the virtual space 60 based on a detection signal from the sensor unit 25. The action of the user 61 in the virtual space 60 is, for example, an action from the user 61 to a virtual character appearing in the virtual space 60. Since the action of the user 61 in the virtual space 60 is virtual within the virtual space 60, it may or may not match the body movement of the user 5 in the real space or the movement of the user 5 in the real space.

[0085] The actions of the user 61 in the virtual space 60 include a first action associated with the movement of the head of the user 61 and a second action not associated with the movement of the head of the user 61. The first actions associated with the movement of the head of the user 61 in the virtual space 60 include an action of the user 61 nodding in the virtual space 60, an action of the user 61 shaking his head in the virtual space 60, and other actions associated with the movement of the head of the user 61 in the virtual space 60. The VR mode execution unit 21c can specify the movement of the head of the user 5 in the real space by calculating the position and orientation of the HMD 10 based on detection information detected by the sensor unit 25 provided in the HMD 10 worn on the head of the user 5.

[0086] The VR mode execution unit 21c detects a head movement of the user 5 in the real world based on one or more pieces of detection information detected by the HMD 10, and specifies an action that matches or corresponds to the detected head movement of the user 5 in the real world as a first action of the user 61 in the virtual space 60. For example, when it is detected that the user 5 performs a nodding action, the action of the user 61 nodding (in the virtual space 60) is specified as the first action of the user 61 in the virtual space 60. When the virtual space 60 is displayed on the display 24 as an image from a first-person perspective, the first action of the user 61 in the virtual space 60 is not included in the image of the virtual space 60.

[0087] The second action of the user 61 in the virtual space 60 is not associated with the movement of the user 61's head in the virtual space 60. This second action is associated with the movement of a part of the user 61 other than the head in the virtual space 60, for example, the movement of the user 61's hand. The second action includes an action of the user 61 "stroking" in the virtual space 60, an action of the user 61 "pecking" in the virtual space 60, and other actions associated with the movement of a part of the user 61 other than the head in the virtual space 60. The action of "stroking" is an example of a first gaze position action, and the action of "pecking" is an example of a second gaze position action.

[0088] In one embodiment, the VR mode execution unit 21c is configured to specify a second action in response to the selection of the menu object 66. The VR mode execution unit 21c can detect that the menu object 66 has been selected based on a gaze position calculated based on a detection signal of the HMD 10. For example, the VR mode execution unit 21c can determine whether or not the selection of the menu object 66 has been completed depending on whether or not the gaze position is on the menu object 66 for a predetermined time or more. For example, if the gaze position is on the menu object 66 for a predetermined time or more, it is determined that the selection of the menu object 66 has been completed.

[0089] When the menu object 66 includes multiple menu items, the VR mode execution unit 21c can detect that one of the multiple menu items has been selected when the gaze position is on that menu item for a predetermined period of time or longer.

[0090] When the VR mode execution unit 21c detects the selection of the menu object 66 or the selection of one of the menu items included in the menu object 66, the VR mode execution unit 21c sets the selected menu object 66 or the action corresponding to the selected menu item as a selected action to be performed by the user 61 on the virtual character. As described above, the menu items may include a menu item corresponding to a hand action related to the movement of the hand of the user 61 in the virtual space 60, and a menu item corresponding to a photo action related to taking a photo in the virtual space 60. When a menu item associated with this hand action is selected, the VR mode execution unit 21c sets this hand action as a selected action for the virtual character.

[0091] When the second action is set, the VR mode execution unit 21c may set one or more character gaze positions on the virtual character. When at least one gaze position of the one or more gaze positions is selected, the VR mode execution unit 21c can specify an action for the virtual character based on the selected gaze position and the selected action. For example, when a menu item corresponding to a hand action is selected, if a gaze position set on the head of the virtual character is selected, an action of stroking the head of the virtual character is specified, and the specified action is performed in the virtual space 60.

[0092] The VR mode execution unit 21c can determine the behavior (reaction) of the virtual character in the virtual space in response to the action of the user 61. The VR mode execution unit 21c generates image information for displaying the behavior of the virtual character determined in response to the action of the user 5, and outputs this generated image information to the display 24 as a stereoscopic image. As a result, an image of the virtual character performing the behavior determined in response to the action of the user 5 is displayed on the display 24. In this way, the VR mode execution unit 21c determines the behavior (reaction) of the virtual character to the action of the user 61, and outputs image information of the virtual character corresponding to the determined behavior to the display 24. In the VR mode, the interaction between the user 5 and the virtual character displayed as a stereoscopic image in this way is Interaction is realized.

[0093] The VR mode execution unit 21c performs a termination process to terminate the VR mode when a termination condition is satisfied. The termination condition is, for example, that a predetermined time (for example, one minute) has elapsed since the start of the VR mode, that an operation to be terminated has been detected, that the last event included in the scenario being executed in the VR mode has ended, and other conditions. The termination process performed when the termination condition is satisfied may include displaying on the display 24 a guidance to detach the information processing device 20 from the wearing tool 11, and displaying a login screen for the chat mode, which is the first mode.

[0094] Next, chat processing in the chat mode will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flow diagram showing the flow of processing in the chat mode in one embodiment, and Fig. 7 shows an example of a display image in the chat mode. It is assumed that, at the start of the chat mode, the HMD 10 is not worn on the head of the user 5, and the information processing device 20 is detached from the wearing equipment 11.

[0095] In the application processing system 1, a game is started in chat mode. In step S11, a login screen for logging in to the chat mode is displayed on the display 24 of the information processing device 20. When the login process is performed by the user 5, the chat process proceeds to step S12.

[0096] In step S12, messages are exchanged between the user 5 and the virtual character, and a chat takes place between them. When logging in to the chat mode, a chat display image for displaying the chat taking place between the user 5 and the virtual character is generated, and the chat display image is displayed on the display 24. FIG. 7 shows a chat display image 70, which is an example of a chat display image displayed on the display 24. The chat display image 70 includes a chat display area 71 including an icon 73 corresponding to the user 5, an icon 74 corresponding to the virtual character, a message 75a of the user 5, messages 76a and 76b of the virtual character, and a transition start object 78. The chat display image 70 also includes a menu area 72 arranged above the chat display area 71. The messages 76a, 76b of the virtual character are specified based on the chat data 28b stored in the storage 72 and other data in accordance with the application processing program. For example, the messages of the virtual character are displayed in order starting from the message at the root node, with reference to the chat data 28b defined in a tree structure. At the branching point of the nodes, a node according to the branching condition is selected, and the message corresponding to the selected node is displayed.

[0097] The user 5 can select one of the options included in the chat display image 70. This selection is made by touching with a finger an area on the display 24 where the option to be selected is displayed. When this selection is made, a message 75b corresponding to the selected option is newly displayed in the chat display area 71 as a message from the user 5.

[0098] In step S13, it is determined whether or not a mode transition condition from the first mode to the second mode is satisfied. As a mode transition condition from the first mode to the second mode, for example, a condition that a predetermined time (e.g., one minute) has elapsed since the start of chat mode is used. The elapsed time from the start of chat mode is measured, for example, using a system clock. The chat process continues by returning to step S12 until it is determined that the mode transition condition is satisfied. On the other hand, if it is determined that the mode transition condition is satisfied, the chat process proceeds to step S14.

[0099] In step S14, the transition start object 78 is displayed on the display 24, and the chat process proceeds to step S15. As shown in FIG. 7, the transition start object 78 is displayed, for example, in the chat area 71 as an object with a circular outer edge. When the selection of the transition start object is confirmed in step S15, the chat process proceeds to step S16. On the other hand, when the selection of the transition start object is not confirmed even after a predetermined time (for example, 10 seconds) has elapsed since the transition start object was displayed, it is determined that the selection of the transition start object has not been made, and the chat process ends. When it is determined that the selection of the transition start object has not been made, the process may return to step S12 and a process for resuming the chat may be performed. Whether or not the transition start object has been selected may be determined according to an operation on the touch panel 24b. For example, when a touch operation (for example, a tap operation) has been made on a position overlapping with the display area of ​​the transition start object is detected from a detection signal by the touch panel 24b, the transition start object is determined to have been selected.

[0100] In step S16, a process of transitioning to the second mode, VR mode, is started. When the transition process is started, the chat process is ended. In the chat mode, a selection of scenes in the destination VR mode may be displayed on the display 24. The user 5 can select a preferred scene from the selection.

[0101] The above chat processing is executed by the chat mode execution unit 21a. The chat mode execution unit 21a can execute the above chat processing alone or in cooperation with other functions as appropriate. When executing the chat processing, in addition to the data stored in the storage 27, data stored in other storages, detection information detected by various sensors, or other data may be used as necessary.

[0102] In a mode transition process from the chat mode, which is the first mode, to the VR mode, which is the second mode, guidance for encouraging the user to wear the HMD 10 is displayed on the display 24 of the information processing device 20. Following this guidance, a transitional object is displayed on the display 24. When this transitional object is selected by the user 5, the VR mode is started. The user 5 can select the transitional object by wearing the HMD 10 and continuously gazing at the transitional object for a predetermined period of time.

[0103] Next, VR processing in the VR mode will be described with reference to Figs. 8 and 9. Fig. 8 is a flow diagram showing the flow of VR processing in the VR mode in one embodiment, and Fig. 9 shows an example of a display image in the VR mode. At the start of the VR mode, it is assumed that the HMD 10 is worn on the head of the user 5. In the following description of the VR mode, it is assumed that the first scene is selected by the user 5 in the chat mode. Various scenes can be set as the first scene. For example, the first scene may be a scene for conversation in an arbitrary indoor or outdoor location. The VR processing in Fig. 8 is repeated until an end condition is met. The end condition may be, for example, a predetermined time (for example, one minute) has elapsed since the start of the VR mode, an operation to be ended has been detected, or the operation to be ended has been performed in the VR mode. The conditions are that the last event in the scenario being played has been completed, and other conditions.

[0104] When the VR mode is started, in step S21, image information of the virtual space corresponding to the first scene is generated, and an image corresponding to the image information is output to the display 24. The image information of the virtual space 60 in the first scene is generated, for example, by referring to the image data 28a. In step S21, first, image information of the initial display area 63 of the virtual space 60 in the first scene is generated, and an image corresponding to the image information is output to the display 24. FIG. 9 shows an example of a display image of the initial display area 63 of the first scene. The VR image 80 shown in FIG. 9 includes images showing the virtual character 65, the menu object 66, objects 82a to 82c corresponding to the background, and various other objects. The menu object 66 is arranged outside the main display area 64. Since the VR image 80 is an image from a first-person perspective, an image showing the user 61 is not displayed.

[0105] The virtual character image 65 can perform various actions in the virtual space. For example, the actions of the virtual character image 65 include talking to the user 61 (to the position of the user 61 in the virtual space), moving in the virtual space, picking up an object placed in the virtual space, and various other actions in the virtual space.

[0106] When the sensor unit 25 of the HMD 10 detects head movement of the user 5 while the VR image 80 is being displayed on the display 24, a first action of the user 61 in the virtual space 60 is identified based on this detected head movement. Then, a behavior (reaction) of the virtual character according to the identified first action of the user 61 is determined, and image information of the virtual character 65 performing this determined behavior is generated. The image information generated in this manner is output to the display 24. For example, when it is detected that the user 5 has performed an action of shaking his head vertically (a nodding action), an action of nodding is identified as the first action of the user 61 in the virtual space 60 based on the detection information of the HMD 10. Then, a virtual character 65 reacting to the first action of the user 61, that is, nodding, is generated. Image information of the virtual character image 65 is generated, and the image information is displayed on the display 24. In this way, in the VR mode, interaction between the user and the virtual character 65 is realized using the stereoscopic image of the virtual character.

[0107] In step S22, it is determined whether or not a termination condition for terminating the VR image display process is satisfied. If it is determined that the termination condition is not satisfied, the process returns to step S21 and the VR image display process continues. On the other hand, if it is determined that the termination condition is satisfied, the VR process proceeds to step S23.

[0108] In step S23, a process of ending the VR mode is performed. This process of ending the VR mode may include displaying on the display 24 a guidance for urging the user to remove the information processing device 20 from the wearing tool 11, and displaying a login screen for the chat mode, which is the first mode.

[0109] The above VR processing is executed by the VR mode execution unit 21c. The VR mode execution unit 21c can execute the above VR processing independently or in cooperation with other functions as appropriate.

[0110] Next, the second action identification process for identifying the second action will be described with reference to Fig. 10 and Fig. 11a to Fig. 11d. Fig. 10 is a flow diagram showing the flow of the second action identification process in one embodiment, and Fig. 11a to Fig. 11d show examples of display images in the process for identifying the second action. The process for identifying the second action is performed in parallel with the VR process described with reference to Fig. 8.

[0111] First, in step S31, it is determined whether or not the menu object 66 has been selected. For example, when the gaze time for gazing at the menu object 66 reaches a predetermined time, it is determined that the menu object 66 has been selected. If the menu object 66 has not been selected, the determination in step S31 is repeated. On the other hand, if it is determined that the menu object 66 has been selected, the second action identification process proceeds to step S32.

[0112] In step S32, the menu items associated with the menu object 66 are displayed on the display 24. FIG. 11a shows an example of an image of a virtual space including menu items associated with the menu object 66. The VR image 80 shown in FIG. 11a includes icons corresponding to two menu items. Specifically, the VR image 80 includes a menu image 67, which includes a menu item icon 68a and a menu item icon 68b. The menu item icon 68a is an icon indicating a menu item corresponding to a hand action related to the hand movement of the user 61 in the virtual space 60, and the menu item icon 68b is an icon indicating a menu item corresponding to a photo action related to taking a photo in the virtual space 60. The menu item icons 68a and 68b are icons indicating menu items to be displayed on the display 24. When the process for displaying the menu items on the display 24 is performed, the second action identification process proceeds to step S33. The menu item icons 68a and 68b displayed in the virtual space may be displayed so as to be parallel to one surface of a virtual structure arranged in the virtual space. For example, in the illustrated embodiment, the menu item icons 68a and 68b may be displayed so as to be parallel to the floor, wall, or ceiling of the room in the virtual space shown in FIG. 11a. By arranging the menu item icons 68a and 68b in this way, the menu item icons 68a and 68b can be arranged in the virtual space. The menu item icon 68a can be displayed as if it were a virtual object placed in the virtual space. This makes it possible to prevent the sense of immersion of the user 61 from being impaired. The menu item icon 68a may be arranged on a surface in the virtual space different from the surface on which the menu item icon 68b is arranged. Components of the menu image 67 other than the menu item icon 68a and the menu item icon 68b may also be displayed so as to be parallel to one surface of a virtual structure arranged in the virtual space.

[0113] In step S33, it is determined whether any of the menu item icons displayed in step S32 has been selected. For example, when the gaze time of gazing at any of the menu item icons included in the VR image 80 reaches a predetermined time, it is determined that the menu item icon has been selected. In the illustrated example, it is determined that the menu item icon 68a has been selected when the gaze time of gazing at the menu item icon 68a reaches a predetermined time, and it is determined that the menu item icon 68b has been selected when the gaze time of gazing at the menu item icon 68b reaches a predetermined time. Two or more menu item icons may be selected. If the selection of the menu item icon is not confirmed in step S33, the display of the menu item icon is stopped, and the second action identification process returns to step S31. On the other hand, if it is determined that the menu item icon has been selected, the second action identification process proceeds to step S34.

[0114] In step S34, one or more character gaze positions are set on the virtual character 65. Setting of the character gaze positions on the virtual character 65 will be described with reference to FIG. 11b. FIG. 11b shows the character gaze positions in the VR image 80 in schematic form to explain the character gaze positions set on the virtual character 65. The character gaze positions may be set to be visible by the user 5 in the VR image 80, or may be set to be invisible. In the example shown in FIG. 11b, a plurality of character gaze positions 81a to 81d are set on the virtual character 65. Each of the one or more character gaze positions set on the virtual character 65 is associated with a specific part of the virtual character 65. In the example of FIG. 11b, the character gaze positions 81a to 81d are respectively associated with the head, hair, shoulders, and arms of the female virtual character 65. When the character gaze position is set on the virtual character 65, the second action identification process proceeds to step S35.

[0115] In step S35, it is determined whether or not any of the character gaze positions among one or more character gaze positions set on the virtual character 65 has been gazed upon. In the example shown in FIG. 11b, when the gaze time for gazing at any of the character gaze positions 81a to 81d reaches a predetermined time, it is determined that the character gaze position has been selected. If none of the character gaze positions have been gazed upon in step S35, the setting of the character gaze position is cancelled, and the second action identification process returns to step S31. On the other hand, if it is determined that any of the character gaze positions has been gazed upon, the second action identification process proceeds to step S36.

[0116] In step S36, an action for virtual character 65 is specified based on the menu item selected in step S33 and the character gaze position gazed at in step S35, and the specified action is performed on virtual character 65. For example, when menu item icon 68a associated with a hand action is selected in step S33, and character gaze position 81a associated with the head of virtual character 65 is gazed at in step S35, an action of stroking the head of virtual character 65 is specified as a second action of user 61 on virtual character 65 in virtual space 60.

[0117] When the menu item icon 68a is selected, a different second action may be specified depending on the character gaze position that is gazed upon. For example, even when the menu item icon 68a is selected as in the above example, when the character gaze position 81c is gazed upon instead of the character gaze position 81a, an action different from the action of "stroking" may be specified as the second action. When the menu item icon 68a is selected and the character gaze position 81c is gazed upon, for example, an action of "poking" the shoulder of the virtual character 65 may be specified as the second action.

[0118] When the second action is specified, the specified second action is performed on the virtual character 65. For example, when an action of stroking the head of the virtual character 65 is specified as the second action, an image expressing the second action is displayed on the display 24 as a VR image 80, as shown in FIG. 11c. In the example shown in FIG. 11c, an image 83a showing the virtual hand of the user 61 is displayed near the head of the virtual character 65 to show the action of the user 61 stroking the head of the virtual character 65 in the virtual space 60. In another example, when an action of poking the shoulder of the virtual character 65 is specified as the second action, an image 83b showing the virtual hand of the user 61 is displayed near the shoulder of the virtual character 65 to show the action of the user 61 poking the shoulder of the virtual character 65 in the virtual space 60, as shown in FIG. 11d. The images 83a and 83b may be different from each other to express the difference between the actions performed on the virtual character 65. In response to the second action of the user 61, the virtual character 65 performs an action (reaction) in accordance with the application processing program.

[0119] In step S33, the menu item icon 68b associated with the photo action may be selected, and one of the character gaze positions set on the virtual character 65 in step S35 (for example, character gaze position 81a) may be gazed upon. In this case, the action of taking a photo including the virtual character 65 is specified as the second action. In response to the selection of the menu item icon 68b, the virtual character 65 may perform an action of posing for the camera held by the user 61. Also, in response to the selection of one of the character gaze positions set on the virtual character 65, an action of releasing the shutter may be performed. In this case, the image output on the display 24 when one of the character gaze positions is selected is displayed as the photo action. The captured action is stored in the storage 27 as an action captured by the action.

[0120] The above-mentioned second action identification process is executed by the VR mode execution unit 21c. The VR mode execution unit 21c can execute the above-mentioned second action identification process independently or in cooperation with other functions as appropriate.

[0121] Next, an application processing system according to another embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an application processing system 101 according to another embodiment. The application processing system 101 includes an information processing device 20, an HMD 110, and the HMD 110. The application processing system 101 differs from the application processing system 1 in that it can provide a VR mode without attaching the information processing device 20 to the HMD 110. The application processing system 101 will be described below, focusing on the differences from the application processing system 1.

[0122] The HMD 110, the information processing device 20, and the server 50 are connected to be able to communicate with each other via a network 40. The HMD 110 and the information processing device 20 may be connected to be able to communicate with each other according to a short-range wireless method such as Bluetooth (registered trademark) without going through the network 40. The HMD 110 differs from the HMD 10 of the application processing system 1 in that the HMD 110 can provide a VR mode even if the information processing device 20 is not attached.

[0123] The HMD 110 includes a computer processor 121, a memory 122, a communication I / F 123, a display 124, a sensor unit 125, a sound collector 126, and a storage 127. The computer processor 121, the memory 122, the communication I / F 123, the display 124, the sensor unit 125, the sound collector 126, and the storage 127 are configured in the same manner as the computer processor 21, the memory 22, the communication I / F 23, the display 24, the sensor unit 25, the sound collector 26, and the storage 27 of the information processing device 20. However, the display 124 does not need to include a touch panel. These components of the HMD 110 are assembled in a housing. This housing may be formed in a goggle shape like the housing 11a. Image data 28a, chat data 28b, and chat data 28c are stored in the housing 11a. The scenario data 28c, the event data 28d, and the game progress data 28e may be stored in the storage 127, the storage 27, or in another storage.

[0124] The functions of the chat mode execution unit 21a, the transition processing unit 21b, and the VR mode execution unit 21c are shared between the information processing device 20 and the HMD 110. Specifically, the function of the chat mode execution unit 21a is realized in the information processing device 20, and the function of the VR mode execution unit 21c is realized in the HMD 110. A part of the function of the transition processing unit 21b is realized in the information processing device 20, and the remaining part is realized in the HMD 110.

[0125] When starting a game in the application processing system 101, the user 5 starts a chat mode using the information processing device 20. The chat mode is processed by the chat mode execution unit 21a of the information processing device 20.

[0126] When a mode transition condition is met in the chat mode and a transition start object is selected, a mode transition process to the VR mode is started.

[0127] When the VR mode is started, the VR mode processing is performed by the VR mode execution unit 21c of the HMD 110.

[0128] In this manner, the application processing system 101 can provide the user 5 with interaction with a virtual character via the HMD 110.

[0129] According to the above embodiment, the following effects can be obtained. According to the above application processing system 1, 101, by gazing at the character gaze positions 81a to 81d on the virtual character 65 after selecting the menu object 66, it is possible to specify the second action that is not associated with the movement of the head of the user 61 in the virtual space 60. This allows the second action that is not associated with the movement of the head of the user 61 in the virtual space 60 to be performed without using any operation device other than the head mounted display 10, 110 such as an operation controller designed to be held by hand.

[0130] The second actions associated with the user's hand movements in the virtual space 60 include, for example, actions of "stroking," "hitting," "poking," and "pinching" the virtual character 65 in the virtual space 60, an action of "holding the hand" of the virtual character, an action of handling an item with the hand (for example, an action of taking a photo), and other actions associated with the hand movements of the user 61 in the virtual space 60.

[0131] According to the above application processing system 1, 101, the second action associated with the hand movement of the user 61 in the virtual space 60 is specified by gazing at the character gaze positions 81a to 81d set on the virtual character 65, rather than by the position and movement of the user's hand in the real world. This allows the user to perform the second action not associated with the movement of the user's head, without using any operation device other than the head mounted display 10, 110.

[0132] According to the above application processing system 1, 101, by selecting a specific object, it becomes possible to perform a second action that is not associated with the movement of the user's head.

[0133] According to the above application processing system 1, 101, it is possible to selectively use a plurality of second actions including a first gaze position action and a second gaze position action according to the character gaze positions 81a to 81d. This provides a wide variety of second actions that are not associated with the movement of the head of the user 61.

[0134] According to the above-described application processing system 1, 101, it is possible to use various types of second actions depending on the selected menu item. This provides a wide variety of second actions that are not associated with the movement of the head of the user 61.

[0135] In the above embodiment, the menu object 66 is a functional object for enabling the second action. Depending on the arrangement and display mode in the virtual space, the user 5 may unintentionally select the menu object 66. In addition, a functional object such as the menu object 66 may reduce the user's immersion in the virtual space. According to the above application processing system 1, 101, the menu object 66 is arranged outside the main display area 63 in the virtual space 61, so that it is possible to prevent erroneous selection and reduction in immersion.

[0136] According to the above application processing system 1, 101, it is possible to determine an appropriate main display area according to the scene. For example, the position of the main display area 63 is changed according to the posture of the user 61 between the first scene and the second scene.

[0137] According to the above application processing system 1, 101, it is possible to make the virtual character 65 react in accordance with the action of the user 61, thereby realizing an interaction between the user and the virtual character 65.

[0138] The embodiments of the present invention are not limited to the above-mentioned embodiments, and various modifications are possible within the scope of the gist of the present invention. For example, a part or all of the functions executed by the computer processor 21 and the computer processor 121 may be realized by a computer processor not shown in the above-mentioned embodiments, as long as the modifications do not depart from the gist of the invention. For example, the application processing system 1 and the application processing system 101 may include a game machine that executes at least a part of the application processing program, and a part of the functions realized by the processing of the computer processor 21 or the computer processor 121 may be realized by the processing of the game machine.

[0139] The embodiment of the present invention may include various other apparatuses, devices, and electronic components in addition to those described above. For example, the application processing system 1 and the application processing system 101 may include an operation device for receiving an operation of the user 5 in addition to the information processing device 20 and the HMDs 10 and 110. The application processing system 1 and the application processing system 101 may detect an operation of the user 5 by the operation device, and perform game processing in accordance with the detected operation.

[0140] In the processing procedures described in this specification, particularly in the processing procedures described using flow diagrams, it is possible to omit some of the steps that make up the processing procedures, to add steps that are not explicitly stated as steps that make up the processing procedures, and / or to change the order of the steps, and processing procedures in which such omissions, additions, or changes in order have been made are also included within the scope of the present invention as long as they do not deviate from the spirit of the present invention.

[0141] The inventions described in the claims of the original application of this application are set forth below. [1] An application processing system including one or more computer processors, the application processing system providing a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head mounted display attached to a user's head, the application processing system comprising: The one or more computer processors execute the computer readable instructions to: Detecting that a menu icon included in the virtual space has been selected based on detection information detected by the head mounted display; when it is detected that the menu icon has been selected, detecting that a first gaze position or a second gaze position set for the virtual character has been gazed upon based on detection information detected by the head mounted display; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon; when it is detected that the second gaze position is gazed upon, a second user action different from the first user action is identified as an action of the user on the virtual character; Application processing system. [2] when the first user action is identified as an action on the virtual character, the one or more computer processors cause the head mounted display to display a first action image corresponding to the first user action, and when the second user action is identified as an action on the virtual character, cause the head mounted display to display a second action image corresponding to the second user action and different from the first action image. [1] An application processing system according to the present invention. [3] the menu icon is arranged parallel to one surface of a virtual structure included in the virtual space; An application processing system according to [1] or [2]. [4] At least one of the first user action and the second user action is an action associated with a hand movement of the user in the virtual space. An application processing system according to any one of claims [1] to [3]. [5] The application processing system according to any one of [1] to [4], wherein the one or more computer processors cause the virtual character to react in response to at least one of the first user action and the second user action. [6] The menu icon is selectable in response to a selection of a specific object included in the virtual space. An application processing system according to any one of claims 1 to 5. [7] the image of the virtual space is an image in a display area centered on a reference line of sight of the user, The one or more computer processors identify a first primary display area in the virtual space in a first scene; the first main display area is set to include the reference line of sight, the specific object is disposed outside the first primary display area in the first scene; [6] An application processing system according to the present invention. [8] the one or more computer processors identify a second primary display area in the virtual space in a second scene; the second main display area is set to include the reference line of sight, the specific object is disposed outside the second primary display area in the second scene; [7] An application processing system according to the present invention. [9] 1. An application processing method for providing a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head mounted display worn on a user's head by executing computer readable instructions by one or more computer processors, comprising: detecting, based on detection information detected by the head mounted display, that a menu icon included in the virtual space has been selected; when it is detected that the menu icon has been selected, detecting that a first gaze position or a second gaze position set for the virtual character has been gazed upon based on detection information detected by the head mounted display; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon, and identifying a second user action different from the first user action as an action of the user on the virtual character when it is detected that the second gaze position is gazed upon; A game processing method comprising:

[10] An application processing program that, when executed by one or more computer processors, outputs an image of a virtual space including a virtual character to a head mounted display that is worn on a user's head, thereby providing the user with an interaction with the virtual character, the one or more computer processors; detecting, based on detection information detected by the head mounted display, that a menu icon included in the virtual space has been selected; when it is detected that the menu icon has been selected, detecting that a first gaze position or a second gaze position set for the virtual character has been gazed upon based on detection information detected by the head mounted display; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon, and identifying a second user action different from the first user action as an action of the user on the virtual character when it is detected that the second gaze position is gazed upon; An application processing program that causes [Explanation of symbols]

[0142] 1,101 Application Processing Systems

Claims

1. An application processing system including one or more computer processors, the application processing system providing a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head mounted display attached to a user's head, the application processing system comprising: The one or more computer processors execute the computer readable instructions to: Detecting that a menu icon included in the virtual space has been selected based on detection information detected by the head mounted display; when it is detected that the menu icon has been selected, detecting that either a first gaze position or a second gaze position set for the virtual character has been gazed upon based on detection information detected by the head mounted display; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon; When it is detected that the second gaze position is gazed upon, a second user action is identified as an action of the user on the virtual character; the first user action is a first type of action and is an action executed on a part corresponding to the first gaze position, the second user action is a second type of action different from the first type of action and is an action executed on a part corresponding to the second gaze position that is different from a part corresponding to the first gaze position, the menu icon is selectable in response to a selection of a specific object included in the virtual space; the image of the virtual space is an image in a display area centered on a reference line of sight of the user, the one or more computer processors: Identifying a first initial display area in the virtual space based on an initial setting of a first scene; specifying a first main display area within the first initial display area so as to include the reference line of sight at the start of the first scene and at least a portion of the virtual character; At the start of the first scene, the specific object is positioned outside the first main display area; outputting an image of the first initial display area to the head mounted display at the start of the first scene; Application processing system.

2. when the first user action is identified as an action on the virtual character, the one or more computer processors cause the head mounted display to display a first action image corresponding to the first user action, and when the second user action is identified as an action on the virtual character, cause the head mounted display to display a second action image corresponding to the second user action; the first action image is a virtual image of the user to represent the first type of action; the second action image is a virtual image of the user for representing the second type of action, and has an appearance different from that of the first action image; 2. The application processing system of claim 1.

3. the menu icon is arranged parallel to one surface of a virtual structure included in the virtual space; 3. The application processing system according to claim 1 or 2.

4. At least one of the first user action and the second user action is an action associated with a hand movement of the user in the virtual space.

4. The application processing system according to claim 1.

5. The application processing system according to claim 1 , wherein the one or more computer processors cause the virtual character to perform a reaction in response to at least one of the first user action and the second user action.

6. the one or more computer processors: Identifying a second initial display area in the virtual space based on an initial setting of a second scene; specifying a second main display area within the second initial display area so as to include the reference line of sight at the start of the first scene and at least a portion of the virtual character; At the start of the second scene, the specific object is positioned outside the second primary display area; outputting an image of the second initial display area to the head mounted display at the start of the second scene; 6. An application processing system according to claim 1.

7. 1. An application processing method for providing a user with an interaction with a virtual character by outputting an image of a virtual space including a virtual character to a head mounted display worn on a user's head by executing computer readable instructions by one or more computer processors, comprising: detecting, based on detection information detected by the head mounted display, that a menu icon included in the virtual space has been selected; when it is detected that the menu icon has been selected, detecting, based on detection information detected by the head mounted display, that either a first gaze position or a second gaze position set for the virtual character has been gazed upon; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon, and identifying a second user action as an action of the user on the virtual character when it is detected that the second gaze position is gazed upon; Equipped with the first user action is a first type of action and is an action executed on a part corresponding to the first gaze position, the second user action is a second type of action different from the first type of action and is an action executed on a part corresponding to the second gaze position that is different from a part corresponding to the first gaze position, the menu icon is selectable in response to a selection of a specific object included in the virtual space; the image of the virtual space is an image in a display area centered on a reference line of sight of the user, The application processing method includes: identifying a first initial display area in the virtual space based on an initial setting of a first scene; specifying a first main display area within the first initial display area so as to include the reference line of sight at the start of the first scene and at least a portion of the virtual character; At the start of the first scene, the specific object is positioned outside the first main display area; outputting an image of the first initial display area to the head mounted display at the start of the first scene; Further comprising: Game processing method.

8. An application processing program that, when executed by one or more computer processors, outputs an image of a virtual space including a virtual character to a head mounted display that is worn on a user's head, thereby providing the user with an interaction with the virtual character, the one or more computer processors; detecting, based on detection information detected by the head mounted display, that a menu icon included in the virtual space has been selected; when it is detected that the menu icon has been selected, detecting, based on detection information detected by the head mounted display, that either a first gaze position or a second gaze position set for the virtual character has been gazed upon; identifying a first user action as an action of the user on the virtual character when it is detected that the first gaze position is gazed upon, and identifying a second user action as an action of the user on the virtual character when it is detected that the second gaze position is gazed upon; Run the command, the first user action is a first type of action and is an action executed on a part corresponding to the first gaze position, the second user action is a second type of action different from the first type of action and is an action executed on a part corresponding to the second gaze position that is different from a part corresponding to the first gaze position, the menu icon is selectable in response to a selection of a specific object included in the virtual space; the image of the virtual space is an image in a display area centered on a reference line of sight of the user, The application processing program is configured to cause the one or more computer processors to: identifying a first initial display area in the virtual space based on an initial setting of a first scene; specifying a first main display area within the first initial display area so as to include the reference line of sight at the start of the first scene and at least a portion of the virtual character; At the start of the first scene, the specific object is positioned outside the first main display area; outputting an image of the first initial display area to the head mounted display at the start of the first scene; Further execute Application processing program.

Images