Information processing system, information processing method, and information processing program
The image processing system improves virtual space usability by allowing multiple users to interact and view virtual spaces from different perspectives using independently controlled displays, enhancing collaborative experiences.
Patent Information
- Application Number
- JP2024161129
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-01-28
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2040-09-18
AI Technical Summary
Existing systems for displaying images in virtual space lack usability improvements, particularly in terms of user interaction and perspective adjustment.
An image processing system with multiple displays, including a first display for a first user and a portable second display for a second user, allows for independent placement and control of virtual objects and cameras based on the position and attitude of the displays in real space, enabling different viewpoints and interactions.
Enables multiple users to view and interact with a virtual space from diverse perspectives, enhancing usability and allowing for collaborative gameplay or creation, with improved object placement and control mechanisms.
Smart Images

Figure 2024169572000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing system, an image processing program, and an image processing method capable of displaying images on a plurality of displays. [Background technology]
[0002] As a prior art, there is a system in which virtual objects are arranged in a virtual space and an image of the virtual space including the virtual objects is displayed on a display (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-74362 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, there is room for improvement in terms of improving usability regarding the display of images in a virtual space.
[0005] Therefore, an object of the present invention is to provide an image processing system, an image processing program, and an image processing method that can improve usability regarding the display of images in a virtual space. [Means for solving the problem]
[0006] In order to solve the above problems, the present invention employs the following configuration.
[0007] The image processing system of the present invention includes a first display visible to a first user, a first input means for receiving an input from the first user, and a portable second display carried by a second user and visible to the second user. The image processing system also includes a first object arranging means, a first object control means, a first virtual camera arranging means, a second virtual camera arranging means, a second object arranging means, a first display control means, and a second display control means. The first object arranging means arranges a first object in a virtual space regardless of the positions of the first display and the first user in the real space. The first object control means controls the first object in response to the input received by the first input means. The first virtual camera arranging means arranges a first virtual camera at a position corresponding to the position of the first object in the virtual space. The second virtual camera arranging means arranges a second virtual camera at a first position in the virtual space that is set according to the position of the second display relative to a reference in the real space, and sets the attitude of the second virtual camera to a first attitude according to the attitude of the second display in the real space. The second object arranging means arranges a second object at a second position in the virtual space that is set according to a position of the second display with respect to the reference in real space. The first display control means causes the first display to display an image of the virtual space including the second object, which is generated based on the first virtual camera. The second display control means causes the second display to display an image of the virtual space including the first object, which is generated based on the second virtual camera.
[0008] The second position may be the same as the first position or may be a different position. Furthermore, the "position corresponding to the position of the first object" may be the same as the position of the first object or may be a different position from the position of the first object.
[0009] According to the above, the first object is set in the virtual space regardless of the position of the first display in the real space, and the second object is set in the virtual space according to the position of the second display relative to the reference of the real space. The first virtual camera is disposed at a position in the virtual space corresponding to the position of the first object. The second virtual camera is disposed in the virtual space according to the position of the second display relative to the reference of the real space, and the attitude of the second virtual camera is set according to the attitude of the second display. Then, an image of the virtual space including the second object is displayed on the first display based on the first virtual camera, and an image of the virtual space including the first object is displayed on the second display based on the second virtual camera. Thereby, an image viewed from a position corresponding to the position of the first object in the virtual space can be displayed on the first display, and an image viewed from a position in the virtual space according to the position of the second display relative to the reference of the real space can be displayed on the second display. Therefore, the first user and the second user can view the virtual space from different viewpoints. Since the position of the second object is set in the virtual space in conjunction with the position of the second display in the real space, the first user can grasp the position of the second user in the real space by viewing the second object in the virtual space. Furthermore, the second user can view the first object in the virtual space according to the position and orientation of the second display in real space.
[0010] The image processing system may further include a second input means for accepting input from the second user, and a control means for affecting the virtual space or the first object in accordance with the input accepted by the second input means.
[0011] Based on the above, it is possible to affect the virtual space or the first object based on an input from the second user.
[0012] The control means may cause a predetermined object to appear in the virtual space in response to the input received by the second input means. The first display control means may cause the first display to display an image including the second object and the predetermined object, and the second display control means may cause the second display to display an image including the first object and the predetermined object.
[0013] Based on the above, it is possible to cause a predetermined object to appear in the virtual space and to display the predetermined object on the first display and the second display in response to an input by the second user.
[0014] The control means may also cause a change to the first object by using the predetermined object.
[0015] Based on the above, a predetermined object can be made to appear in response to an input by the second user, and a change can be imparted to the first object using the predetermined object.
[0016] The image processing system may further include a game control means for controlling a game using the first object, and the predetermined object may be an object that is advantageous or disadvantageous to the first object in the game.
[0017] According to the above, a game can be played using a first object controlled by a first user, and a predetermined object that is advantageous or disadvantageous to the first object in the game can be made to appear in a virtual space in response to an input from a second user, thereby allowing the second user to participate in the game played by the first user.
[0018] Moreover, the first display may be a head mounted display. The second display may include a camera that captures an image of real space in a rearward direction. The second virtual camera placement means may detect a marker that is placed in real space as the reference based on an image from the camera, and place the second virtual camera at the first position according to a position of the second display relative to the marker. The second display control means may superimpose an image of the virtual space generated based on the second virtual camera on an image of real space captured by the camera, and display the superimposed image on the second display.
[0019] According to the above, the first user wears the head mounted display and views the virtual space, while the second user can capture an image of the marker in the real space and view the virtual space from a position corresponding to the position of the second display relative to the marker, and can view the image of the virtual space superimposed on the image of the real space. The second user can view the virtual space from various viewpoints in the virtual space by moving and turning in the real space, and can view the virtual space viewed by the first user as if it were real space.
[0020] The image processing system may also include a controller held by the first user or the second user. The first display or the second display may be a head-mounted display. The image processing system may also include a camera capable of capturing an image of a hand of the first user or the second user wearing the head-mounted display. The image processing system may also further include an instruction object placement means and a third display control means. The instruction object placement means detects the hand of the first user or the second user or the controller based on an input image from the camera, places a virtual controller object simulating the controller at the position of the hand, and further places an instruction object for indicating a position in the virtual space at a predetermined position of the controller object. The third display control means causes the head-mounted display to display an image including the controller object and the instruction object.
[0021] According to the above, since a virtual controller object simulating a controller is placed at the position of the hand, even when the user is wearing a head mounted display, the user can easily perform operations using the controller while looking at the controller object. Also, since a pointing object is provided at a predetermined position of the controller object, the user can accurately point to a position in the virtual space while looking at the pointing object, thereby improving operability.
[0022] The image processing system may further include a camera that captures an image of the real space, and a virtual object placement means. The virtual object placement means detects a specific object existing in the real space based on an input image from the camera, and places a virtual object corresponding to the specific object at a position in the virtual space according to a position of the specific object in the real space. The first display control means may cause the first display to display an image of the virtual space including the virtual object.
[0023] Based on the above, it is possible to make a virtual object corresponding to an actual object existing in real space appear in the virtual space, and to display the virtual object on the first display.
[0024] The second display control means may cause the second display to display an image of the virtual space including the virtual object.
[0025] Based on the above, it is possible to display a virtual object corresponding to an actual object also on the second display.
[0026] Furthermore, the virtual object placing means may cause the virtual object to continue in the virtual space when the specific object is no longer detected based on the input image from the camera after placing the virtual object by detecting the specific object. The first display control means may cause the first display to display an image of the virtual space including the virtual object even when the specific object is no longer detected based on the input image from the camera.
[0027] According to the above, after a virtual object is placed in the virtual space by the camera detecting a specific object in real space, the virtual object can be made to persist in the virtual space even if, for example, the camera moves or its orientation changes and the specific object can no longer be detected.
[0028] The first display may be a head mounted display.The image processing system may include a first notification means for determining whether the first user has gone outside a predetermined range in a real space and, when it is determined that the first user has gone outside the predetermined range, providing a predetermined notification to the first user, and a second notification means for determining whether the second user has entered the predetermined range and, when it is determined that the second user has entered the predetermined range, providing a predetermined notification to the second user.
[0029] Based on the above, when the first user goes out of the predetermined range or when the second user enters the predetermined range, it is possible to notify the first user and the second user.
[0030] The first input means may be a controller capable of key input. The first object control means may control a position or an attitude of the first object in response to a key input to the first input means.
[0031] Based on the above, the first user can control the first object by key input.
[0032] The first virtual camera may be disposed at a position of the first object.
[0033] Based on the above, an image of the virtual space seen from the first object can be displayed on the first display.
[0034] The first virtual camera may be disposed at a position a predetermined distance away from the position of the first object so as to include the first object in an imaging range.
[0035] Based on the above, it is possible to display, on the first display, an image of the virtual space including the first object from a position that is a predetermined distance away from the first object.
[0036] Another invention may be a program executed in one or more computers included in the image processing system. Another invention may be an image processing method performed in the image processing system. Effect of the Invention
[0037] According to the present invention, the first display can display an image viewed from a position corresponding to the position of a first object in virtual space, and the second display can display an image viewed in the virtual space from a position according to the position of the second display relative to a reference in real space. [Brief description of the drawings]
[0038] [Figure 1] FIG. 1 is a diagram showing an example of how a virtual space image is viewed by a plurality of users using the image processing system 1 of this embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the overall configuration of an image processing system 1. [Diagram 3] FIG. 1 is a diagram showing an example of the configuration of a first HMD 10 (or a second HMD 30) included in an image processing system 1. [Figure 4] FIG. 1 is a diagram showing an example of the configuration of a smart device 20 included in an image processing system 1. [Diagram 5] FIG. 1 shows an example of the configuration of a controller 18 included in the image processing system 1. [Figure 6] FIG. 2 is a diagram for explaining a coordinate system of a virtual space that is set by capturing an image of a marker 40 with a camera 23 of a smart device 20. [Figure 7] FIG. 1 illustrates an example of a virtual space 100 including multiple virtual objects. [Figure 8] FIG. 1 is a diagram showing the first HMD 10 and the virtual camera 111 as viewed from the right side, illustrating how the attitude of the virtual camera 111 changes in conjunction with the change in attitude of the first HMD 10. [Figure 9] FIG. 1 shows an example of a method for detecting the positions of a first HMD 10, a second HMD 30, and controllers 18 and 38. [Figure 10] FIG. 2 is a diagram showing an example of an image displayed on the display 12 of the first HMD 10, and is an example of an image when the first user faces forward. [Figure 11] FIG. 13 is a diagram showing an example of an image displayed on the display 12 of the first HMD 10, in which the first user is facing to the right. [Figure 12] FIG. 11 is a diagram showing an example of a bullet object 112 being shot into a virtual space in the state shown in FIG. [Figure 13] FIG. 2 is a diagram showing an example of an image displayed on the display 22 of the smart device 20a, when the second user a is in the position shown in FIG. [Figure 14] FIG. 13 is a diagram showing an example of an image displayed on the display 22 of the smart device 20a after the second user a moves closer to the third user. [Figure 15] FIG. 13 is a diagram showing an example of a state in which a character 120a shoots a bullet object 122 in response to an operation on a smart device 20a. [Figure 16] FIG. 13 is a diagram showing an example of an image displayed on the display 32 of the second HMD 30, and is an example of an image when the virtual camera 131 is set to the first mode. [Figure 17]FIG. 13 is a diagram showing an example of an image displayed on the display 32 of the second HMD 30, and is an example of an image when the virtual camera 131 is set to the second mode. [Figure 18] FIG. 13 is a diagram showing an example of an image viewed by a user in the creation mode. [Figure 19] FIG. 13 is a diagram showing a state in which a plurality of part objects 173 are arranged on a plane. [Figure 20] FIG. 13 is a diagram showing an example of grid lines for displaying a grid on which each part object is placed. [Figure 21] A diagram showing a can 80 as an actual object placed on a table in real space. [Figure 22] FIG. 13 is a diagram showing a state in which a virtual object 180 corresponding to a can 80 is placed in a virtual space. [Figure 23] FIG. 2 shows an example of data D20 stored in each smart device 20. [Figure 24] FIG. 13 is a diagram showing an example of data D10 stored in a first HMD 10. [Diagram 25] FIG. 13 is a diagram showing an example of data D30 stored in the second HMD 30. [Figure 26] FIG. 1 shows an example of data D60 stored in an information processing device 60. [Figure 27] 1 is a flowchart showing an example of an HMD process executed in the first HMD 10 or the second HMD 30. [Figure 28] 1 is a flowchart showing an example of a smart device process executed in the smart device 20. [Figure 29] 1 is a flowchart showing an example of a game control process executed in the information processing device 60. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0039] (Image Processing System Configuration) The image processing system 1 of the present embodiment is a system that allows multiple users to view an image of the same virtual space, for example, a system for playing a game among multiple users. Hereinafter, the image processing system 1 of the present embodiment will be described with reference to the drawings.
[0040] FIG. 1 is a diagram showing an example of how an image of a virtual space is viewed by multiple users using the image processing system 1 of this embodiment. FIG. 2 is a diagram showing an example of the overall configuration of the image processing system 1. FIG. 3 is a diagram showing an example of the configuration of a first HMD 10 (or a second HMD 30) included in the image processing system 1. FIG. 4 is a diagram showing an example of the configuration of a smart device 20 included in the image processing system 1. FIG. 5 is a diagram showing an example of the configuration of a controller 18 included in the image processing system 1.
[0041] As shown in FIG. 1, the image processing system 1 includes a first head mounted display 10 (hereinafter referred to as the "first HMD") worn by a first user, and a second head mounted display 30 (hereinafter referred to as the "second HMD") worn by a third user.
[0042] The first HMD 10 is worn on the head of a first user so as to cover both eyes of the first user. The second HMD 30 is worn on the head of a third user so as to cover both eyes of the third user. The first HMD 10 and the second HMD 30 are devices having basically the same configuration. Hereinafter, an example of the configuration of the first HMD 10 and the second HMD 30 will be described with reference to FIG. 3.
[0043] As shown in FIG. 3, the first HMD 10 has a right-eye display 12R and a left-eye display 12L. When the user wears the first HMD 10, the right-eye display 12R is viewed by the user's right eye through a lens (not shown). When the user wears the first HMD 10, the left-eye display 12L is viewed by the user's left eye through a lens (not shown). Any display device may be used as the right-eye display 12R and the left-eye display 12L, for example, a liquid crystal display device or an organic EL display device. When the user wears the first HMD 10, the right eye and the left eye of the first user are covered by the right-eye display 12R and the left-eye display 12L, respectively, and the field of view of the first user is almost covered by the display 12. Therefore, when the first HMD 10 is worn, the first user does not see the real space but only sees the image displayed on the display 12.
[0044] The display 12 may be a transmissive display. In this case, the first user can view the real space through the display 12 and can view the image displayed on the display 12.
[0045] In addition, a right camera 13R located on the right side as seen from the user and a left camera 13L located on the left side as seen from the user are provided in front of the first HMD 10. The right camera 13R and the left camera 13L are cameras for receiving visible light and generating an image (RGB image). Hereinafter, the right camera 13R and the left camera 13L may be collectively referred to as "camera 13".
[0046] In addition, a right infrared camera 14R located on the right side as seen from the user and a left infrared camera 14L located on the left side as seen from the user are provided in front of the first HMD 10. The right infrared camera 14R and the left infrared camera 14L are cameras for receiving infrared light and generating infrared images. As will be described in detail later, the right infrared camera 14R and the left infrared camera 14L are used to detect the user's hands. In the following, the right infrared camera 14R and the left infrared camera 14L may be collectively referred to as the "infrared camera 14."
[0047] The first HMD 10 is set with an x-axis which is an axis in the right direction of the first HMD 10, a y-axis which is an axis in the upward direction of the first HMD 10, and a z-axis which is an axis in the front direction of the first HMD 10. The first HMD 10 is equipped with an inertial sensor 15 (see FIG. 2) which detects acceleration and angular velocity in each of the x-axis, y-axis, and z-axis directions. The attitude of the first HMD 10 can be calculated based on the acceleration and angular velocity detected by the inertial sensor 15.
[0048] The first HMD 10 also includes a communication module 16 for connecting to a network (such as a LAN, a WAN, or the Internet) (see FIG. 2).
[0049] The second HMD 30 has a similar configuration to the first HMD 10. Specifically, the second HMD 30 has left and right displays 32 (32L and 32R), left and right cameras 33 (33L and 33R), and left and right infrared cameras 34 (34L and 34R) (see FIG. 2). The second HMD 30 also includes an inertial sensor 35 that detects acceleration and angular velocity in the x-axis, y-axis, and z-axis directions, and a communication module 36 (see FIG. 2). Note that, hereinafter, the first HMD 10 and the second HMD 30 may be collectively referred to as "HMD".
[0050] As shown in FIG. 1, the image processing system 1 also includes a smart device 20 carried by a second user. There may be multiple second users. Here, the second users include a second user a and a second user b. The second user a holds the smart device 20a, and the second user b holds the smart device 20b. In the following, when there is no need to distinguish between the smart devices 20a and 20b, they will be referred to as "smart devices 20."
[0051] The smart device 20 is, for example, a smartphone or a tablet terminal. As shown in FIG. 4, a display 22 is provided on the front side of the smart device 20. The display 22 is, for example, a liquid crystal display device or an organic EL display device. A camera 23 is provided on the rear side of the smart device 20 (the side opposite the display 22). The camera 23 is a camera for receiving visible light and generating an RGB image. The camera 23 may be provided in multiple locations, including on a surface different from the rear side.
[0052] The smart device 20 is also set with an x-axis which is an axis in the right direction of the smart device 20, a y-axis which is an axis in the upward direction, and a z-axis which is an axis in the front direction. The smart device 20 is equipped with an inertial sensor 24 (see FIG. 2) which detects acceleration and angular velocity in each of the x-axis, y-axis, and z-axis directions. The smart device 20 can calculate the attitude of the smart device 20 based on the acceleration and angular velocity detected by the inertial sensor 24. The smart device 20 is also equipped with a communication module 25 for connecting to a network (see FIG. 2).
[0053] Furthermore, image processing system 1 includes a controller 18 held by the first user, and a controller 38 held by a third user. An example of the configuration of controller 18 will be described with reference to Fig. 5.
[0054] As shown in FIG. 5, the controller 18 includes a plurality of buttons 181 (A button, B button, C button, D button) that can be pressed by the user, an analog stick 182 for instructing a direction, and a cross key 183 for instructing a direction. The controller 18 also includes an inertial sensor (see FIG. 2) for detecting acceleration and angular velocity in each of the x-axis, y-axis, and z-axis directions shown in FIG. 5. The attitude and movement of the controller 18 can be calculated based on the acceleration and angular velocity detected by the inertial sensor of the controller 18. For example, it can be determined whether the controller 18 has been shaken based on the acceleration and angular velocity from the inertial sensor. The controller 18 also includes a communication module for connecting to a network (see FIG. 2). The controller 38 has basically the same configuration as the controller 18, and therefore a description thereof will be omitted.
[0055] As shown in FIG. 1, the image processing system 1 also includes a marker 40, a display 50, and an information processing device 60 connected to the display 50 in a wired or wireless manner.
[0056] The marker 40 is, for example, a card on which a predetermined image (for example, an image of an arrow) is printed, and is placed on the upper surface of a table or the like. The display 50 is a stationary display device such as a television, and may be, for example, a liquid crystal display device or an organic EL display device. The display 50 is a display device that allows a fourth user who is not carrying an HMD (first HMD 10 or second HMD 30) or a smart device 20 to view an image in a virtual space.
[0057] As shown in FIG. 2, the information processing device 60 includes one or more processors 61, a memory 62, and a communication module 63 for connecting to a network. The information processing device 60 is a device for controlling the entire game of this embodiment. The information processing device 60 manages the positions and attitudes of various virtual objects in a virtual space, and controls the virtual objects. Although details will be described later, the memory 62 of the information processing device 60 stores character information regarding each character placed in the virtual space. The character information includes information regarding the position and attitude of the character. In addition, the information processing device 60 stores virtual camera information regarding the position and attitude of each virtual camera.
[0058] Furthermore, although details will be described later, the information processing device 60 generates an image of a virtual space to be displayed on the display 50, and outputs the generated image to the display 50. Furthermore, the information processing device 60 generates an image of a virtual space to be displayed on the first HMD 10 and the second HMD 30, and transmits the generated image to the first HMD 10 and the second HMD 30 via a network. As the information processing device 60, any information processing device such as a personal computer, a game device, a server device, a smartphone, a tablet terminal, etc. may be used. Furthermore, the information processing device 60 may be configured by a plurality of devices, or may be configured by connecting a plurality of devices via a network (LAN, WAN, Internet, etc.).
[0059] In this embodiment, the first user to the fourth user are basically located in the same place (for example, the same room), and multiple people play a game using the image processing system 1. The first user performs operations for the game on, for example, a mat 51 placed at a position slightly away from the table. The first user performs game operations, for example, by swinging the controller 18 while viewing images of the virtual space displayed on the left and right displays 12 of the first HMD 10. Specifically, the right-eye virtual space image generated based on the right virtual camera arranged in the virtual space is displayed on the right-eye display 12R of the first HMD 10, and the left-eye virtual space image generated based on the left virtual camera arranged in the virtual space is displayed on the left-eye display 12L of the first HMD 10. This allows the first user to view a stereoscopic image of the virtual space, and to experience virtual reality (VR) as if he or she were present in the virtual space.
[0060] The first user (first HMD 10) can play the game even if he / she is far away from the table (marker 40), regardless of his / her position relative to the marker 40 placed on the table. In other words, even if the camera 13 of the first HMD 10 does not capture the image of the marker 40, the first user can view the image of the virtual space.
[0061] The second user is positioned near the table, captures an image of the marker 40 placed on the table with the camera 23 of the smart device 20, and plays the game while viewing the image displayed on the smart device 20. An image (hereinafter, referred to as a superimposed image) in which an image of the virtual space is superimposed on an image of the real space captured by the camera 23 is displayed on (the display 22 of) the smart device 20. This allows the second user to have a sense that, for example, a virtual object in the virtual space exists in the real space, and to experience augmented reality (AR). Although the details will be described later, a coordinate system of the virtual space is defined based on the marker 40 in the real space, and the virtual space and the real space are associated with each other.
[0062] In addition, the smart device 20 does not necessarily need to display a superimposed image in which an image of the virtual space is superimposed on an image of the real space, and only the image of the virtual space may be displayed. Even if only the image of the virtual space is displayed on the smart device 20, the virtual space and the real space are associated with each other because the coordinate system of the virtual space is defined based on the marker 40 in the real space. Therefore, the second user can have a sense that a virtual object in the virtual space exists in the real space. That is, in the augmented reality (AR) in this embodiment, as long as the reference of the real space and the virtual space are associated with each other, it is not necessarily required that the image of the virtual space is superimposed on the image of the real space and displayed, and only the image of the virtual space may be displayed on the display.
[0063] Furthermore, a third user is positioned near the table, captures an image of the marker 40 with the camera 33 of the second HMD 30, and plays the game by operating the controller 38 while viewing images of the virtual space displayed on the left and right displays 32 of the second HMD 30. Note that, although details will be described later, there are cases in which the third user can play the game even if the camera 33 of the second HMD 30 cannot capture an image of the marker 40.
[0064] Moreover, the fourth user views the image of the virtual space displayed on the display 50 and watches the game being played between the first user to the third user.
[0065] (Definition of the virtual space coordinate system based on marker 40) Next, a description will be given of associating the real space with the virtual space by capturing an image of the marker 40 with a camera. Fig. 6 is a diagram for explaining a coordinate system of the virtual space that is set by capturing an image of the marker 40 with the camera 23 of the smart device 20.
[0066] As shown in FIG. 6, when the real space is captured by the camera 23 of the smart device 20, an image including the marker 40 is acquired. The smart device 20 detects the marker 40 from the acquired image using various image recognition techniques. When the smart device 20 detects the marker 40 from the acquired image, the smart device 20 calculates the position of the smart device 20 (the camera 23 of the smart device 20) relative to the marker 40 in the real space based on the size, shape, orientation, etc. of the image of the marker 40. For example, the smart device 20 sets an XYZ coordinate system in which a predetermined vertex of the marker 40 is the origin O, the direction opposite to the direction of the arrow in the marker 40 is the X-axis, the direction perpendicular to the arrow is the Z-axis, and the direction perpendicular to the marker 40 (upward) is the Y-axis. This XYZ coordinate system is used as a coordinate system for defining a position in the virtual space. Furthermore, the smart device 20 calculates the position of the smart device 20 with respect to the marker 40 (relative position from the marker) as coordinate values (X1, Y1, Z1) in the XYZ coordinate system based on the image of the marker 40 included in the image acquired by the camera 23. By placing the virtual camera at this position (X1, Y1, Z1) in the virtual space, the position of the smart device 20 based on the marker 40 in the real space and the position of the virtual camera in the virtual space become consistent. Then, for example, when a virtual object 101 is placed at a predetermined position on the XZ plane of the virtual space, the virtual camera captures an image of the virtual space and displays it on the display 22, so that the virtual object 101 is displayed superimposed on the image of the real space as shown in FIG. 6.
[0067] (Explanation of virtual space) Next, various virtual objects arranged in the virtual space will be described. An XYZ coordinate system is set based on a marker 40 on a table, and a virtual space including a plurality of virtual objects is set. Fig. 7 is a diagram showing an example of a virtual space 100 including a plurality of virtual objects.
[0068] As shown in Fig. 7, in the game of this embodiment, a virtual space is formed on a table. Various virtual objects for forming the terrain in the virtual space 100 are arranged on the XZ plane (i.e., on the table). For example, a ground object 141, a tree object 142, a river object 143, a bridge object 144, and a hill object 145 are arranged on the XZ plane. The hill object 145 is a virtual object having height, and the character 110 is arranged on the top of the hill object 145. Also, for example, the character 130 is initially arranged on the ground object 141.
[0069] Character 110 is a virtual object corresponding to a first user and is controlled by the first user. In the game of this embodiment, the position of character 110 is fixed at the top of hill object 145. Character 110 holds bullet object 112. For example, when the first user swings controller 18, character 110 shoots bullet object 112 into virtual space 100. The shot bullet object 112 moves within virtual space 100. For example, when bullet object 112 hits character 130, the vitality value of character 130 decreases or character 130 falls. When character 130 falls, the first user side wins.
[0070] Also, a virtual camera 111 is placed at a position corresponding to the position of the character 110. Specifically, the virtual camera 111 is placed at a position that coincides with the character 110 (for example, the position of the eyes of the character 110). Note that in FIG. 7, the position of the character 110 and the position of the virtual camera 111 appear to be shifted, but in reality, these positions coincide. The virtual camera 111 includes a left virtual camera 111L and a right virtual camera 111R. The left virtual camera 111L is a virtual camera for generating a left-eye virtual space image displayed on the left-eye display 12L of the first HMD 10. The right virtual camera 111R is a virtual camera for generating a right-eye virtual space image displayed on the right-eye display 12R of the first HMD 10. The left virtual camera 111L and the right virtual camera 111R are placed in the virtual space 100 at a predetermined distance (for example, a virtual space distance corresponding to the average eye distance of the user) away from each other.
[0071] It is not necessary for the virtual camera 111 to be positioned exactly at the position of the character 110's eyes. For example, the virtual camera 111 may be positioned at a position where the character 110 is displayed, but shifted vertically or horizontally from the position of the character 110's eyes. The virtual camera 111 may not be positioned at the same position as the character 110, but may be positioned at a position having a predetermined positional relationship with the character 110 (a position a predetermined distance away from the character 110). For example, the virtual camera 111 may be positioned at a predetermined position behind the character 110. The virtual camera 111 may also be positioned at a predetermined position diagonally behind the character 110. In other words, "the virtual camera 111 is positioned at a position corresponding to the position of the character 110" includes both the case where the virtual camera 111 is positioned at the same position as the character 110 and the case where the virtual camera 111 is positioned at a position a predetermined distance away from the character 110.
[0072] The orientations of the character 110 and the virtual camera 111 (111R and 111L) are set according to the orientation of the first user (first HMD 10) in real space. That is, the orientations of the character 110 and the virtual camera 111 are linked to the orientation of the first HMD 10 in real space.
[0073] 8 is a diagram of the first HMD 10 and virtual camera 111 as viewed from the right side, showing how the attitude of virtual camera 111 changes in conjunction with the change in attitude of first HMD 10. A Zc axis that is an axis of the line of sight, an Xc axis that is an axis in the rightward direction, and a Yc axis that is an axis in the upward direction are set in virtual camera 111. As shown in FIG. 8, the attitude of virtual camera 111 is controlled so that the Xc axis, Yc axis, and Zc axis of virtual camera 111 are parallel to the x axis, y axis, and z axis of first HMD 10, respectively.
[0074] Specifically, a state in which the direction of the z-axis of the first HMD 10 has not changed since the orientation of the first HMD 10 was initialized and the y-axis of the first HMD 10 faces in the opposite direction to the direction of gravity is called the reference orientation. When the first HMD 10 is in the reference orientation, the virtual camera 111 faces forward (for example, in the positive direction of the X-axis) in the virtual space. That is, the Zc-axis indicating the line of sight of the virtual camera 111 is parallel to the X-axis of the virtual space. As shown in the right diagram of FIG. 8, when the first user faces diagonally upward in the real space, the orientation of the first HMD 10 changes from the reference orientation, and the z-axis of the first HMD 10 faces diagonally upward. In conjunction with this change in the orientation of the first HMD 10, the Zc-axis of the virtual camera 111 also faces diagonally upward in the virtual space. Also, although not shown in the figure, for example, if the first HMD 10 rotates 30 degrees to the right from the reference posture (rotates 30 degrees counterclockwise around the y axis), the virtual camera 111 also rotates 30 degrees to the right (30 degrees counterclockwise around the Yc axis) in the virtual space.
[0075] In this way, the posture of the virtual cameras 111 (111R and 111L) is set so as to match the posture of the first HMD 10 in real space. The posture of the character 110 also links with the posture of the virtual camera 111. That is, a change in the posture of the first HMD 10 changes the postures of the virtual camera 111 and the character 110. Note that when the virtual camera 111 is placed at a position a predetermined distance away from the character 110 (for example, a predetermined position behind the character 110), not only the posture but also the position of the virtual camera 111 may change in response to a change in the posture of the character 110.
[0076] On the other hand, even if the position of the first HMD 10 in the real space changes, the positions of the virtual camera 111 and the character 110 do not change, and the virtual camera 111 and the character 110 are fixed at the top of the hill object 145.
[0077] Returning to FIG. 7, the character 130 is placed on the ground object 141. The character 130 is a virtual object corresponding to the third user and is controlled by the third user. Specifically, the character 130 moves in the virtual space 100 in response to a key input (input to a button, an analog stick, or a cross key) to the controller 38. The posture of the character 130 also changes in response to the key input. The character 130 can move on the ground object 141, the bridge object 144, and the hill object 145. The third user uses the controller 38 to move the character 130 on the ground object 141, for example, to the top of the hill object 145 through the bridge object 144. Then, when the character 130 reaches the top of the hill object 145, a battle between the character 130 and the character 110 begins. If the character 130 defeats the character 110 in the battle, the third user wins.
[0078] A virtual camera 131 corresponding to a third user is set in the virtual space 100. The virtual camera 131 includes a left virtual camera 131L and a right virtual camera 131R. The left virtual camera 131L is a virtual camera for generating a left-eye virtual space image to be displayed on the left-eye display 32L of the second HMD 30. The right virtual camera 131R is a virtual camera for generating a right-eye virtual space image to be displayed on the right-eye display 32R of the second HMD 30. The left virtual camera 131L and the right virtual camera 131R are disposed a predetermined distance apart.
[0079] The third user plays the game while viewing an image generated based on the virtual camera 131. The third user can switch the viewpoint (i.e., the position of the virtual camera 131) between the first mode and the second mode. When the first mode is set, the virtual camera 131 is placed at a position overlooking the virtual space 100, independent of the position of the character 130.
[0080] Specifically, when the first mode is set, the virtual camera 131 is placed at a position in the virtual space according to the position of the second HMD 30 in the real space. The position of the second HMD 30 based on the marker 40 is calculated by the same method as that described with reference to FIG. 6, and the virtual camera 131 is placed at the calculated position. That is, the marker 40 is detected based on an image captured by the camera 33 of the second HMD 30, and the position of the second HMD 30 relative to the marker 40 is calculated based on the size, shape, and orientation of the detected marker 40. The virtual camera 131 is set at a position in the virtual space according to the calculated position of the second HMD 30. For example, the left virtual camera 131L is set at a position in the virtual space that coincides with the position of the left camera 33L of the second HMD 30 based on the marker 40, and the right virtual camera 131R is set at a position in the virtual space that coincides with the position of the right camera 33R of the second HMD 30 based on the marker 40.
[0081] On the other hand, when the second mode is set, the virtual camera 131 is set to a position corresponding to the position of the character 130 (position 131' in FIG. 7). For example, the virtual camera 131 may be set to a position that coincides with the character 130 (for example, the position of both eyes of the character 130). The virtual camera 131 may also be set to a predetermined position behind the character 130. When the second mode is set, the virtual camera 131 moves while following the character 130.
[0082] Characters 120a and 120b are placed in the virtual space 100. The character 120a is a virtual object corresponding to a second user a (smart device 20a). The character 120b is a virtual object corresponding to a second user b (smart device 20b).
[0083] Virtual camera 121a is placed at a position corresponding to the position of character 120a. Virtual camera 121a may be set at a position that matches character 120a (for example, the position of the head of character 120a). Virtual camera 121a may also be set at a predetermined position behind character 120a. Similarly, virtual camera 121b is placed at a position that corresponds to the position of character 120b. Virtual camera 121b may be set at a position that matches character 120b, or may be set at a predetermined position behind character 120b.
[0084] As described with reference to Fig. 6, the positions of the character 120a and the virtual camera 121a are set according to the position of the smart device 20a in the real space. For example, the positions of the character 120a and the virtual camera 121a in the virtual space 100 correspond to the position of the smart device 20a relative to the marker 40 in the real space. Similarly, the positions of the character 120b and the virtual camera 121b in the virtual space 100 correspond to the position of the smart device 20b relative to the marker 40 in the real space.
[0085] In addition, the orientations of the character 120a and the virtual camera 121a are controlled to match the orientation of the smart device 20a in the real space. Similarly, the orientations of the character 120b and the virtual camera 121b are controlled to match the orientation of the smart device 20b in the real space.
[0086] The position and posture of the character 120 (and the virtual camera 121) are controlled according to the position and posture of the smart device 20 in the real space, so when the second user moves while holding the smart device 20 or changes the orientation of the smart device 20, the character 120 (and the virtual camera 121) also moves in the virtual space 100, and the orientation of the character 120 (and the virtual camera 121) changes.
[0087] 7, a virtual camera 150 is placed in the virtual space 100. The virtual camera 150 is a virtual camera for generating an image to be displayed on a stationary display 50, and is fixed at a predetermined position in the virtual space 100 in a predetermined attitude. The virtual camera 150 is placed at a position overlooking the entire virtual space 100. For example, the imaging range of the virtual camera 150 includes the characters 110, 120, and 130, and an image of the virtual space including the characters 110, 120, and 130 is displayed on the display 50. The position and / or attitude of the virtual camera 150 may be changed in response to a user operation.
[0088] Note that the display 50 may display an image of the virtual space seen from the virtual camera 111 or the virtual camera 131. Also, the display 50 may display an image of the virtual space seen from the virtual camera 121. Also, the user may select one of the virtual cameras 111, 121, 131, and 150, and an image generated based on the selected virtual camera may be displayed on the display 50. Also, the display 50 may display the same image as the image displayed on the display 22 of the smart device 20 (an image in which an image of the virtual space is superimposed on an image of the real space).
[0089] In this manner, in this embodiment, the marker 40 is placed on the table, and the coordinate system of the virtual space is set with the marker 40 as a reference. The top surface of the table corresponds to the XZ plane of the virtual space, and virtual objects are placed on the XZ plane. As a result, when the table is viewed through the smart device 20, the virtual objects are displayed on the table, and the table can be used as a display to play a game.
[0090] The positions of the first HMD 10 and / or the second HMD 30 may be detected by the following method. FIG 9 is a diagram showing an example of a method for detecting the positions of the first HMD 10, the second HMD 30, and the controllers 18, 38.
[0091] As shown in FIG. 9, two sensor bars 55 are arranged at predetermined positions in real space. The sensor bars 55 output, for example, infrared light from their tips at predetermined time intervals. For example, the infrared camera 14 (or another infrared receiving chip) of the first HMD 10 receives the infrared light from the sensor bars 55, and can calculate the distance and angle from the sensor bars 55 to calculate the position. The same applies to the second HMD 30. The controllers 18 and 38 are equipped with an infrared receiving chip that receives the infrared light from the sensor bars 55, and detect their positions in the same manner. Note that the method of detecting the positions of the first HMD 10, the second HMD 30, and the controllers 18 and 38 is not limited to this, and their positions may be detected by other methods. For example, one or more cameras may be installed in real space, the first HMD 10, the second HMD 30, and the controllers 18, 38 may be equipped with light-emitting units that emit a predetermined light, and the real-space positions of the first HMD 10, the second HMD 30, and the controllers 18, 38 may be detected by receiving light from the light-emitting units with a camera installed in real space.
[0092] The detected positions of the first HMD 10, the second HMD 30, and the controllers 18, 38 may be used for game control. For example, in the above, the position of the second HMD 30 relative to a reference (marker) in real space is detected by capturing an image of the marker 40 with the camera 33 provided in the second HMD 30, but the position of the second HMD 30 relative to a reference (sensor bar) in real space may be detected by the method shown in Fig. 9, and the detected position may be used for game control as described above. In this case, by determining the positional relationship between the marker in real space and the sensor bar, the position of the second HMD 30 in the XYZ coordinate system based on the marker can be calculated.
[0093] Also, for example, it may be possible to detect whether the controller 18 has been swung by detecting the position in real space of the controller 18. When the controller 18 has been swung, a bullet object 112 may be fired.
[0094] Information on the position and attitude of each virtual object (characters 110, 120a, 120b, 130, virtual cameras 111, 112a, 112b, 131, etc.) shown in FIG. 7 is transmitted to the information processing device 60 via a network and managed in the information processing device 60. For example, the smart device 20a detects its own position and attitude in the real space, and sets the position and attitude of the character 120a in the virtual space so as to match the detected position and attitude. The smart device 20a transmits information on the set position and attitude of the character 120a to the information processing device 60. In addition, position information and the like of virtual objects (141 to 145, etc.) related to the terrain arranged in the virtual space 100 are managed in the information processing device 60. In addition, operation information corresponding to operations on the controllers 18 and 38 is transmitted to the information processing device 60.
[0095] Game control processing is performed in the information processing device 60 based on information about each virtual object and operation information of the controllers 18, 38. As a result of this game control processing, an image of the virtual space is displayed in the first HMD 10 and the second HMD 30. In addition, the smart device 20 acquires information about each virtual object from the information processing device 60, performs game processing in its own device, and displays an image based on the result on the display 22. An example of an image displayed on each device will be described below.
[0096] (Example of image displayed on each device) 10 is a diagram showing an example of an image displayed on the display 12 of the first HMD 10, and is an example of an image when the first user faces forward. An image of the virtual space 100 viewed from the virtual camera 111 is displayed on the display 12 of the first HMD 10. Note that, although one image is shown two-dimensionally in FIG. 10, in reality, images having parallax with respect to each other are displayed on the right-eye display 12R and the left-eye display 12L, respectively, and the first user recognizes it as a stereoscopic image. The field of view of the first user is almost entirely covered by the display 12 of the first HMD 10.
[0097] As shown in FIG. 10, the display 12 of the first HMD 10 displays an image of looking down on the entire terrain including the character 130 from the top of a hill object 145. The first user sees a virtual space spreading out in front of him, and feels as if he is standing on the top of the hill object 145. A part of the right hand of the character 110 is visible in the field of view of the first user, and the right hand appears to be holding a bullet object 112. The size of the right hand of the character 110 as seen by the first user is similar to the size of the right hand of the first user when the first user sees it in real space. That is, the scale of the virtual space as seen by the first user is approximately the same as the scale of the real space. The posture of the right hand of the character 110 is the same as the posture of the controller 18 in real space, and when the first user raises the right hand holding the controller 18, the right hand of the character 110 is also raised.
[0098] From the first user's perspective, character 130 appears to be located on the right side in the depth direction on ground object 141. From the first user's perspective, character 130 appears to be present, for example, several tens of meters away. Furthermore, character 120a appears to be in the upper left direction, away from ground object 141. Character 120a appears to be floating in the virtual space.
[0099] When the first user turns to the right in real space from the state shown in Fig. 10, an image as shown in Fig. 11 is displayed. Fig. 11 is a diagram showing an example of an image displayed on the display 12 of the first HMD 10, and is an example of an image when the first user turns to the right.
[0100] As shown in Fig. 11, when the first user faces to the right, the virtual camera 111 and the character 110 in the virtual space also face to the right, and for example, the character 130 is displayed almost directly in front. Also, the character 120b that was not displayed in Fig. 10 is displayed in the upper right corner in Fig. 11. Also, the character 120a that was displayed in Fig. 10 is no longer displayed in Fig. 11.
[0101] Here, the first user attacks the character 130 using the bullet object 112. For example, when the first user swings the controller 18 held by the first user, the character 110 performs an action of swinging its right arm in the virtual space, and the bullet object 112 is shot into the virtual space. Note that the bullet object 112 may be shot into the virtual space in response to an operation on the button 181 of the controller 18.
[0102] Fig. 12 is a diagram showing an example of a state in which bullet object 112 is shot into virtual space in the state shown in Fig. 10. As shown in Fig. 12, for example, when the first user swings controller 18, bullet object 112 is shot into virtual space and moves within the virtual space. The moving direction of bullet object 112 may be determined by, for example, the direction in which controller 18 is swung by the first user or the direction indicated by analog stick 182 or cross key 183. When bullet object 112 hits character 130, the vitality value of character 130 decreases or character 130 falls down.
[0103] FIG. 13 is a diagram showing an example of an image displayed on the display 22 of the smart device 20a, and is an example of an image when the second user a is in the position shown in FIG. 1. The display 22 of the smart device 20a displays an image of the virtual space 100 viewed from the virtual camera 121a. Specifically, the display 22 displays an image of the virtual space generated based on the virtual camera 121a superimposed on an image of the real space captured by the camera 23 of the smart device 20a. The display 22 displays an image in which the virtual space is arranged on a table in the real space. For example, in FIG. 13, each image (141 to 145, 110, 130) on the table is an image of the virtual space, and an image of the periphery of the table including the image of the second user b is an image of the real space. Note that in the example of FIG. 13, the smart device 20b is not included in the imaging range of the camera 23 of the smart device 20a, so the character 120b is not displayed on the display 22 of the smart device 20a. When the smart device 20b is included in the imaging range of the camera 23 of the smart device 20a, the character 120b is displayed at the position of the smart device 20b superimposed on the image of the second user b. Alternatively, when the image of the real space captured by the camera 23 is not displayed on the display 22 of the smart device 20a, only the character 120b may be displayed at the position of the smart device 20b when the smart device 20b is included in the imaging range of the camera 23 of the smart device 20a.
[0104] In this way, the second user a uses the smart device 20a to view the small virtual space formed on the table from the position of the second user a shown in FIG. 1 (the position of the virtual camera 121a shown in FIG. 7) as if looking down on it.
[0105] When the second user a changes the position or attitude of the smart device 20a from the state shown in FIG. 13, the image displayed on the display 22 of the smart device 20a also changes.
[0106] FIG. 14 is a diagram showing an example of an image displayed on the display 22 of the smart device 20a after the second user a moves closer to the third user. When the smart device 20a moves in the real space, the virtual camera 121a also moves in the virtual space. When the second user a moves closer to the third user, the camera 23 of the smart device 20a acquires an image of a table including the marker 40 as seen obliquely, and the position of the virtual camera 121a is set based on the acquired image. As shown in FIG. 14, the image of the virtual space seen from the virtual camera 121a after the movement is superimposed on the image of the real space seen from the camera 23 after the movement, and is displayed on the display 22.
[0107] Here, the second user can eject a virtual object into the virtual space to change the character 130 in response to an operation on the smart device 20. Specifically, the second user can eject a virtual object into the virtual space to support the character 130 (third user).
[0108] FIG. 15 is a diagram showing an example of a state in which a character 120a shoots a bullet object 122 in response to an operation on the smart device 20a.
[0109] For example, a touch panel is provided on the display 22 of the smart device 20a. In response to a touch operation (for example, a tap operation, a swipe operation, or the like) on the touch panel of the smart device 20a, a bullet object 122 is shot from the character 120a. This bullet object 122 is a virtual object for supporting the character 130 (third user). Specifically, the bullet object 122 is an object for preventing the bullet object 112 shot by the character 110 (first user) from hitting the character 130. For example, when the bullet object 122 hits the bullet object 112, the bullet object 112 disappears and the attack on the character 130 becomes ineffective.
[0110] The second user may assist the third user by another method. For example, an obstacle object for blocking the bullet object 111 from the character 110 may be placed in the virtual space in response to the operation of the second user. The obstacle object may prevent the bullet object 111 from the character 110 from hitting the character 130. Also, for example, the character 120 may shoot a recovery object for recovering the vitality value of the character 130 in the virtual space in response to the operation of the second user. The recovery object shot from the character 120 is placed in the virtual space. When the character 130 moves to a position where the recovery object is placed and acquires the recovery object, the vitality value of the character 130 is recovered. In this way, the second user may assist the third user. Also, the character 120 may attack the character 110 by shooting a bullet object in response to the operation of the second user. When the bullet object from the character 120 hits the character 110, the vitality value of the character 110 may decrease, or the attack by the character 110 on the character 130 may be stopped. In this way, the second user may assist the third user.
[0111] In this way, the second user can participate in the combat game played between the first user and the third user by supporting the character 130 (third user).
[0112] Conversely, the second user may support the first user instead of supporting the third user. For example, the character 120 may launch an object for attacking the character 130 into the virtual space. The character 120 may also place a recovery object for recovering the physical strength of the character 110 in the virtual space. The character 120 may also place an obstacle object for defending against an attack from the character 130 to the character 110 in the virtual space.
[0113] That is, the character 120 may place a predetermined object in the virtual space that is advantageous or disadvantageous to the character 130 (or the character 110) in the game. The character 120 uses the predetermined object to affect the character 110. The character 120 is a character that affects the virtual space or the character 110 according to an input from the second user.
[0114] Also, the second user may participate in the game on an equal footing with the first and third users. That is, a game may be played in which each of the first, second, and third users (characters) attacks and defeats the other users.
[0115] Alternatively, the game is not limited to a game in which each character fights, and other games in which each character acts in a virtual space may be played. For example, a game in which each user's character cooperates with others to go on an adventure, or a racing game in which each user's character races may be played.
[0116] 16 is a diagram showing an example of an image displayed on the display 32 of the second HMD 30, and is an example of an image when the virtual camera 131 is set to the first mode. The image displayed on the display 32 of the second HMD 30 is also a stereoscopic image, similar to the first HMD 10.
[0117] As described above, when the virtual camera 131 is set to the first mode, the virtual camera 131 is disposed according to the position of the second HMD 30 based on the marker 40 in the real space. For example, when the second HMD 30 is disposed at a position overlooking the table (for example, the position shown in FIG. 1), the virtual camera 131 is disposed at a position overlooking the virtual space on the table (for example, the position of 130 shown in FIG. 7). In this case, as shown in FIG. 16, the display 32 of the second HMD 30 displays each virtual object (character 130, character 110, virtual objects 141 to 145) in the virtual space formed on the table. In addition, when the character 120b located at a position away from the table is included in the field of view of the virtual camera 131, the character 120b is also displayed. When the position or attitude of the second HMD 30 changes, the position or attitude of the virtual camera 131 also changes accordingly.
[0118] When character 110 fires bullet object 112 into the virtual space, an image of bullet object 112 moving within the virtual space is displayed. When character 120b or character 120a fires bullet object 122 into the virtual space, an image of bullet object 122 moving within the virtual space is displayed.
[0119] On the display 32 of the second HMD 30, an image of the virtual space generated based on the virtual camera 131 is displayed superimposed on an image of the real space captured by the camera 33 of the second HMD 30. When the virtual camera 131 is set to the first mode, it appears to the third user that a small virtual space is formed on the table in the real space. That is, when the first mode is set, the third user views the small virtual space formed on the table as if overlooking it, similar to the second user of the smart device 20. Note that the image of the real space may not be displayed on the display 32 of the second HMD 30, and only the image of the virtual space generated based on the virtual camera 131 may be displayed.
[0120] The third user can use the controller 38 to switch between the first mode and the second mode.
[0121] FIG. 17 is a diagram showing an example of an image displayed on the display 32 of the second HMD 30, and is an example of an image when the virtual camera 131 is set to the second mode.
[0122] As shown in Fig. 17, an image of the virtual space viewed from the character 130 is displayed on the display 32 of the second HMD 30. When set to the second mode, the virtual camera 131 moves in accordance with the movement of the character 130. Note that Fig. 17 shows an image in which the virtual camera 131 is placed at the position of the character 130, but when the virtual camera 131 is set at a predetermined position behind the character 130, for example, an image of the virtual space viewed from behind the character 130 is displayed.
[0123] When the virtual camera 131 is set to the second mode, the third user sees a virtual space spreading out in front of him / her, and feels as if he / she is standing on the ground object 141. For the third user, it appears as if a bridge object 144 exists several tens of meters away, for example.
[0124] The third user moves the character 130 in the virtual space by operating the controller 38 while viewing an image such as that shown in Fig. 16 or 17. For example, the third user moves the character 130 in response to an operation of an analog stick or a cross key of the controller 38. Furthermore, when the character 130 reaches the position of the character 110, the third user makes the character 130 attack the character 110 in response to an operation of a button of the controller 38.
[0125] The actions of the first user, the second user, and the third user and the operation on the controller are reflected in the virtual space in real time. For example, when the orientation of the first user (first HMD 10) changes, the orientation of the character 110 changes in the virtual space in response to the change in orientation. When the first user swings the controller 18, the character 110 performs an action of throwing a bullet object 112. When the character 110 is included within the imaging range of the virtual camera 121 or the virtual camera 131, the smart device 20 or the second HMD 30 displays the state in which the orientation of the character 110 changes and the state in which the character 110 throws the bullet object 112. This allows the second user or the third user to grasp the action of the first user.
[0126] Similarly, when the orientation or position of the second user (smart device 20) changes, the orientation or position of the character 120 changes in the virtual space in response to the change. When the second user performs a predetermined operation on the smart device 20, the character 120 performs an action of throwing a bullet object 122. When the character 120 is included within the imaging range of the virtual camera 111 or the virtual camera 131, the first HMD 10 or the second HMD 30 displays the manner in which the orientation or position of the character 120 changes and the manner in which the character 120 throws the bullet object 122. This allows the first user or the third user to grasp the action of the second user.
[0127] As described above, in this embodiment, the first user wears the first HMD 10, and the second user carries the smart device 20. Regardless of the position of the first HMD 10 in the real space, the character 110 is placed in the virtual space, and the virtual camera 111 is placed at a position corresponding to the position of the character 110. The attitude of the character 110 changes according to the attitude of the first HMD 10, and the character 110 moves according to the operation on the controller 18 (for example, throws a bullet object 112). In addition, the virtual camera 121 is placed at a position in the virtual space according to the position of the smart device 20 relative to the marker 40 in the real space, and the attitude of the virtual camera 121 is set according to the attitude of the smart device 20 in the real space. Specifically, the XYZ coordinate system for defining the position of the virtual space is set with the marker 40 in the virtual space as a reference, so that the real space and the virtual space are associated with each other. The position of the smart device 20 with the marker 40 as a reference is calculated, and the character 120 and the virtual camera 121 are set at the calculated position. Then, an image of the virtual space generated based on the virtual camera 111 is displayed on the display 12 of the first HMD 10, and an image of the virtual space generated based on the virtual camera 121 is displayed on the display 22 of the smart device 20.
[0128] As a result, the first user sees the virtual space from the position of the character 110 in the virtual space (or a position at a predetermined distance from the character 110), and sees the virtual space from a subjective viewpoint. Therefore, the first user can enter the virtual space and play the game with the same sense of scale as the real space. In addition, when the virtual camera 131 is set to the second mode, the third user also sees the virtual space from the viewpoint of the character 130 (or a position at a predetermined distance from the character 130). In contrast, the second user sees the virtual space associated with the real space from the position of the smart device 20 relative to the marker 40 in the real space. The second user can play the game while being aware of the real space and the virtual space, as if a virtual object in the virtual space exists in the real space, and can play the game while overlooking the virtual space formed on the table. For the second user, it feels like the virtual space is formed on a small scale on the table. In this way, the first user who sees the virtual space from a subjective viewpoint and the second user who sees the virtual space from a bird's-eye viewpoint can play the game. Each user can view the virtual space from a different viewpoint and at a different sense of scale, allowing multiple users to play a new, unprecedented game.
[0129] In this embodiment, the marker 40 is placed on the table, a virtual space is formed on the table, and a game is played in the virtual space. A first user enters the virtual space formed on the table and plays a game, while a second user can play the game while overlooking the small virtual space formed on the table. This allows multiple users to play a new game that has never been seen before.
[0130] In this embodiment, the first user to the third user can see the movements of the characters corresponding to the other users by looking at the display of their own devices, and can recognize the movements of the other users. For example, the first user can recognize the movements of the second user by looking at the movements of the character 120 displayed on the display 12 of the first HMD 10. Since the position of the character 120 in the virtual space is set in conjunction with the position of the smart device 20 in the real space, the first user can grasp the position of the second user in the real space by looking at the character 120 in the virtual space. In addition, the second user can recognize the movements of the first user by looking at the display of his / her own smart device 20, such as the change in the posture of the character 110 and the action of the character 110 throwing the bullet object 112.
[0131] Basically, when a user wears an HMD, other users cannot know what kind of image the user wearing the HMD is watching or what he / she is doing in the virtual space. However, in the system of this embodiment, the second user can see the state of the virtual space and the first user in the real space through the smart device 20, so that the second user can know what kind of image the first user is watching and what he / she is doing in the virtual space. That is, the second user can visually recognize the movement of the character 110 (first user) in the virtual space by looking at the smart device 20, and can see the movement of the first user by looking at the real space. This allows the second user to enter the virtual space where the user wearing the HMD is and see the virtual space.
[0132] Here, an image of the virtual space seen from the virtual camera 150 is displayed on the stationary display 50. Also, an image seen by the user wearing the HMD may be displayed on the stationary display 50. The fourth user can see the image displayed on the display 50 and grasp the state of the virtual space in which the user wearing the HMD is located. The second user can also grasp the state of the virtual space in which the user wearing the HMD is located through the smart device 20, but the second user can not only see the inside of the virtual space, but can also have a new experience that has not been seen before using AR technology. That is, since the virtual space is formed corresponding to the real space, the second user can move or change direction in the real space to peek into the virtual space from the same viewpoint as the second user's viewpoint in the real space. This allows the second user to see the virtual space in which the first user is present as if he or she were looking at the real space.
[0133] Note that the first user usually plays the game without moving away from the mat 51. However, when the first user leaves a predetermined range based on the mat 51 or when the first user moves a predetermined distance or more, a notification is given to the first user. For example, an image or text may be displayed on the display 12 of the first HMD 10, or a voice may be displayed, or a vibration may be displayed. When the first user leaves the predetermined range or when the first user moves a predetermined distance or more, a notification is given to the second user. For example, an image or text may be displayed on the display 22 of the smart device 20, or a voice may be displayed, or a vibration may be displayed. Similarly, a notification may be given to the third user and the fourth user. When the second user enters or approaches the predetermined range, a notification is given to the first user and the second user using an image, text, voice, vibration, or the like.
[0134] For example, the image processing system 1 detects the position of the first HMD 10 by the method shown in FIG. 9. The image processing system 1 determines whether the first user has left the predetermined range or whether the first user has moved a predetermined distance or more based on the position of the first HMD 10. Alternatively, the first user may be captured by the camera of the smart device 20 or the second HMD 30, or other cameras arranged in the real space, and based on the captured image, it may be determined whether the first user has left the predetermined range or whether the first user has moved a predetermined distance or more. Also, the camera of the first HMD 10 may capture an image of the periphery of the first user, and based on the captured image, it may be determined whether the first user has left the predetermined range or whether the first user has moved a predetermined distance or more. Also, based on images captured by the first HMD 10, the smart device 20, the second HMD 30, or other cameras, it may be determined whether the second user has entered the predetermined range or approached the predetermined range.
[0135] An image showing the predetermined range may be displayed on the display 22 of the smart device 20. That is, when the camera 23 of the smart device 20 captures an image of a predetermined range based on the mat 51, a range image showing the predetermined range may be displayed superimposed on the predetermined range.
[0136] A predetermined range may also be set for the third user. When the third user leaves the predetermined range or moves a predetermined distance or more, a notification may be given to the third user, the second user, and the first user. In this case, the camera 33 of the second HMD 30 may capture an image of the marker 40, and the virtual camera 131 may be positioned according to the position of the second HMD 30 relative to the marker 40 in the real space. That is, the third user experiences AR in the same way as the second user. When the third user leaves the predetermined range or moves a predetermined distance or more, a notification may be given to the third user (notification using the display 32, speaker, vibration motor, etc.) and also to the second user (notification using the display 22, speaker, vibration motor, etc.).
[0137] (Creating a virtual space) In this embodiment, in addition to the game mode in which the above-mentioned game is played, there is a creation mode in which the user creates a virtual space for playing the above-mentioned game. The following describes how the user creates a virtual space in the creation mode. Figure 18 is a diagram showing an example of an image viewed by the user in the creation mode.
[0138] In the virtual space creation mode, the user wears the first HMD 10, for example, and places virtual objects in the virtual space using the controller 18 while looking at the display 12 of the first HMD 10. A marker 40 is placed on a table, and an image of the marker 40 on the table is captured by the camera 13 of the first HMD 10, and an image such as that shown in FIG.
[0139] In the creation mode, a virtual camera is placed at a position in the virtual space that corresponds to the position of the first HMD 10 relative to the marker 40 in real space. The user creates a virtual space while looking at an image seen from the virtual camera.
[0140] Specifically, a left hand object 160L that imitates the user's left hand and a right hand object 160R that imitates the user's right hand are displayed on the display 12 of the first HMD 10. The left hand object 160L and the right hand object 160R are virtual objects. The user's hands are detected based on an image captured by the infrared camera 14 of the first HMD 10. The user's hands are detected using a known image recognition technique. When the user's left hand and right hand are detected, the left hand object 160L and the right hand object 160R are placed at the positions of the detected left hand and right hand.
[0141] The user's hand may be detected based on an image from the camera 13 of the first HMD 10. The user's hand may be captured and detected by an infrared camera or a camera other than the camera included in the first HMD 10.
[0142] For example, when the user turns the palm of his / her left hand toward the first HMD 10, a UI object 170 is displayed superimposed on the left hand object 160L or in the vicinity of the left hand object 160L. The UI object 170 is a user interface for creating a virtual space, and includes a plurality of part objects for placement in the virtual space. For example, the UI object 170 includes a part object 171 for forming a ground object 141 whose surface is covered with grass, a part object 172 for forming a river object 143, and a part object 173 for forming blocks. The UI object 170 may also include a plurality of part objects for forming the topography of the virtual space.
[0143] Each part object is formed of, for example, a cube, and a plurality of part objects are arranged in a grid pattern to form the terrain of the virtual space. For example, a ground object 141 covered with grass is formed by arranging a plurality of part objects 171 in a grid pattern. For example, a green texture image is attached to the upper surface of the part object 171, and a plurality of these part objects 171 are arranged in a plane to form the ground object 141 with the upper surface covered with grass.
[0144] Further, a controller object 161 simulating the controller 18 is displayed on the right hand object 160R. An image is displayed in which the right hand object 160R is holding the controller object 161, as if the controller 18 were held in the right hand in real space. For example, the controller 18 may be detected based on an image captured by the camera 13 of the first HMD 10, and the controller object 161 may be displayed when the controller 18 is detected. Alternatively, the controller 18 may not be detected, and the controller object 161 may be displayed together with the right hand object 160R when the right hand of the user is detected.
[0145] The controller object 161 is a virtual object that has substantially the same appearance as the controller 18, and has portions that imitate a number of buttons, just like the controller 18. The controller object 161 also has portions that imitate an analog stick and a cross key.
[0146] A pointing object 162 for pointing to a part object in the UI object 170 is provided at a predetermined position of the controller object 161. The pointing object 162 is an object that protrudes from a predetermined position of the controller object 161, and when the tip of the pointing object is brought close to the part object, it is possible to point to the part object.
[0147] For example, when the user moves the tip of the instruction object 162 close to the part object 173 while looking at the display 12 of the first HMD 10, the part object 173 is instructed. In this state, when the user presses, for example, a predetermined button on the controller 18, the part object 173 is in a selected state (a state in which the part object 173 is grasped). As shown in FIG. 18, the selected part object is displayed in a display mode different from other unselected objects. Furthermore, characters indicating the selected part object (the characters "Block" in FIG. 18) are displayed on the UI object 170. When the user moves the right hand onto the table with the part object 173 selected and performs a predetermined operation (for example, an operation of releasing the finger from the button), the part object 173 is placed on the table (on the XZ plane). The user grasps the part objects one by one and places them on the table, and uses the controller 18 to arrange the multiple part objects in the virtual space, thereby forming a terrain in the virtual space on which the character is arranged.
[0148] Whether or not a part object has been selected may be determined based on the movement of the fingers of the user's right hand. For example, whether or not the user has bent his / her finger may be determined, and if the user has bent his / her finger, the designated part object may be selected.
[0149] In this way, when the user's right hand is detected, the controller object 161 is displayed in the right hand object 160R. Also, the pointing object 162 is displayed at a predetermined position of the controller object 161. By arranging the controller object 161, which has substantially the same appearance as the controller 18 held by the user, in the virtual space, the user wearing the first HMD 10 can see an image similar to the controller 18 actually held, and it becomes easier to arrange the part object in the virtual space. Also, since the pointing object 162, which does not exist in the real space, is displayed in the virtual space, it becomes easier for the user wearing the first HMD 10 to indicate a position using the pointing object 162, and the user can easily select and arrange the part object.
[0150] When a user selects and arranges a part object, a plurality of selected part objects may be arranged in the virtual space. For example, when a user moves his / her right hand parallel to the surface of a table with one part object selected, a plurality of selected part objects may be arranged in a plane.
[0151] Fig. 19 is a diagram showing a state in which a plurality of part objects 173 are arranged in a plane. As shown in Fig. 19, for example, when a user selects a part object 173 and moves his / her right hand in a semicircle approximately parallel to the table, the part objects 173 may be arranged in a plane in an area surrounded by the trajectory of the right hand. Furthermore, when the user performs a similar motion, a plurality of part objects 173 may be further arranged in a plane on the plurality of part objects 173 arranged in a plane.
[0152] Also, a grid may be displayed in the virtual space to facilitate the user to place the part object. FIG. 20 is a diagram showing an example of grid lines for displaying the grid on which each part object is placed. As shown in FIG. 20, grid lines 175 may be displayed when creating the virtual space. The grid lines 175 include lines parallel to the X axis, lines parallel to the Y axis, and lines parallel to the Z axis. One part object is placed in an area surrounded by the grid lines 175. For example, when the part object 171 is selected and the tip of the instruction object 162 enters an area surrounded by the grid lines 175, the part object 171 is placed in that area. Note that the part object may be placed in a floating state (at a position away from the XZ plane) without any other object below.
[0153] By arranging the part objects 171 for each grid in this manner, the part objects can be easily arranged in the virtual space even if the user does not specify the positions of the part objects accurately.
[0154] (Arrangement of real objects) In the image processing system 1 of this embodiment, it is possible to recognize an actual object in real space and place a virtual object corresponding to the actual object in virtual space. Fig. 21 is a diagram showing how a can 80 as an actual object is placed on a table in real space. Fig. 22 is a diagram showing how a virtual object 180 corresponding to the can 80 is placed in virtual space.
[0155] For example, when an image including a can 80 and a marker 40 is acquired from the camera 13 of the smart device 20, the smart device 20 detects the can 80 based on the image. The smart device 20 has learned a plurality of real objects in advance, and can recognize the real objects based on the image acquired from the camera 13. For example, when the smart device 20a recognizes the can 80 from the acquired image, the can object 180 is placed at a position in the virtual space corresponding to the position of the can 80 in the real space. Information on the placement of the can object 180 and on its position and orientation is transmitted to the information processing device 60 and stored in the information processing device 60. As a result, the placement of the can object 180 in the virtual space and its position and orientation are shared with each device of the image processing system 1.
[0156] After can object 180 is placed in the virtual space, an image of the virtual space including can object 180 is acquired based on each virtual camera. For example, an image of the virtual space including can object 180 is displayed on display 12 of first HMD 10. In addition, an image of the virtual space including can object 180 is also displayed on display 32 of second HMD 30.
[0157] Also, an image including the can object 180 is displayed on the display 22 of the smart device 20, superimposed on the image of the real space. Specifically, when the camera 23 of the smart device 20 captures an image including the can 80 and the can object 180 is placed in the virtual space, the image of the can object 180 is displayed on the display 22 of the smart device 20, superimposed on the image of the can 80. Therefore, the second user visually recognizes the image of the can object 180 as a virtual object through the display 22 of the smart device 20, rather than the image of the can 80 as a real object.
[0158] Note that an image of can 80 as an actual object captured by camera 23 may be displayed on display 22 of smart device 20, instead of an image of can object 180. In this case, for the image of can 80, an image of the virtual space is not superimposed on the image of the real space captured by camera 23, and for other parts different from the image of can 80 (for example, the area of ground object 141 or the character), an image of the virtual space is superimposed on the image of the real space captured by camera 23.
[0159] For example, if smart device 20a detects can 80 in the real space and places can object 180 in the virtual space, and then the position or orientation of smart device 20a is changed so that can 80 moves out of the imaging range of camera 23, can object 180 remains placed at that position. In other words, even if can object 180 is not moved after it is placed in the virtual space and can 80 is no longer detected by smart device 20a, can object 180 continues to exist in the virtual space.
[0160] Specifically, when smart device 20a detects can 80 and places can object 180 in the virtual space, it continues to detect can 80 thereafter. Information on the position and orientation of can 80 is transmitted to information processing device 60 in real time. When the position or orientation of smart device 20a is changed without moving can 80, the position of can 80 in the image captured by camera 23 of smart device 20a changes, but the position of can 80 based on marker 40 does not change. Therefore, the position of can object 180 does not change. Even if can 80 moves out of the imaging range of camera 23 due to a change in the position or orientation of smart device 20a, the position of can object 180 does not change unless a change in the position of can 80 is detected.
[0161] Note that even if the can 80 falls outside the imaging range of the camera 23 of the smart device 20a, if the can 80 is included in the imaging range of the camera 23 of the smart device 20b, the smart device 20b continues to detect the can 80. Alternatively, an image may be captured by the camera 33 of the second HMD 30, and the can 80 may be detected. In this manner, for example, after the can 80 is detected by the smart device 20a and the can object 180 is placed, the position of the can 80 is tracked by devices other than the smart device 20a.
[0162] On the other hand, if the can object 180 is placed in the virtual space and then the can 80 is moved and removed from the table, the can 80 will no longer be detected by the smart device 20 and the can object 180 will be erased from the virtual space.
[0163] Specifically, when the can 80 moves, the can 80 is continuously detected by the smart device 20a or 20b, and the position of the can object 180 is updated. Alternatively, an image may be captured by the camera 33 of the second HMD 30, and the can 80 may be detected. When the can 80 moves, the position of the can object 180 is continuously updated until the can 80 is removed from the table, and when the position of the can object 180 exceeds a predetermined range (for example, when it moves outside the table), the can object 180 is erased from the virtual space.
[0164] Furthermore, after the can object 180 is placed at a predetermined position in the virtual space, if the position or orientation of the smart device 20 changes and the can 80 moves out of the imaging range of the camera 23, the can object 180 remains at the predetermined position. However, if the position or orientation of the smart device 20 returns to its original position and a real-space position corresponding to the predetermined position (the position where the can 80 was before it moved out of the imaging range) comes back within the imaging range of the camera 23, and the smart device 20 does not detect the can 80, the can object 180 is erased from the virtual space.
[0165] Furthermore, after can object 180 is placed at a first position in the virtual space, if the position or orientation of smart device 20 changes, causing the imaging range of camera 23 to change and can 80 to move out of the imaging range of camera 23, and can 80 is detected within the changed imaging range of camera 23, can object 180 may be placed at a second position in the virtual space corresponding to the newly detected position of can 80, and can object 180 may be erased from the first position. In other words, in this case, can object 180 may be instantaneously moved from the first position to the second position.
[0166] Also, for example, after can object 180 is placed at a first position in the virtual space, if the position or orientation of smart device 20 changes, causing the imaging range of camera 23 to change and can 80 to move out of the imaging range of camera 23, and if can 80 is detected within the changed imaging range of camera 23, a new can object 180 may be placed at a second position in the virtual space corresponding to the newly detected position of can 80, and can object 180 may be allowed to remain at the first position. In this case, after one can 80 is detected and the first can object 180 is placed in the virtual space, the same can 80 may be detected at another location and the second can object 180 may be placed in the virtual space.
[0167] Also, a condition regarding the number of can objects 180 to be placed in the virtual space may be set, and depending on the condition, when a can 80 is detected within the imaging range of the changed camera 23, control may be performed as to whether the can object 180 at the first position is erased or whether the can object 180 is allowed to remain at the first position. For example, it may be determined whether the number of can objects 180 to be placed in the virtual space is set to one or multiple, and control may be performed according to the determination result. For example, when the number of can objects 180 to be placed in the virtual space is set (limited) to one, when the can object 180 is placed at the first position in the virtual space and then the can 80 moves out of the imaging range of the camera 23, and the can 80 is detected within the imaging range of the changed camera 23, the can object 180 may be placed at a second position in the virtual space corresponding to the position of the detected can 80, and the can object 180 at the first position may be erased. Furthermore, in the case where the number of can objects 180 to be placed in the virtual space is set (allowed) to be multiple, if can object 180 is placed at a first position in the virtual space and then can 80 moves out of the imaging range of camera 23, and can 80 is detected within the changed imaging range of camera 23, can object 180 is placed at a second position in the virtual space according to the detected position of can 80, and can object 180 may also remain at the first position.
[0168] Furthermore, after can 80 is recognized and can object 180 is placed in the virtual space, can object 180 may remain in the virtual space even if can 80 is removed from the table. That is, after can object 180 is placed in the virtual space, can object 180 may remain in the virtual space even in a situation in which can 80 can be tracked by a camera to be removed from the table. Thereby, for example, a first specific object is placed on the table, and then the first specific object is removed and a second specific object is placed on the table, whereby a first virtual object corresponding to the first specific object and a second virtual object corresponding to the second specific object can be placed in the virtual space.
[0169] The game is controlled using can object 180 arranged in the virtual space. For example, can object 180 may be used to defend against an attack from character 110, and when bullet object 112 from character 110 hits can object 180, bullet object 112 may be eliminated. Also, another change may be added to the virtual space using can object 180. For example, when a user performs a predetermined operation on can object 180, a scene of liquid spurting out from can object 180 may be displayed.
[0170] In this way, in the image processing system 1 of this embodiment, a virtual object corresponding to a specific object in real space can appear in the virtual space, giving the first user the sensation that real space and virtual space are mixed together.
[0171] Although the above describes an example in which a specific object is captured by the camera 23 of the smart device 20, the specific object may be captured by a camera other than the camera 23 (for example, a camera arranged at a predetermined position in real space, a camera of the second HMD 30 or the first HMD 10). Then, the information processing device 60 or the smart device 20 may detect the specific object based on an image from the camera, and when the specific object is detected, a virtual object corresponding to the specific object may be placed in the virtual space.
[0172] (Data stored on each device) Next, data stored in each device will be described. Fig. 23 is a diagram showing an example of data D20 stored in each smart device 20.
[0173] 23, the smart device 20 stores a program for performing smart device processing, which will be described later. The smart device 20 also stores image data acquired from the camera 23, and angular velocity data and acceleration data from the inertial sensor 24. The smart device 20 also stores position data indicating the position of the smart device 20 relative to the marker 40, which is calculated based on the image data. The smart device 20 also stores attitude data indicating attitude information of the smart device 20, which is calculated based on the angular velocity data and the acceleration data.
[0174] Additionally, the smart device 20 stores character 120 data related to the character 120 corresponding to the user of the own device, and virtual camera 121 data indicating the position and attitude of the virtual camera 121. The character 120 data includes data indicating the position and attitude of the character 120 in the virtual space, data indicating the shape of the character 120, and data related to a virtual object (e.g., a bullet object 122) possessed by the character 120.
[0175] In addition, the smart device 20 stores other character data, terrain object data, projectile object data, and detected object data.
[0176] The other character data is data on each character corresponding to a user of a device other than the player's own device. The other character data is acquired from the information processing device 60. For example, the smart device 20a stores data on the characters 110, 120b, and 130 as the other character data. The other character data includes data indicating the position and posture of each character, and data on the shape of each character.
[0177] The terrain object data is data of objects (141-145) related to the terrain that forms the virtual space, and includes information on a plurality of part objects and their arrangement. The terrain object data is acquired from the information processing device 60. The terrain object data includes data created in advance by a game creator and stored in the image processing system 1, and data created by a user as described above and stored in the image processing system 1.
[0178] The projectile object data is data on a bullet object projected into a virtual space, and includes data on the position, moving speed, and moving direction of the bullet object. The projectile object data includes data on a bullet object 122 projected from the player's character 120, and data on a bullet object projected from a character corresponding to another device, which is obtained from the information processing device 60.
[0179] The detected object data is data on a virtual object (e.g., can object 180) that is placed in a virtual space when a predetermined specific object (e.g., can 80) is detected. The detected object data includes data on the shape, position, and orientation of the virtual object.
[0180] Fig. 24 is a diagram showing an example of data D10 stored in the first HMD 10. Fig. 25 is a diagram showing an example of data D30 stored in the second HMD 30.
[0181] As shown in FIG. 24, the first HMD 10 stores a program for performing the HMD processing described below, angular velocity data and acceleration data from the inertial sensor 15, and posture data regarding the posture information of the first HMD 10 calculated from the angular velocity data and acceleration data.
[0182] In addition, the second HMD 30 stores a program for performing HMD processing, angular velocity data and acceleration data from the inertial sensor 35, posture data regarding the posture information of the second HMD 30 calculated from the angular velocity data and acceleration data, and image data from the camera 33.
[0183] FIG. 26 is a diagram showing an example of data D60 stored in the information processing device 60. As shown in FIG.
[0184] 26, the information processing device 60 stores a program for performing a game control process described later, HMD data, smart device data, controller data, character data, and virtual camera data. The information processing device 60 also stores the terrain object data, projected object data, and detected object data.
[0185] The HMD data is data related to the first HMD 10 and the second HMD 30. Specifically, the HMD data includes posture data acquired from the first HMD 10. The HMD data also includes position data of the second HMD 30 calculated based on image data from the second HMD 30, and posture data acquired from the second HMD 30.
[0186] The smart device data includes data indicating the position and orientation of each smart device 20 acquired from each smart device 20, and data related to operations performed on each smart device 20.
[0187] The controller data is data related to each controller (18, 38). The controller data includes data related to operations performed on the buttons, sticks, cross keys, etc. of each controller. The controller data also includes data related to the posture and movement of each controller.
[0188] The character data is data about each character (110, 120, 130) and includes data about the position and posture of each character. The virtual camera data is data about each virtual camera (111, 121, 131, 150) and includes data about the position and posture of each virtual camera.
[0189] (Details of processing in HMD) Next, the details of the processing performed in each device will be specifically described. Fig. 27 is a flowchart showing an example of HMD processing executed in the first HMD 10 or the second HMD 30. The processing shown in Fig. 27 is started in response to an instruction to start a game being given in the image processing system 1. Note that, in the following, the first HMD 10 (its processor 11) will be described as executing the processing shown in Fig. 27, but the second HMD 30 also performs the same processing.
[0190] 27, the first HMD 10 performs an initial process (step S100). In the initial process, the posture of the first HMD 10 is initialized. After performing the initial process, the first HMD 10 repeatedly executes steps S101 to S105 at a predetermined time interval (for example, 1 / 60 second interval).
[0191] Next, the first HMD 10 acquires angular velocity data and acceleration data detected by the inertial sensor 15 (step S101). The first HMD 10 calculates posture information of the first HMD 10 based on the acquired angular velocity data and acceleration data (step S102). Specifically, the first HMD 10 calculates a change in posture from the time when the initial processing was performed. Next, the first HMD 10 transmits the calculated posture information to the information processing device 60 (step S103).
[0192] Next, the first HMD 10 acquires a left-eye virtual space image to be displayed on the left-eye display 12L and a right-eye virtual space image to be displayed on the right-eye display 12R from the information processing device 60 (step S104). Then, the first HMD 10 displays the acquired left-eye virtual space image and right-eye virtual space image on the left-eye display 12L and the right-eye display 12R, respectively (step S105).
[0193] In addition to the above processes, the second HMD 30 performs a process of acquiring an image captured by the camera 33 and a process of transmitting the acquired image to the information processing device 60.
[0194] (Details of processing on smart devices) Next, a process executed in the smart device 20 will be described. Fig. 28 is a flowchart showing an example of smart device process executed in the smart device 20. The process shown in Fig. 28 is performed by the processor 21 of the smart device 20. It is assumed that, before the process shown in Fig. 28 is started, a user issues an instruction to start a game, and necessary data (for example, terrain object data and other character data) is acquired from the information processing device 60.
[0195] As shown in FIG. 28, the smart device 20 acquires an image from the camera 23 (step S201), and acquires angular velocity data and acceleration data from the inertial sensor 24 (step S202).
[0196] Next, the smart device 20 performs a marker detection process (step S203). Specifically, the smart device 20 detects the marker 40 from the image acquired in step S201. If the smart device 20 does not detect the marker 40, it repeatedly executes the processes of steps S201 to S203.
[0197] When the marker 40 is detected, the smart device 20 performs a position calculation process (step S204). Specifically, the smart device 20 sets an XYZ coordinate system with the detected marker 40 as a reference, and calculates the position (XYZ coordinate value) of the smart device 20 relative to the marker 40. Specifically, the smart device 20 calculates the distance from the marker 40 to the smart device 20 and the direction from the marker 40 to the smart device 20 based on the size of the image of the marker 40, the direction of the arrow, etc., and calculates the relative position of the smart device 20 with the marker 40 as a reference.
[0198] Next, the smart device 20 performs an attitude calculation process (step S205). Specifically, the smart device 20 calculates the attitude of the smart device 20 based on the angular velocity data and acceleration data acquired in step S202. The smart device 20 may calculate the attitude based on an image from the camera 23 (detection result of the marker 40). The smart device 20 may also calculate the attitude based on the image from the camera 23 and the angular velocity data and acceleration data from the inertial sensor 24.
[0199] Next, the smart device 20 sets the position and attitude of the character 120 based on the position calculated in step S204 and the attitude calculated in step S205 (step S206). Also, the smart device 20 sets the position and attitude of the virtual camera 121 based on the position calculated in step S204 and the attitude calculated in step S205. For example, the smart device 20 sets the positions of the character 120 and the virtual camera 121 to match the positions calculated in step S204. Also, the smart device 20 sets the attitudes of the character 120 and the virtual camera 121 to match the attitudes calculated in step S205. Note that the position and attitude of the virtual camera 121 do not necessarily have to match the position and attitude of the character 120. For example, the position of the virtual camera 121 may be set to a predetermined position behind the character 120.
[0200] Next, the smart device 20 determines whether or not a predetermined operation has been performed on the operation unit (step S207). For example, the smart device 20 determines whether or not a swipe operation for firing the bullet object 122 has been performed on the touch panel.
[0201] When a predetermined operation is performed (step S207: YES), the smart device 20 launches the bullet object 122 into the virtual space (step S208). As a result, the bullet object 122 is launched into the virtual space, and the movement of the bullet object 122 is displayed.
[0202] When the process of step S208 is executed, or when the result of step S207 is NO, the smart device 20 judges whether or not a predetermined specific object is detected from the image acquired in step S201 (step S209). The smart device 20 has learned a plurality of specific objects in advance, and judges whether or not a specific object is present in the acquired image.
[0203] When a specific object is detected (step S209: YES), the smart device 20 places a virtual object corresponding to the detected specific object in the virtual space (step S210). Specifically, the smart device 20 calculates the position of the specific object from the positional relationship between the marker 40 and the specific object, and places the virtual object corresponding to the detected specific object at the same position in the virtual space as the calculated position. For example, when a can 80 is detected as the specific object, the smart device 20 places a can object 180 in the virtual space.
[0204] When the process of step S210 is executed or when the result of the determination in step S209 is NO, the smart device 20 transmits and receives information to and from the information processing device 60 (step S211). Specifically, the smart device 20 transmits the position calculated in step S204 and the attitude calculated in step S205 to the information processing device 60. In addition, the smart device 20 transmits information on the position and attitude of the character 120 and the virtual camera 121 set in step S206 to the information processing device 60. In addition, when the smart device 20 shoots the bullet object 122 into the virtual space in step S208, the smart device 20 transmits data on the shooting direction, shooting speed, and the like of the bullet object 122 as shot object data to the information processing device 60. In addition, when the smart device 20 places a virtual object corresponding to a specific object in step S210, the smart device 20 transmits data on the placed virtual object (type, position, and attitude of the virtual object) to the information processing device 60 as detected object data.
[0205] In addition, in step S211, the smart device 20 receives other character data from the information processing device 60. Specifically, the smart device 20 receives, as the other character data, information on the positions and postures of the character 110 corresponding to the first HMD 10, the character 120 corresponding to the other smart device 20, and the character 130 corresponding to the second HMD 30. As a result, the smart device 20 updates the other character data stored in the smart device 20. In addition, the smart device 20 receives, from the information processing device 60, ejection object data related to an ejection object ejected in response to an operation on the other device, and detection object data related to a virtual object corresponding to a specific object detected by the other device.
[0206] Following step S212, the smart device 20 performs other game processing (step S212). Here, the smart device 20 performs game processing according to the positions of each character and each projected object. For example, the smart device 20 determines whether the projected object 112 hits the character 130, and if so, reduces the vitality value of the character 130.
[0207] Next, the smart device 20 performs image generation and display processing (step S213). Specifically, the smart device 20 first generates an image of the virtual space based on the virtual camera 121. As a result, an image of the virtual space viewed from the virtual camera 121 is generated. For example, if the character 110 or the character 130 is included in the imaging range of the virtual camera 121, an image of the virtual space including the character 110 or the character 130 is generated. In addition, the smart device 20 generates a superimposed image by superimposing the generated image of the virtual space on the image of the real space acquired in step S201. Then, the smart device 20 causes the display 22 to display the generated superimposed image.
[0208] After the process of step S213, the smart device 20 returns the process to step S201. The processes of steps S201 to S213 are repeated at predetermined time intervals (for example, 1 / 60 second intervals) to progress the game.
[0209] (Details of processing in information processing device) 29 is a flowchart showing an example of a game control process executed in the information processing device 60. The process shown in FIG.
[0210] 29, information processing device 60 first performs an initial process in response to an instruction to start a game (step S600). In the initial process, various objects (141-145) related to terrain are placed in the virtual space, character 110 is placed on the top of hill object 145, character 130 is placed in an initial position, etc. Also, virtual cameras corresponding to each character are placed.
[0211] Next, the information processing device 60 acquires information from each device (step S601). Specifically, the information processing device 60 acquires the attitude information of the first HMD 10 transmitted from the first HMD 10. The information processing device 60 also acquires the attitude information of the second HMD 30 transmitted from the second HMD 30 and an image captured by the camera 33. The information processing device 60 also acquires the position information and attitude information calculated in each smart device 20. The information processing device 60 also receives shot object data from the smart device 20 when the bullet object 122 is shot in the smart device 20 (when the above step S208 is executed). The information processing device 60 also acquires detected object data from the smart device 20 when a specific object is detected in the smart device 20 and a virtual object corresponding to the specific object is placed (when the above step S210 is executed).
[0212] In addition, in step S601, the information processing device 60 acquires operation data corresponding to operations on the controllers 18, 38. The operation data includes data indicating whether a button has been pressed, data corresponding to operations on the analog stick, data corresponding to operations on the cross key, and angular velocity data and angular velocity data from the inertial sensor.
[0213] Next, the information processing device 60 calculates position information of the second HMD 30 (step S602). Specifically, the information processing device 60 detects the marker 40 based on the image from the second HMD 30 acquired in step S601, and calculates the position of the second HMD 30 relative to the marker 40 based on the detection result of the marker 40. Note that the information processing device 60 may calculate orientation information of the second HMD 30 based on the image from the second HMD 30 in addition to the position information of the second HMD 30.
[0214] Next, the information processing device 60 updates the position information and posture information of each character (step S603). Specifically, the information processing device 60 updates the position information and posture information of the character 120 corresponding to each smart device 20 according to the position information and posture information acquired from each smart device 20 in step S601. Also, the information processing device 60 updates the posture information of the character 110 according to the posture information acquired from the first HMD 10 in step S601. Also, the information processing device 60 moves the character 130 in the virtual space based on the operation data acquired from the controller 38 in step S601, and updates the position information of the character 130.
[0215] Next, the information processing device 60 updates the position information and orientation information of each virtual camera (step S604). Specifically, the information processing device 60 updates the position information and orientation information of each virtual camera 121 according to the position information and orientation information acquired from each smart device 20 in step S601. For example, the information processing device 60 sets the position and orientation of the virtual camera 121 so as to match the position and orientation of the character 120. In addition, the information processing device 60 updates the orientation information of the virtual camera 111 according to the orientation information acquired from the first HMD 10 in step S601. In addition, when the virtual camera 131 is set to the first mode, the information processing device 60 updates the position information and orientation information of the virtual camera 131 according to the position information and orientation information of the second HMD 30.
[0216] When virtual camera 131 is set to the second mode, the position information and posture information of virtual camera 131 are set according to the position information and posture information of character 130. For example, the position and posture of virtual camera 131 are set so as to match the position and posture of character 130.
[0217] Following step S604, the information processing device 60 controls each character based on the operation data (step S605). For example, the information processing device 60 causes the character 120 to fire a bullet object 122 based on the operation data from the smart device 20. Also, the information processing device 60 causes the character 110 to fire a bullet object 112 based on the operation data from the controller 18. Also, the information processing device 60 causes the character 130 to perform a predetermined action (for example, attack the character 110) based on the operation data from the controller 38.
[0218] Next, the information processing device 60 performs other game processing (step S606). For example, when the bullet object 112 is moving, the information processing device 60 moves the bullet object 112 in the virtual space and determines whether the bullet object 112 has hit the character 130. When the bullet object 112 has hit the character 130, the information processing device 60 decreases the vitality value of the character 130. Also, when the bullet object 122 has hit the character 110, the information processing device 60 decreases the vitality value of the character 110.
[0219] Next, the information processing device 60 performs a transmission process (step S607). Specifically, the information processing device 60 transmits the results of the processes of steps S603, S605, and S606 to each smart device 20. As a result, the position information and posture information of each character (110, 120, 130) are transmitted to each smart device 20. In addition, information that the character has shot a bullet object, and information regarding the shooting direction and shooting speed of the bullet object are transmitted to each smart device 20.
[0220] Following step S607, the information processing device 60 performs a first image generation / transmission process to display an image on the display 12 of the first HMD 10 (step S608). Specifically, the information processing device 60 generates a left-eye virtual space image based on the left virtual camera 111L, and generates a right-eye virtual space image based on the right virtual camera 111R. Then, the information processing device 60 transmits the generated left-eye virtual space image and right-eye virtual space image to the first HMD 10. As a result, the left-eye virtual space image and right-eye virtual space image based on the virtual camera 111 are displayed on the left-eye display 12L and right-eye display 12R of the first HMD 10, respectively.
[0221] Next, the information processing device 60 performs a second image generation / transmission process to display an image on the display 32 of the second HMD 30 (step S609). Specifically, the information processing device 60 generates a left-eye virtual space image based on the left virtual camera 131L, and generates a right-eye virtual space image based on the right virtual camera 131R. Then, the information processing device 60 transmits the generated left-eye virtual space image and right-eye virtual space image to the second HMD 30. As a result, the left-eye virtual space image and right-eye virtual space image based on the virtual camera 131 are displayed on the left-eye display 32L and right-eye display 32R of the second HMD 30, respectively.
[0222] Next, the information processing device 60 performs a third image generation / transmission process to display an image on the display 50 (step S610). Specifically, the information processing device 60 generates a virtual space image based on the virtual camera 150, and outputs the generated image to the display 50. As a result, the image based on the virtual camera 150 is displayed on the display 50.
[0223] As described above, in this embodiment, the character 110 is placed in the virtual space regardless of the position of the first HMD 10 in the real space. The virtual camera 111 is placed at a position corresponding to the position of the character 110 (S600). In addition, the virtual camera 121 is placed at a position in the virtual space according to the position of the smart device 20 relative to the marker 40 in the real space, and the attitude of the virtual camera 121 is set according to the attitude of the smart device 20 (S604). In addition, the character 120 is set at a position in the virtual space according to the position of the smart device 20 relative to the marker 40. Then, an image of the virtual space including the character 120 seen from the virtual camera 111 is displayed on the display 12 of the first HMD 10. In addition, an image of the virtual space including the character 110 seen from the virtual camera 121 is displayed on the display 22 of the smart device 20.
[0224] This allows the first user to play the game while viewing the virtual space from the viewpoint of the character 110 using the first HMD 10, and experience as if he or she were actually present in the virtual space. The second user can view the same virtual space using the smart device 20, and experience as if virtual objects in the virtual space were actually present in the real space.
[0225] (Modification) The image processing of this embodiment has been described above, but the above embodiment is merely an example, and the following modifications may be made, for example.
[0226] For example, in the above embodiment, the first user wears the first HMD 10 on his head and views images displayed on the left and right displays 12 of the first HMD 10. In other embodiments, the display 12 viewed by the first user is not limited to a head-mounted display, and may be a portable display held in the hand (e.g., a smartphone, a tablet terminal, a portable game device, a portable personal computer, etc.), or a stationary display (e.g., a television or a display of a stationary personal computer).
[0227] In the above embodiment, the second user holds the portable smart device 20 and visually recognizes the image displayed on the display 22 of the smart device 20. In other embodiments, as such a portable device, the portable display 22 visually recognized by the second user may be a display of a smartphone or a game device, a head-mounted display, or a glasses-type device equipped with a see-through display.
[0228] In the above embodiment, the character 110 is fixed at a predetermined position in the virtual space, the orientation of the character 110 is controlled according to the attitude of the first HMD 10, and the character 110 is made to perform a predetermined action (throwing the bullet object 112) according to a swing operation on the controller 18. In other embodiments, the position and attitude of the character 110 may be controlled, or the character 110 may be made to perform a predetermined action, according to an operation on the controller 18 (key input (input to a button, analog stick, or cross key) on the controller 18, swing operation on the controller 18, etc.). Also, the position of the character 110 may be controlled, or the character 110 may be made to perform a predetermined action, according to an operation on the first HMD 10. That is, the character 110 may be controlled in the virtual space according to an input to any input device including the first HMD 10 and the controller 18 (an inertial sensor of the first HMD 10, a key or inertial sensor of the controller 18, or another input device different from the first HMD 10 and the controller 18). Here, "control of the character 110" includes changes in the position and posture of the character 110, predetermined movements, and the like.
[0229] In the above embodiment, the position and orientation of the character 130 are controlled in response to key input to the controller 38. In other embodiments, the character 130 may be controlled in response to an operation on the controller 18 (such as a key input to the controller 38 or a swing operation on the controller 38). In other embodiments, the character 130 may be controlled in response to an input to an arbitrary input device. For example, the position of the character 130 may be controlled in response to an operation on the controller 38, and the orientation of the character 130 may be controlled in response to the attitude of the second HMD 30.
[0230] In the above embodiment, the character 120 is caused to perform a predetermined action (for example, throwing the bullet object 122) in response to an input to the smart device 20. In other embodiments, the character 120 may be controlled in response to an input to any input device including the smart device 20 (a touch panel, a button, an inertial sensor of the smart device 20, or another input device separate from the smart device 20).
[0231] In the above embodiment, the position of the smart device 20 relative to the marker 40 is calculated based on the image captured by the camera 23 of the smart device 20. In other embodiments, the position of the smart device 20 relative to the marker 40 may be calculated by another method. For example, the position of the smart device 20 relative to the marker 40 may be calculated by capturing images of the marker 40 and the smart device 20 with a camera disposed in a real space different from that of the smart device 20. In addition, the position of the smart device 20 in the real space may be calculated using the sensor bar 55, as described with reference to FIG. 9.
[0232] In addition, in the above embodiment, the position of the smart device 20 relative to the marker 40 is calculated, but in other embodiments, the position of the smart device 20 relative to a predetermined reference in real space, not limited to the marker 40, may be calculated. Here, the predetermined reference may be the marker 40, the sensor bar 55, another object, or a predetermined location. In addition, the "position of the smart device 20 relative to the predetermined reference" may be expressed by coordinate values.
[0233] In the above embodiment, the orientation of the first HMD 10 is calculated based on data from the inertial sensor 15 of the first HMD 10. In other embodiments, the orientation of the first HMD 10 may be calculated by other methods. For example, the first HMD 10 may be captured by a camera, and the orientation of the first HMD 10 may be calculated based on an image from the camera. The same applies to the second HMD 30. The orientation of the smart device 20 may also be calculated based on an image from the camera.
[0234] In addition, the processes shown in the above flowcharts are merely examples, and the order and contents of the processes may be changed as appropriate. Each process in the above flowchart may be executed by any of the first HMD 10, the smart device 20, the second HMD 30, and the information processing device 60.
[0235] For example, in the above embodiment, the first HMD 10 calculates the attitude based on the sensor data from the inertial sensor included in the first HMD 10, and transmits the calculated attitude information to the information processing device 60. In other embodiments, the information processing device 60 may acquire sensor data from the first HMD 10 and calculate the attitude of the first HMD 10. In the above embodiment, the first HMD 10 calculates only the attitude, and the information processing device 60 performs substantial processing related to the game (setting the attitude of the character 110, setting the attitude of the virtual camera 111, and generating images). In other embodiments, the first HMD 10 may set the attitudes of the character 110 and the virtual camera 111 and generate images. The same applies to the second HMD 30. In addition, a part or all of the processing performed by the smart device 20 (for example, the above S204 to S206, S207 to S208, S209 to S210, S211, S212, and S213) may be performed by the information processing device 60. In addition, a part or all of the above-described processes performed in the information processing device 60 may be executed in another device. For example, all of the processes of the information processing device 60 may be executed in the smart device 20.
[0236] The information processing device 60 may be configured by one or more devices. The information processing device 60 may be configured by multiple devices connected to a network (for example, the Internet).
[0237] In the above embodiment, the first HMD 10, the smart device 20, the second HMD 30, and the display 50 are located in one local location. In other embodiments, they may be located in remote locations and connected via a network. For example, the first HMD 10, the smart device 20, the second HMD 30, and the information processing device 60 may be connected via the Internet.
[0238] In the above embodiment, two images having a parallax between them are generated based on the left and right virtual cameras. In another embodiment, an image may be generated based on one virtual camera, and the generated image may be deformed to generate two images having a parallax between them.
[0239] In the above embodiment, a head mounted display fixedly attached to the user's head is used as the first HMD 10 and the second HMD 30. In other embodiments, a device that the user holds in their hands and peers into the left and right displays may be used as the first HMD 10 or the second HMD 30.
[0240] In addition, the configurations according to the above-described embodiments and their modifications can be combined in any manner as long as they are not inconsistent with each other. Furthermore, the above is merely an example of the present invention, and various improvements and modifications other than those described above may be made. [Explanation of symbols]
[0241] 1. Image Processing System 10 First HMD 12 Display 18 Controller 20. Smart Devices 22 Display 30 2nd HMD 40 Marker 50 Display 110 characters 111 Virtual Camera 120 characters 121 Virtual Camera 130 characters 131 Virtual Camera
Claims
1. A first device is a head-mounted display used by a first user and worn on the head or held in the hand; a second device used by a second user; a game processing means for performing game processing related to a game using a first virtual object operated by the first device and a second virtual object operated by the second device; a determination means for determining whether or not the first user has left a predetermined range in real space or whether or not the first user has moved a predetermined distance or more during the game processing; and a notification means for notifying the second user using the second device when the determination means determines that the first user has left the specified range or has moved more than the specified distance.
2. The information processing system according to claim 1 , wherein the second device is a portable device.
3. The information processing system according to claim 2 , wherein the game processing means performs the game processing based on a position of the second device in real space.
4. The information processing system according to claim 2 , wherein the second device is a head mounted display.
5. 5. An information processing system according to claim 1, wherein the notification means notifies the first user using the first device when the first user leaves the specified range or moves more than the specified distance.
6. An information processing system according to any one of claims 1 to 5, wherein the notification means notifies a third user watching the game using a third device when the first user leaves the specified range or moves more than the specified distance.
7. A first device is a head-mounted display used by a first user and worn on the head or held in the hand; a portable second device used by a second user; and a game processing means for performing game processing related to a game using a first virtual object operated by the first device and a second virtual object operated by the second device; a determination means for determining whether the second user has entered a predetermined range in real space during the game processing; a notification means for notifying the second user using the second device when the determination means determines that the second user has entered the specified range.
8. The information processing system according to claim 7 , wherein the game processing means performs the game processing based on a position of the second device in real space.
9. The information processing system according to claim 7 , wherein the notification means notifies the first user by using the first device when the second user enters the predetermined range.
10. 10. The information processing system according to claim 7, wherein the notification means notifies the first user by using the first device when the first user leaves the predetermined range.
11. The information processing system according to claim 7 , further comprising a display control unit that causes the second device to display an image showing the predetermined range.
12. An information processing method performed in an information processing system including a first device that is a head-mounted display that is worn on a head or held by a hand and is used by a first user, and a second device that is used by a second user, a game processing step of performing game processing for a game using a first virtual object operated by the first device and a second virtual object operated by a second device; a determining step of determining whether or not the first user has left a predetermined range in real space or whether or not the first user has moved a predetermined distance or more during the game processing; An information processing method comprising: a notification step of notifying the second user using the second device if it is determined in the determination step that the first user has left the specified range or if it is determined that the first user has moved more than the specified distance.
13. An information processing program executed on a computer of an information processing system including a first device, which is a head-mounted display worn on a head or held by a hand and used by a first user, and a second device, which is used by a second user, the information processing program comprising: a game processing means for performing game processing related to a game using a first virtual object operated by the first device and a second virtual object operated by a second device; a determination means for determining whether or not the first user has left a predetermined range in real space or whether or not the first user has moved a predetermined distance or more during the game processing; An information processing program that functions as a notification means that notifies the second user using the second device when the determination means determines that the first user has left the specified range or has moved more than the specified distance.
14. An information processing method performed in an information processing system including a first device which is a head-mounted display worn on a head or held by a hand and is used by a first user, and a portable second device which is used by a second user, a game processing step of performing game processing for a game using a first virtual object operated by the first device and a second virtual object operated by the second device; a determination step of determining whether or not the second user has entered a predetermined range in a real space during the game processing; a notification step of notifying the second user using the second device when it is determined in the determination step that the second user has entered the predetermined range.
15. An information processing program executed on a computer of an information processing system including a first device, which is a head-mounted display worn on a head or held by a hand and used by a first user, and a portable second device, which is used by a second user, the information processing program comprising: a game processing means for performing game processing related to a game using a first virtual object operated by the first device and a second virtual object operated by the second device; a determination means for determining whether the second user has entered a predetermined range in real space during the game processing; an information processing program that causes the information processing program to function as a notification means that notifies the second user using the second device when the determination means determines that the second user has entered the specified range.
Citation Information
Patent Citations
Information-processing device, information-processing system, and information-processing method
WO2015159561A1
Attitude position estimating system, attitude position estimating method, and attitude position estimating device
JP2019074362A
