Server device, terminal device, information processing system and information processing method

Abstract

[problem] To provide technology capable of reducing processing loads on a server device during cloud rendering. [Solution] The server device according to the present invention is provided with a control unit. The control unit groups terminal devices present within a predetermined region on the basis of pieces of positional information for the respective terminal devices capable of implementing AR display, generates an AR image shared by the terminal devices included in the group, and sends the AR image to the terminal devices.

Description

[Technical field]

[0001] The present technology relates to a technology for a server device or the like that performs cloud rendering. [Background technology]

[0002] In recent years, a service called cloud gaming has become widely known. In this cloud gaming, a server device on the cloud generates videos necessary for the progress of the game and transmits them to each terminal device. Therefore, even terminal devices with relatively low processing power such as smartphones can provide users with a complex and realistic gaming experience.

[0003] In the case of cloud gaming, the server device needs to generate individual videos for each terminal device, which results in a large processing load on the server device.

[0004] The following Patent Document 1 discloses a technology for reducing the processing load of a server device in cloud gaming by generating drawing commands in text format for each terminal device instead of moving images and transmitting them to each terminal device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-89507 A Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 does not reduce the processing load on the server device very much, and is therefore insufficient.

[0007] In view of the above circumstances, an object of the present technology is to provide a technology capable of reducing the processing load on a server device in cloud rendering. [Means for solving the problem]

[0008] A server device according to the present technology includes a control unit that groups terminal devices present within a predetermined area based on position information of each terminal device capable of displaying an augmented reality (AR) image, and generates and transmits a common AR image to the terminal devices included in the group.

[0009] In this server device, since it is sufficient to generate an AR image for each predetermined area (each group), it is possible to appropriately reduce the processing load of the server device in cloud rendering.

[0010] A terminal device according to the present technology includes a control unit that groups terminal devices present within a predetermined area based on location information of each terminal device capable of executing AR display, receives a common AR image from a server device that generates and transmits the common AR image to the terminal devices included in the group, and executes AR display based on the common AR image.

[0011] The information processing system according to the present technology includes a server device and a terminal device. The server device groups terminal devices present within a predetermined area based on the position information of each terminal device capable of executing AR display, and generates and transmits a common AR image to the terminal devices included in the group. The terminal device receives the common AR image and performs AR display based on the common AR image.

[0012] The information processing method related to the present technology groups terminal devices present within a specified area based on the respective location information of each terminal device capable of executing AR display, and generates and transmits a common AR image to the terminal devices included in the group. [Brief description of the drawings]

[0013] [Figure 1]FIG. 1 is a diagram illustrating an information processing system according to a first embodiment of the present technology. [Diagram 2] FIG. 2 is a block diagram showing an internal configuration of a server device. [Diagram 3] FIG. 2 is a block diagram showing the internal configuration of a terminal device. [Figure 4] 13 is a flowchart showing a process of the server device. [Diagram 5] 13 is a flowchart showing a process of the terminal device. [Figure 6] 13 is a flowchart showing a process of the terminal device. [Figure 7] FIG. 13 is a diagram showing a state when a point is set in a global coordinate system. [Figure 8] FIG. 13 is a diagram showing an example of a common AR image. [Figure 9] FIG. 13 is a diagram for explaining a flying object. [Figure 10] 13 is a flowchart showing a process of a terminal device on an information transmitting side; [Figure 11] 13 is a flowchart showing a process of a terminal device on an information receiving side; [Figure 12] FIG. 13 is a diagram showing how a flying object appears on a terminal device receiving information. [Figure 13] 13 is a diagram showing how an enemy virtual object is occluded by an occluding object in a first user and a first terminal device located northwest of the point. FIG. [Figure 14] 13 is a diagram showing how an enemy virtual object is occluded by an occluding object for a second user and a second terminal device located east of the point. FIG. [Figure 15] 13 is a diagram showing RGB information of an enemy virtual object, depth information of the enemy virtual object, and depth information of an occluding object. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0015] First Embodiment <Overview of Information Processing System 100> 1 is a diagram showing an information processing system 100 according to a first embodiment of the present technology. As shown in FIG. 1, the information processing system 100 includes a server device 10, a plurality of terminal devices 20, and an obstructing object database 30.

[0016] The information processing system 100 according to the first embodiment of the present technology is a system used for a cloud gaming service related to a multiplayer AR game (AR: Augmented Reality). In this embodiment, a user can enjoy a multiplayer AR game by moving in a real space while wearing or carrying a terminal device 20.

[0017] Furthermore, in this embodiment, the server device 10 on the cloud generates an AR image (cloud rendering) and transmits it to each terminal device 20, and each terminal device 20 AR-displays the AR image received from the server device 10. Therefore, in this embodiment, it is possible to AR-display, on each terminal device 20, a high-quality AR image (for example, a three-dimensional CG image (CG: Computer Graphics)) that cannot be realized by the processing capability of the terminal device 20.

[0018] On the other hand, if the server device 10 generates an AR image for each terminal device 20 individually, the processing load of the server device 10 increases in proportion to the number of terminal devices 20 (number of users) participating in the game. For this reason, in this embodiment, the server device 10 performs processing such as grouping the terminal devices 20 that exist within a predetermined area in real space based on the position information of each terminal device 20, and generating and transmitting a common AR image to the terminal devices 20 included in the group.

[0019] Position information and shape information of the shielding object 6 in real space are stored in a database in the shielding object database 30. The shielding object 6 is, for example, a building, a wall, etc. in real space. How the position information and shape information of the shielding object 6 are used will be described in detail later with reference to Figs. 13 to 15 etc.

[0020] <Configuration of each part of the information processing system 100> [Server device 10] 2 is a block diagram showing the internal configuration of the server device 10. As shown in FIG. 2, the server device 10 includes a control unit 11, a storage unit 12, and a communication unit 13.

[0021] The control unit 11 executes various calculations based on various programs stored in the storage unit 12, and performs overall control of each unit of the server device 10. The processing of the control unit 11 of the server device 10 will be described in detail later in the section explaining the operation.

[0022] The control unit 11 is realized by hardware or a combination of hardware and software. The hardware is configured as a part or the whole of the control unit 11, and examples of this hardware include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of two or more of these. This also applies to the control unit 11 in the terminal device 20.

[0023] The storage unit 12 includes a non-volatile memory for storing various programs and various data required for processing by the control unit 11, and a volatile memory used as a work area for the control unit 11. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from another server device on the network.

[0024] The communication unit 13 is configured to be able to communicate with each terminal device 20 via the network.

[0025] [Terminal device 20] The terminal device 20 may be, for example, a wearable device that can be attached to the user's body, or a mobile device that can be carried by the user.

[0026] Examples of wearable devices include head mounted displays (HMDs), wristbands, wristwatches, finger rings, pendants, etc. Examples of mobile devices include mobile phones (including smartphones), tablet PCs (Personal Computers), portable game consoles, portable music players, etc.

[0027] Typically, the terminal device 20 may be any device that can satisfy the following four conditions: (1) being capable of AR display, (2) being capable of estimating its own position and orientation in real space, (3) being capable of communicating with the server device 10, and (4) being capable of moving in accordance with the movement of the user.

[0028] Fig. 3 is a block diagram showing the internal configuration of the terminal device 20. As shown in Fig. 3, the terminal device 20 includes a control unit 21, a storage unit 22, a display unit 23, an imaging unit 24, an inertial sensor 25, a GPS (Global Positioning System) 26, an operation unit 27, and a communication unit 28.

[0029] The control unit 21 executes various calculations based on various programs stored in the storage unit 22, and performs overall control of each unit of the terminal device 20. The processing of the control unit 21 of the terminal device 20 will be described in detail later in the section explaining the operation.

[0030] The storage unit 22 includes a non-volatile memory for storing various programs and various data required for processing by the control unit 21, and a volatile memory used as a work area for the control unit 21. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from a server device on a network.

[0031] The display unit 23 performs AR display of the virtual object in response to the control of the control unit 21. The display unit 23 may be a see-through type display unit or a video see-through type display unit. In the case of a see-through type display unit (e.g., HMD), the virtual object is AR-displayed on a transparent display unit located in front of the user's line of sight. In the case of a video see-through type display unit, the AR display is performed by displaying on the display unit 23 an image in which the virtual object is superimposed on an image currently captured by the imaging unit 24.

[0032] The imaging unit 24 is, for example, a camera, and includes an imaging element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. The imaging unit 24 acquires image information in the real space around the terminal device 20, and outputs the acquired image information to the control unit 21.

[0033] The inertial sensor 25 includes an acceleration sensor that detects acceleration in three axial directions and an angular velocity sensor that detects angular velocity around three axes. The inertial sensor 25 outputs the acceleration in the three axial directions and the angular velocity around the three axes obtained by the detection to the control unit 21 as inertial information.

[0034] In this embodiment, the inertial sensor 25 has three detection axes, but the detection axes may be one or two. In this embodiment, two types of sensors are used as the inertial sensor 25, but one type or three or more types of sensors may be used as the inertial sensor 25. Other examples of the inertial sensor 25 include a speed sensor and an angle sensor.

[0035] The GPS 26 receives signals from GPS satellites and estimates its own position in a global coordinate system.

[0036] The operation unit 27 is, for example, a pressing type, a proximity type, or other type of operation unit, and detects an operation by the user and outputs the operation to the control unit 21. The operation unit 27 includes a function as a game controller that can detect a command operation of a game by the user. The game controller (operation unit 27) may be provided in the terminal device 20 itself, or may be provided separately from the terminal device 20.

[0037] The communication unit 28 is configured to be able to communicate with the server device 10 on the network.

[0038] <Operation description> Next, a description will be given of the processing of the information processing system 100. Fig. 4 is a flowchart showing the processing of the server device 10. Figs. 5 and 6 are flowcharts showing the processing of the terminal device 20.

[0039] [Self-position estimation process of terminal device 20] First, the self-position estimation process of the terminal device 20 will be described with reference to Fig. 5. The control unit 21 of each terminal device 20 repeatedly executes a process of estimating its own position and attitude (step 201) and outputting the estimated self-position information to the server device 10 (step 202) at a predetermined cycle.

[0040] In step 201, the control unit 21 of the terminal device 20 performs estimation of its own position and attitude (hereinafter simply referred to as self-position estimation) based on image information obtained by the imaging unit 24, inertial information obtained from the inertial sensor 25, GPS information obtained by the GPS 26, etc.

[0041] In the self-location estimation, for example, a group of feature points extracted from image information acquired by the imaging unit 24 is compared with a group of feature points included in map information, and the self-location estimation of the terminal device 20 is performed.

[0042] The map information used for self-location estimation may be created in advance or may be created simultaneously with self-location estimation without being created in advance, and either method may be used. The method of creating map information simultaneously with self-location estimation is generally called SLAM (Simultaneous Localization and Mapping).

[0043] The self-location estimation includes relocalization and motion tracking. Relocalization is a technique for estimating a self-location in a global coordinate system based on a group of feature points in image information and a group of feature points in map information.

[0044] This relocalization is performed immediately after the terminal device 20 is turned on, when self-position estimation based on motion tracking fails, etc. Alternatively, a process of comparing a group of feature points from image information with a group of feature points included in map information may be constantly performed, and relocalization may be performed when matching between these groups of feature points is successful.

[0045] Motion tracking is a technology that calculates the amount of change (movement) in self-position and attitude at short intervals based on image information (or inertial information), and estimates the current self-position in a global coordinate system by sequentially adding up these changes.

[0046] In motion tracking, first, image information acquired by the imaging unit 24 is subjected to image processing to extract a group of feature points from the image information. Then, the group of feature points in the previous image information is compared with the group of feature points in the current image information to calculate the amount of change between the previous self-position and orientation and the current self-position and orientation. This amount of change is added to the previous self-position and orientation, thereby estimating the current self-position in the global coordinate system.

[0047] In the description here, a case has been described in which image information from the imaging unit 24 is used for motion tracking, but inertial information from the inertial sensor 25 may be used instead of image information. Alternatively, both image information and inertial information may be used.

[0048] In addition, in the self-location estimation, GPS information obtained by the GPS 26 may be used.

[0049] Currently, various methods have been proposed as self-location estimation techniques, but typically, the terminal device 20 may execute the self-location estimation process by any method.

[0050] [Processing of server device 10] Next, the processing of the control unit 11 of the server device 10 will be described with reference to Fig. 4. First, the control unit 11 of the server device 10 sets point 1 in the global coordinate system in response to an input by the game creator (step 101). Fig. 7 is a diagram showing the state when point 1 is set in the global coordinate system.

[0051] 7, in this embodiment, the game creator sets any number of points 1 at any positions in the global coordinate system according to the content of the game. Point 1 is a point that is the center of a predetermined area 2 in which the terminal devices 20 are grouped. In the following description, the predetermined area including point 1 and in which the terminal devices 20 are grouped will be referred to as a grouping area 2 for convenience.

[0052] 7 shows an example in which the number of points 1 and grouping regions 2 is four. On the other hand, the number of points 1 and grouping regions 2 may be one or more, and is not particularly limited.

[0053] 7 shows an example in which the shape of the grouping area 2 is a circle, but the shape of the grouping area 2 may be an ellipse, a polygon (triangle, rectangle, etc.), a star, etc., and is not particularly limited. In this embodiment, the area within a range of a predetermined distance (first distance) x from the point 1 is set as the grouping area 2.

[0054] In this embodiment, the grouping area 2 corresponds to the play area, and when the terminal device 20 (user) is present in this play area, the game can be executed by the terminal device 20. Note that the play area does not need to completely match the grouping area 2, and for example, the play area may be set to be larger or smaller than the grouping area 2.

[0055] After setting the point 1 and the grouping area 2 in the global coordinate system, the control unit 11 of the server device 10 next receives self-position information from each of the terminal devices 20 (step 102). Next, the server device 10 groups the terminal devices 20 located within the grouping area 2 (step 103).

[0056] 7, four terminal devices 20 are present in the upper left grouping area 2, and so these four terminal devices 20 are grouped into the same group. Grouping is performed in the same manner for each of the grouping areas 2 in the upper right, lower left, and lower right.

[0057] It should be noted that, for terminal devices 20 that do not belong to any grouping area 2, grouping is not performed, and no AR image is transmitted from the server device 10.

[0058] After grouping the terminal devices 20 for each grouping area 2 (each point 1), the control unit 11 of the server device 10 then generates a common AR image for each of the terminal devices 20 included in the same group and transmits the common AR image to each of the terminal devices 20 (step 104). After that, the control unit 11 of the server device 10 repeatedly executes the processes of steps 102 to 104.

[0059] As described above, in the present embodiment, the server device 10 does not need to generate an AR image for each terminal device 20 individually, but only needs to generate an AR image for each grouping area 2. Therefore, the processing load of the server device 10 in cloud rendering can be appropriately reduced.

[0060] Fig. 8 is a diagram showing an example of a common AR image. As shown in Fig. 8, in this embodiment, the common AR image is a 360-degree AR image including an enemy virtual object 5. The inner circumference of the rendering area in the common AR image is set to a distance (second distance) r from point 1, and the outer circumference is set to a distance (third distance) R from point 1. That is, the rendering area of ​​the common AR image is the area from distance (second distance) r to distance R (third distance) from point 1, and the virtual object is placed within this area.

[0061] In this embodiment, the common AR image is a 360-degree image, but the common AR image typically only needs to be set to a wider angle than the display angle of the display unit 23 of the terminal device 20 (e.g., 270°, 180°, etc.).

[0062] Furthermore, the distance r on the inner circumference is typically set to be equal to or greater than the distance x for grouping (r≧x). Therefore, in this embodiment, the rendering area of ​​the enemy virtual object 5 is set at a position outside the grouping area 2 (play area).

[0063] In the example shown in FIG. 8, as an example of enemy virtual objects 5, a virtual object of a bird-type monster is placed in the northwest (upper left), a virtual object of a wizard-type monster is placed in the east (right), and a virtual object of a dragon-type monster is placed in the southwest (lower left).

[0064] Here, when a common AR image is used in each of the terminal devices 20 belonging to the same group, how the enemy virtual object 5 is viewed by the user in each terminal device 20 will be described. In the example shown in Fig. 8, three terminal devices 20 (three users) belong to the same group, but are located at different positions in real space.

[0065] On the other hand, the AR image used in the three terminal devices 20 is a common AR image. Therefore, when each terminal device 20 faces the same direction in the real space, each user sees the enemy virtual object 5 facing the same direction at the same distance from the user in the AR space. In other words, although each user exists at a different position in the real space, all users see the enemy virtual object 5 as if they are looking at it from the position of point 1 in the AR space.

[0066] An example will be given for specific explanation. It is assumed that all the terminal devices 20 are facing north (upward) in FIG. 8. Regardless of the position of the user (terminal device 20), all users see the enemy virtual object 5 as if they are looking at it from the position of point 1, and the bird-type monster virtual object is currently located in the northwest (upper left) in the AR image. Therefore, in this case, all users see the enemy virtual object 5 facing in the same direction as the other users, diagonally forward to the left, at the same distance as the other users, regardless of the position of the user (terminal device 20).

[0067] Here, the enemy virtual object 5 is an example of the first virtual object. In this embodiment, the first virtual object is a virtual object included in a common AR image, and is a virtual object whose AR display position relative to each terminal device 20 is the same for each position of the terminal device 20. Note that the first virtual object is not limited to the enemy virtual object 5, and may be any object as long as it satisfies this condition.

[0068] [Processing of terminal device 20] Next, the process of the terminal device 20 will be described. Referring to Fig. 6, the control unit 21 of the terminal device 20 first determines whether or not an AR image common to the group has been transmitted from the server device 10 to the terminal device 20 itself and the AR image has been received (step 301). As described above, the common AR image is transmitted from the server device 10 to the terminal devices 20 located within the grouping area 2 (see step 104).

[0069] If an AR image for the user himself / herself has been received (YES in step 301), the control unit 21 of the terminal device 20 determines whether or not the user himself / herself is currently in combat with an enemy virtual object 5 (step 302).

[0070] If the terminal device 20 is not in combat even though the AR image has been received (NO in step 302), that is, if the terminal device 20 that was located outside the grouping area 2 enters the grouping area 2, the control unit 21 of the terminal device 20 starts combat with the enemy (step 303). When combat starts, the control unit 21 of the terminal device 20 may notify the user by sound, image, or the like that combat with the enemy has started.

[0071] When the battle starts, the control unit 21 of the terminal device 20 proceeds to step 304. If it is determined in step 302 that a battle is already underway, the control unit 21 of the terminal device 20 skips step 303 and proceeds to the next step 304.

[0072] In step 304, the control unit 21 of the terminal device 20 sets a display area from the 360° AR image based on the current attitude of the terminal device 20 and the display angle of view of the display unit 23. Next, the control unit 21 of the terminal device 20 AR-displays the AR image (enemy virtual object 5) corresponding to the display area (step 305).

[0073] At this time, as described above, although the users belonging to the same group are located at different positions in the real space, all users appear to be viewing the enemy virtual object 5 from the position of point 1.

[0074] In step 301, if a common AR image for the terminal device 20 has not been received (NO in step 301), the control unit 21 of the terminal device 20 determines whether the terminal device 20 is currently in combat with an enemy (step 306).

[0075] If an AR image for the terminal device 20 has not been received and the terminal device 20 is not currently fighting an enemy (NO in step 306), that is, if the terminal device 20 continues to be located outside the grouping area 2, the control unit 21 of the terminal device 20 returns to step 301.

[0076] On the other hand, if an AR image for the terminal device 20 has not been received but the terminal device 20 is currently engaged in combat with an enemy (YES in step 306), that is, if the terminal device 20 within the grouping area 2 has moved outside the grouping area 2, the control unit 21 of the terminal device 20 proceeds to step 307.

[0077] In step 307, the control unit 21 of the terminal device 20 determines whether a predetermined time (about several seconds) has elapsed since the last common AR image was received (step 307). If the predetermined time has not elapsed (NO in step 307), the control unit 21 of the terminal device 20 returns to step 301.

[0078] On the other hand, if the predetermined time has elapsed (YES in step 307), the control unit 21 of the terminal device 20 ends the battle with the enemy (step 308) and returns to step 301. When the battle ends, the control unit 21 of the terminal device 20 may notify the user by sound, image, or the like that the battle with the enemy has ended.

[0079] <Flying Object> Next, the flying object (second virtual object) will be described. Fig. 9 is a diagram for explaining the flying object.

[0080] A flying object is a virtual object that is displayed in AR when the user attacks an enemy by command, and is, for example, a virtual object such as a spell, a bullet, or an arrow that moves from the user (terminal device 20) toward the enemy virtual object 5.

[0081] 9, the user and the terminal device 20 located northwest (upper left) of point 1 are conveniently referred to as the first user and the first terminal device 20a, and the user and the terminal device 20 located east (right) of point 1 are conveniently referred to as the second user and the second terminal device 20b. Also, assume that the flying object is a virtual object serving as a common AR image.

[0082] First, assume that a first user inputs a command through the first terminal device 20a to perform, for example, a magic attack on an enemy virtual object 5. In this case, the first user and the first terminal device 20a are not located at point 1, but the first user sees the enemy virtual object 5 as if he or she were looking at the enemy virtual object 5 from the position of point 1. In this case, it is natural for the first user to display in AR a flying object resulting from the magic attack or the like, so that the flying object flies from the first user's position (i.e., point 1) toward the enemy virtual object 5.

[0083] Although the second user and the second terminal device 20b are not located at point 1, the second user sees the enemy virtual object 5 as if he / she were looking at the enemy virtual object 5 from the position of point 1. On the other hand, the second user does not perform a magic attack or the like when the first user performs a magic attack or the like. However, when the flying object is a virtual object based on a common AR image, the flying object is also displayed in AR for the second user as if the flying object based on the magic attack or the like is flying from the user's position toward the enemy virtual object 5.

[0084] That is, if the flying object is a virtual object based on a common AR image, the second user may see the flying object unnaturally. This problem may not be a problem in a single-player AR game, but it is a problem in a multiplayer AR game. Therefore, in this embodiment, the AR image of the flying object is generated individually by each terminal device 20 and superimposed on the common AR image generated by the server device 10.

[0085] Here, the flying object is an example of the second virtual object. The second virtual object is a virtual object whose AR display position relative to each terminal device 20 differs for each position of the terminal device 20. The second virtual object is not limited to a flying object and may be any virtual object as long as it satisfies this condition.

[0086] For example, the second virtual object may be a virtual object of a fellow character displayed in AR near the user. For example, assume that there is a fellow virtual object located on the right side of the first user in FIG. 9. In this case, a fellow monster located on the right side as seen from the first user should be located on the left side as seen from the second user, and it is unnatural for it to be located on the right side as seen from the second user. Therefore, for such fellow virtual objects, AR images may be generated individually by each terminal device 20 and superimposed on a common AR image generated by the server device 10.

[0087] [Processing of Terminal Device 20: Information Transmitting Side] Next, a description will be given of the process of the terminal device 20 on the information transmitting side. Fig. 10 is a flow chart showing the process of the terminal device 20 on the information transmitting side.

[0088] First, the control unit 21 of the terminal device 20 determines whether or not a battle is occurring between the terminal device 20 and an enemy virtual object 5 (whether the object is located within the grouping area 2) (step 401) (see steps 303 and 308). If a battle is not occurring (NO in step 401), the control unit 21 of the terminal device 20 returns to step 401 again.

[0089] On the other hand, if a battle is in progress (YES in step 401), the control unit 21 of the terminal device 20 determines whether or not a flying attack command has been input by the user via the operation unit 27 (step 402). If a flying attack command has not been input (NO in step 402), the control unit 21 of the terminal device 20 returns to step 401.

[0090] On the other hand, if a flying attack command has been input (YES in step 402), the control unit 21 of the terminal device 20 proceeds to the next step 403. In step 403, the control unit 21 of the terminal device 20 transmits the flying object related information to the server device 10, and causes the server device 10 to transmit the flying object related information and its own position information to the other terminal devices 20 belonging to the same group.

[0091] The related information of the flying object includes identification information of the flying object (for example, information identifying whether it is magic, a bullet, or a bow and arrow), information on the flying direction of the flying object, and information on the flying speed of the flying object.

[0092] After the server device 10 transmits the related information of the flying object and its own position information to the other terminal devices 20 belonging to the same group, the control unit 21 of the terminal device 20 proceeds to the next step 404. In step 404, the control unit 21 of the terminal device 20 superimposes the flying object on the common AR image transmitted from the server device 10, and performs AR display so that the flying object flies from its own position (i.e., point 1) in the flying direction at a speed according to the flying speed. The type of the flying object is determined according to the type of flying attack command.

[0093] After that, the control unit 21 of the terminal device 20 returns to step 401. In addition, in the terminal device 20 where the flying attack command has been executed, the flying object appears to fly in the flying direction from its own position (i.e., from point 1) as shown in FIG.

[0094] [Processing of Terminal Device 20: Information Receiving Side] Next, the processing of the terminal device 20 on the information receiving side will be described. Fig. 11 is a flowchart showing the processing of the terminal device 20 on the information receiving side. Fig. 12 is a diagram showing how the flying object appears on the terminal device 20 on the information receiving side.

[0095] First, the control unit 21 of the terminal device 20 determines whether or not a battle is occurring between the terminal device 20 and an enemy virtual object 5 (whether the object is located within the grouping area 2) (step 501) (see steps 303 and 308). If a battle is not occurring (NO in step 501), the control unit 21 of the terminal device 20 returns to step 501 again.

[0096] On the other hand, if a battle is in progress (YES in step 501), the control unit 21 of the terminal device 20 proceeds to the next step 502. In step 502, the control unit 21 of the terminal device 20 determines whether or not related information about the flying object and position information about another terminal device 20 (a terminal device 20 that executed a flying attack command) have been received from the server device 10.

[0097] If this information has not been received from the server device 10 (NO in step 502), the control unit 21 of the terminal device 20 returns to step 501. On the other hand, if this information has been received from the server device 10 (YES in step 502), the control unit 21 of the terminal device 20 proceeds to the next step 503.

[0098] In step 503, the control unit 21 of the terminal device 20 calculates the difference D between its own position and the position of the other terminal device 20 (the terminal device 20 that executed the flying attack command) in the global coordinate system, and sets the starting point of the flying object (AR display position) based on the calculated difference D.

[0099] Fig. 12 shows an example of the difference D between the self-position and the other terminal device 20 and the position of the starting point of the flight command. In the explanation of Fig. 12, similarly to Fig. 9, the user and the terminal device 20 located northwest (upper left) of point 1 will be conveniently called the first user and the first terminal device 20a, and the user and the terminal device 20 located east (right) of point 1 will be conveniently called the second user and the second terminal device 20b.

[0100] When a flying attack command is executed on the first terminal device 20a side, the difference D and the starting point are calculated on the second terminal device 20b side. Although the second terminal device 20b is not located at point 1, the second user sees the enemy virtual object 5 as if he or she were looking at the enemy virtual object 5 from the position of point 1. On the other hand, when the first terminal device 20a is viewed from the position of the second terminal device 20b in the real space, the first terminal device 20a is located at the position of difference D. Therefore, in this case, the starting point of the flying object is set at a position corresponding to the difference D from the position of point 1.

[0101] After calculating the starting point, the control unit 21 of the terminal device 20 calculates the flight direction of the flying object as seen from the terminal device 20 itself based on the flight direction information included in the received related information of the flying object (step 504) (see the direction of the arrow in Figure 12).

[0102] Next, the control unit 21 of the terminal device 20 superimposes a flying object on the common AR image transmitted from the server device 10, and performs AR display so that the flying object flies from the starting point position in the flying direction at a speed corresponding to the flying speed (moving speed) (step 505) (see the arrow in FIG. 12). The type of the flying object is determined by the identification information included in the received flying object related information.

[0103] Note that when the flying object hits the enemy virtual object 5, by adjusting the flying speed (moving speed) of the flying object, it will appear to hit the enemy virtual object 5 at the same timing for all users. The adjustment of the flying speed can be realized, for example, by setting the time from when the attack by the flying object starts until the flying object hits the enemy virtual object 5 to a certain constant.

[0104] When the flying object hits the enemy virtual object 5, an effect indicating that the flying object has hit the enemy object or an action of the enemy virtual object 5 falling backward may be AR displayed. In this case, the effect and the falling backward action are typically realized by the common AR image generated by the server device 10.

[0105] Through the processes described above, in this embodiment, the starting point, flying direction, flying speed (timing of hitting the enemy virtual object 5), etc. of the flying object are natural without discomfort for all users included in the group.

[0106] <AR Occlusion> Next, the AR occlusion of the enemy virtual object 5 by the occlusion object 6 will be described. Here, the occlusion object 6 is a real object in the real space where the relative positions from each terminal device 20 are different for each position of the terminal device 20, and is an object that can occlude the enemy virtual object 5 (the first virtual object). This occlusion object 6 is, for example, a building or a wall in the real space.

[0107] A description will now be given of how the enemy virtual object 5 is occluded by the occluding object 6 when a common AR image is used in each terminal device 20.

[0108] Fig. 13 is a diagram showing how an enemy virtual object 5 is occluded by an occluding object 6 for a first user and a first terminal device 20a located northwest of point 1. Fig. 14 is a diagram showing how an enemy virtual object 5 is occluded by an occluding object 6 for a second user and a second terminal device 20b located east of point 1.

[0109] First, Fig. 13 will be described. As shown in Fig. 13, a first user (first terminal device 20a) is located northwest (upper left) of point 1. In the AR space, the first user sees the enemy virtual object 5 as if he or she were looking at the enemy virtual object 5 from the position of point 1, regardless of the position of the first user in the real space.

[0110] For example, when the first user faces north, the field of view of the first user in the AR space is as shown by the inverted triangle of dotted lines in Fig. 13. Therefore, as shown in the upper center of Fig. 13, the first user sees an enemy virtual object 5 located in the front left in the AR space.

[0111] On the other hand, the position and apparent shape of the occluding object 6 seen from the position of the first user in the real space changes according to the position of the first user in the real space. For example, when the first user faces north, the field of view of the first user in the real space is like an inverted triangle of a solid line in Fig. 13. Therefore, as shown in the lower center of Fig. 13, the first user sees the smaller occluding object 6 to the left front and the larger occluding object 6 to the right front in the real space.

[0112] Therefore, based on the positional relationship between the enemy virtual object 5 in the AR space and the occluding object 6 in the real space, the enemy virtual object 5 and the occluding object 6 ultimately appear to the first user as shown on the right side of Fig. 13. In other words, when the enemy virtual object 5 is viewed by the first user, the enemy virtual object 5 should be displayed in AR such that parts of the torso and legs occluded by the occluding object 6 are missing.

[0113] Next, Fig. 14 will be described. As shown in Fig. 14, a second user (second terminal device 20b) is located to the east (right) of point 1. In the AR space, regardless of the position of the second user in the real space, the second user sees the enemy virtual object 5 as if he or she were looking at the enemy virtual object 5 from the position of point 1.

[0114] For example, when the second user faces north, the field of view of the second user in the AR space is as shown by a dotted inverted triangle in Fig. 14. Therefore, as shown in the upper center of Fig. 14, the second user sees an enemy virtual object 5 located in the front left in the AR space. The first user and the second user see the enemy virtual object 5 in the same way.

[0115] On the other hand, the position and apparent shape of the occluding object 6 as seen from the position of the second user in the real space changes according to the position of the second user in the real space. For example, when the second user faces north, the field of view of the second user in the real space is like an inverted triangle of a solid line in Fig. 14. Therefore, as shown in the lower center of Fig. 14, the second user sees the larger occluding object 6 in the real space to the left and in front.

[0116] It should be noted that, although the smaller obscuring object 6 is also included in the field of view from the first user, the smaller obscuring object 6 is not included in the field of view from the second user.

[0117] Due to the positional relationship between the enemy virtual object 5 in the AR space and the occluding object 6 in the real space, the enemy virtual object 5 and the occluding object 6 ultimately appear to the second user as shown on the right side of Fig. 14. In other words, when the enemy virtual object 5 is viewed by the second user, the enemy virtual object 5 should be displayed in AR such that most of the lower part occluded by the occluding object 6 is missing.

[0118] Comparing the diagram on the right side of Figure 13 with the diagram on the right side of Figure 14, it can be seen that which part of the enemy virtual object 5 is occluded by the occluding object 6 depends on the position of each terminal device 20 and is not common to all terminal devices 20.

[0119] Even under such circumstances, in order to perform accurate AR occlusion in each terminal device 20, the following processing is executed in this embodiment.

[0120] For example, when transmitting information for AR display to each terminal device 20 in step 104 of FIG. 5, the control unit 11 of the server device 10 transmits the following three pieces of information: (1) RGB (Red Green Blue) information of enemy virtual object 5 (2) Depth information of enemy virtual object 5 (3) Depth information of occluding object 6 (occluding object information regarding occluding object 6)

[0121] FIG. 15 is a diagram showing RGB information of the enemy virtual object 5, depth information of the enemy virtual object 5, and depth information of the occluding object 6. As shown in FIG.

[0122] Here, the RGB information of the enemy virtual object 5 and the depth information of the enemy virtual object 5 are common information for each terminal device 20 belonging to the same group. On the other hand, the depth information of the occluding object 6 is individual information for each terminal device 20.

[0123] That is, the control unit 11 of the server device 10 generates depth information of the occluding object 6 individually for each terminal device 20 and transmits the depth information to each terminal device 20. The depth information of the occluding object 6 is generated by the control unit 11 of the server device 10 based on the position information of the terminal device 20 and the position information and shape information of the occluding object 6 stored in the occluding object database 30.

[0124] After receiving the RGB information of the enemy virtual object 5, the depth information of the enemy virtual object 5, and the depth information of the occluding object 6 from the server device 10, the control unit 21 of the terminal device 20 executes the following process.

[0125] First, the control unit 21 of the terminal device 20 determines the front-to-back relationship between the enemy virtual object 5 and the occluding object 6 based on the depth information of the enemy virtual object 5 and the depth information of the occluding object 6. Next, when the enemy virtual object 5 is located behind the occluding object 6, the control unit 21 of the terminal device 20 omits a portion located behind the occluding object 6 from the RGB information of the enemy virtual object 5. Then, the control unit 21 of the terminal device 20 executes AR display for the portion of the RGB information of the enemy virtual object 5 that is not occluded by the occluding object 6.

[0126] The right side of FIG. 15 shows the RGB information of the enemy virtual object 5 when the portion occluded by the occluding object 6 is missing and the portion not occluded by the occluding object 6 is displayed in AR.

[0127] By the above-described processing, even in the present technology in which a common AR image is used in each terminal device 20, accurate AR shading can be performed.

[0128] <Effect, etc.> As described above, in this embodiment, the server device 10 groups the terminal devices 20 present in the grouping area 2 based on the respective position information of each terminal device 20, and generates and transmits a common AR image to the terminal devices 20 included in the group.

[0129] As a result, in this embodiment, the server device 10 does not need to generate an AR image for each terminal device 20 individually, but only needs to generate an AR image for each grouping area 2. Therefore, the processing load of the server device 10 in cloud rendering can be appropriately reduced.

[0130] Furthermore, in this embodiment, each terminal device 20 individually generates an AR image of a flying object (second virtual object) and superimposes the AR image on a common AR image.

[0131] As a result, in this embodiment, it is possible to present the position and movement of the flying object in a natural way to each user included in the group.

[0132] Furthermore, in this embodiment, the server device 10 generates depth information of the occluding object 6 individually for each terminal device 20. Then, based on the depth information of the occluding object 6, the terminal device 20 performs AR display for the part of the enemy virtual object 5 that is not occluded by the occluding object 6.

[0133] As a result, even in the present technology in which a common AR image is used in each terminal device 20, accurate AR shading can be performed.

[0134] <<Various modified examples>> In the above description, the present technology is applied to a game. However, the present technology can also be used for purposes other than games. Typically, the present technology can be used for any purpose such as various events where multiple people gather in a certain area.

[0135] The present technology can also be configured as follows. (1) a control unit that groups terminal devices present within a predetermined area based on position information of each terminal device capable of executing AR display, and generates and transmits a common AR image to the terminal devices included in the group A server device comprising: (2) The server device according to (1), The control unit groups the terminal devices that are present within a region of a first distance from a predetermined point. Server device. (3) The server device according to (2), The common AR image is an image having a wider angle than the AR display angle of the terminal device. Server device. (4) The server device according to (3), The common AR image is a 360-degree image centered on the point. Server device. (5) The server device according to any one of (2) to (4), The common AR image is an image within a region that is equal to or greater than a second distance and equal to or less than a third distance from the point. Server device. (6) The server device according to (5), The second distance is greater than or equal to the first distance. Server device. (7) The server device according to any one of (1) to (6), the common AR image includes a first virtual object; Each of the terminal devices individually generates an AR image of a second virtual object whose AR display position relative to the terminal device varies depending on the position of each of the terminal devices, and superimposes the AR image on the common AR image. Server device. (8) The server device according to (7), The terminal device acquires position information of the other terminal device, and sets an AR display position of the second virtual object based on its own position information and the position information of the other terminal device. Server device. (9) The server device according to (8), the second virtual object is a movable object; The terminal device adjusts the moving speed of the second virtual object. Server device. (10) The server device according to any one of (1) to (9), the common AR image includes a first virtual object; the control unit generates occluding object information relating to an occluding object, which is a real object in a real space and whose relative position from each of the terminal devices differs for each of the terminal devices, and which may occlude the first virtual object, for each of the terminal devices, and transmits the occluding object information to each of the terminal devices. Server device. (11) The server device according to (10), The control unit generates the shielding object information based on position information of the terminal device and position information of the shielding object. Server device. (12) The server device according to (10) or (11), The terminal device performs the AR display for a portion of the first virtual object that is not occluded by the occluding object based on the occluding object information. Server device. (13) A control unit that groups terminal devices present within a predetermined area based on position information of each terminal device capable of executing AR display, receives a common AR image from a server device that generates and transmits a common AR image to the terminal devices included in the group, and executes AR display based on the common AR image. A terminal device comprising: (14) A server device that groups terminal devices present within a predetermined area based on position information of each terminal device capable of executing AR display, and generates and transmits a common AR image to the terminal devices included in the group; a terminal device that receives the common AR image and performs AR display based on the common AR image; An information processing system comprising: (15) Grouping terminal devices present within a predetermined area based on the location information of each terminal device capable of executing AR display, and generating and transmitting a common AR image to the terminal devices included in the group. Information processing methods. [Explanation of symbols]

[0136] 1. Points 2...Grouping area 5…Enemy virtual object 6…Occluding object 10. Server equipment 20...Terminal device 30…Occluded Object Database

Claims

1. A control unit that groups terminal devices present within a predetermined area based on position information of each terminal device capable of displaying AR (Augmented Reality), and generates and transmits a common AR image to the terminal devices included in the group. A server device comprising: the common AR image includes a first virtual object; each of the terminal devices individually generates an AR image of a second virtual object, the AR display position of which is different for each of the terminal devices relative to the position of the terminal device, and superimposes the AR image on the common AR image; the terminal device acquires position information of the other terminal device, and sets an AR display position of the second virtual object based on its own position information and the position information of the other terminal device; the second virtual object is a movable object; The terminal device adjusts the moving speed of the second virtual object. Server device.

2. 2. The server device according to claim 1, The control unit groups the terminal devices that are present within a region of a first distance from a predetermined point. Server device.

3. 3. The server device according to claim 2, The common AR image is an image with a wider angle than the AR display angle of the terminal device. Server device.

4. 4. The server device according to claim 3, The common AR image is a 360-degree image centered on the point. Server device.

5. 3. The server device according to claim 2, The common AR image is an image within a region that is a second distance or more and a third distance or less from the point. Server device.

6. 6. The server device according to claim 5, The second distance is greater than or equal to the first distance. Server device.

7. 2. The server device according to claim 1, the control unit generates occluding object information relating to an occluding object, which is a real object in a real space and whose relative position from each of the terminal devices differs for each of the terminal devices, and which may occlude the first virtual object, for each of the terminal devices, and transmits the occluding object information to each of the terminal devices. Server device.

8. 8. The server device according to claim 7, The control unit generates the shielding object information based on position information of the terminal device and position information of the shielding object. Server device.

9. 8. The server device according to claim 7, The terminal device performs the AR display for a portion of the first virtual object that is not occluded by the occluding object based on the occluding object information. Server device.

10. A control unit that groups terminal devices present within a predetermined area based on the position information of each terminal device capable of executing AR display, receives a common AR image from a server device that generates and transmits the common AR image to the terminal devices included in the group, and executes AR display based on the common AR image. A terminal device comprising: the common AR image includes a first virtual object; the control unit generates an AR image of a second virtual object, the AR display position of which is different for each of the terminal devices relative to the position of each of the terminal devices, and superimposes the AR image on the common AR image; the control unit acquires position information of the other terminal device, and sets an AR display position of the second virtual object based on the position information of the other terminal device and the position information of the own terminal device; the second virtual object is a movable object; The control unit adjusts the moving speed of the second virtual object. Terminal device.

11. a server device that groups terminal devices present within a predetermined area based on position information of each terminal device capable of executing AR display, and generates and transmits a common AR image to the terminal devices included in the group; a terminal device that receives the common AR image and performs AR display based on the common AR image; An information processing system comprising: the common AR image includes a first virtual object; each of the terminal devices individually generates an AR image of a second virtual object, the AR display position of which is different for each of the terminal devices relative to the position of the terminal device, and superimposes the AR image on the common AR image; the terminal device acquires position information of the other terminal device, and sets an AR display position of the second virtual object based on its own position information and the position information of the other terminal device; the second virtual object is a movable object; The terminal device adjusts the moving speed of the second virtual object. Information processing system.

12. The terminal devices present within a predetermined area are grouped based on the position information of each terminal device capable of executing AR display, and a common AR image is generated and transmitted to the terminal devices included in the group.

1. An information processing method, comprising: the common AR image includes a first virtual object; each of the terminal devices individually generates an AR image of a second virtual object, the AR display position of which is different for each of the terminal devices relative to the position of the terminal device, and superimposes the AR image on the common AR image; the terminal device acquires position information of the other terminal device, and sets an AR display position of the second virtual object based on its own position information and the position information of the other terminal device; the second virtual object is a movable object; The terminal device adjusts the moving speed of the second virtual object. Information processing methods.

Images