Information Processing Apparatus, Information Processing Method, and Recording Medium

Abstract

The information processing device (200) includes: a control device (220) that determines whether a first terminal (100a) is at a first position in the real space, controls the first terminal (100a) to start superimposing a virtual object on the real space according to the first position based on the determination that the first terminal (100a) is at the first position, determines whether a second terminal (100b) different from the first terminal (100a) is at the first position, and controls the second terminal (100b) to start superimposing the virtual object on the real space based on the determination that the second terminal (100b) is at the first position. The control device (220) performs the following operations differently with respect to at least one of the display start time and the display start position of the virtual object: causing the first terminal (100a) to start a first action of the virtual object and causing the second terminal (100b) to start a second action of the virtual object, and synchronizing the end of the first action and the end of the second action.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a recording medium. Background Art

[0002] In a multiplayer game such as a video game, from the perspective of any user, a character moves in a state that is almost the same. In addition, a method such as a video game or the like that changes a clip to switch to a movie scene to display a certain specific scene is adopted. A video game or the like also adopts a method for forcibly prohibiting user input for character control and automatically moving a rendering camera or a character to a place where the user wants to see.

[0003] In addition, as a technique for controlling the movement of a character, for example, a technique for controlling the movement of a character displayed on the other side in the case of a conversation via a network is known.

[0004] Citation List

[0005] Patent Document

[0006] Patent Document 1: JP 2012-181704 A Summary of the Invention

[0007] Technical Problem

[0008] However, in a multiplayer game using augmented reality (AR) technology, it is difficult for each user to experience a specific scene in the same way. For example, in such a multiplayer game using AR technology, for a user wearing a head-mounted display (HMD) or the like, a character as a virtual object is superimposed and displayed in the real space. In this case, it is difficult to adopt a method for forcibly controlling a rendering camera by controlling a rendering range of a character or the like for a user according to the position and orientation of the user. In addition, it is difficult to always capture a character or the like within the viewing angle due to restrictions on the HMD viewing angle and differences in the movement methods between users.

[0009] For this reason, when a character is displayed using AR technology or the like for a user wearing an HMD or the like, it is difficult to provide a specific scene required for the progress of a multiplayer game, such as an animation, to each user in the same way.

[0010] Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a recording medium that enable each user to experience a specific scene in the same way even using a display device with a limited viewing angle.

[0011] Solution to the Problem

[0012] According to the present disclosure, an information processing apparatus includes a control device configured to: determine whether a first mobile terminal is at a first position in the real space, and based on the determination that the first mobile terminal is at the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and determine whether a second mobile terminal different from the first mobile terminal is at the first position, and based on the determination that the second mobile terminal is at the first position, control the second mobile terminal to start superimposing the virtual object on the real space, wherein the control device performs the following operations differently at least according to the display start time or the display start position of the virtual object: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and cause the first action and the second action to end synchronously. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 FIG. is a diagram showing an example of information processing according to the first embodiment.

[0014] Figure 2 FIG. is a diagram showing an example of an information processing system according to the first embodiment.

[0015] Figure 3 FIG. is a diagram showing an example of an information processing system according to the first embodiment.

[0016] Figure 4 FIG. is a diagram showing an example of the structure of a mobile terminal according to the first embodiment.

[0017] Figure 5 FIG. is a diagram showing an example of how to wear a mobile terminal according to the first embodiment.

[0018] Figure 6 FIG. is a diagram showing an example of the structure of an information processing apparatus according to the first embodiment.

[0019] Figure 7 FIG. is a sequence diagram showing a processing flow according to the first embodiment.

[0020] Figure 8 FIG. is a diagram showing an example of information processing according to the second embodiment.

[0021] Figure 9 FIG. is a diagram showing an example of information processing according to the third embodiment.

[0022] Figure 10 FIG. is a hardware structure diagram of an example of a computer that implements the functions of the information processing apparatus. DETAILED DESCRIPTION

[0023] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in each of the following embodiments, the same components are denoted by the same reference numerals to avoid redundant description.

[0024] The present disclosure will be described in the order of the following items.

[0025] 1. First Embodiment

[0026] 1-1. Example of Information Processing According to the First Embodiment

[0027] 1-2. Structure of the System According to the First Embodiment

[0028] 1-3. Structure of the Mobile Terminal According to the First Embodiment

[0029] 1-4. Structure of the Information Processing Device According to the First Embodiment

[0030] 1-5. Information Processing Flow According to the First Embodiment

[0031] 1-6. Effects

[0032] 2. Second Embodiment

[0033] 2-1. Example of the Appearance Range According to the Second Embodiment

[0034] 3. Third Embodiment

[0035] 3-1. Example of the Appearance Range According to the Third Embodiment

[0036] 4. Modifications of the Embodiments

[0037] 5. Hardware Structure

[0038] 6. Effects

[0039] (1. First Embodiment)

[0040] [1-1. Example of Information Processing According to the First Embodiment]

[0041] Figure 1 is a diagram showing an example of information processing according to the first embodiment. The information processing according to the first embodiment is performed by Figure 1 the mobile terminals 100a and 100b and the information processing device 200 shown.

[0042] The mobile terminals 100a and 100b are information processing terminals configured to provide so-called AR technology and the like. According to the first embodiment, the mobile terminals 100a and 100b are wearable displays used by being respectively worn on the heads of users U1 and U2. More specifically, the mobile terminals 100a and 100b can each be referred to as an HMD, AR glasses, etc. In addition, in the following description, the mobile terminal 100a of user U1 corresponds to the first mobile terminal, and the mobile terminal 100b of user U2 corresponds to the second mobile terminal. Note that when the mobile terminals 100a and 100b of users U1 and U2 are not distinguished, the mobile terminals 100a and 100b are each simply referred to as the mobile terminal 100.

[0043] The mobile terminal 100 includes a display 101 as a transmissive display. For example, the mobile terminal 100 superimposes a virtual object represented by computer graphics (CG) or the like on the real space and displays the virtual object superimposed on the real space on the display 101. Note that in the following description, the virtual object superimposed on the real space is also referred to as an AR object. In addition, examples of AR objects include characters in a multiplayer game. In Figure 1 the example shown, the mobile terminal 100a worn by user U1 displays the AR object V1. In addition, the mobile terminal 100b worn by user U2 displays the AR object V2. The AR objects V1 and V2 are displayed in a superimposed manner within an appearance range E1 set near a door in the real space, for example. The mobile terminal 100 may have another element for outputting a predetermined output signal in addition to the display 101. For example, the mobile terminal 100 may include a speaker 102 for outputting sound, etc.

[0044] The information processing device 200 acquires the position information from the mobile terminals 100a and 100b and makes a determination regarding the display of the AR objects V1 and V2. The information processing device 200 sends a display start trigger to each of the mobile terminals 100a and 100b based on the determined result.

[0045] Based on the detection result from the sensor 110 described later, the AR objects V1 and V2 are set in a global coordinate system associated with the real space. At this time, according to the position, orientation, and viewing angle A1 of user U1, the AR object V1 is set at a position within the appearance range E1. In addition, according to the position, orientation, and viewing angle A2 of user U2, the AR object V2 is set at a position within the appearance range E1. In Figure 1 the example shown, the AR objects V1 and V2 are set at positions with different heights within the appearance range E1. Note that the viewing angles A1 and A2 are, for example, 40 degrees in the horizontal direction and 20 degrees in the vertical direction.

[0046] The AR objects V1 and V2 move from positions within the appearance range E1 along the movement paths R1 and R2 to, for example, the center of a road in the real space and become the AR object V3. The information processing device 200 instructs the mobile terminals 100a and 100b to start a specific scenario based on the AR object V3. The mobile terminals 100a and 100b provide the specific scenario based on the AR object V3 to the users U1 and U2.

[0047] However, the mobile terminals 100a and 100b differ from each other in position, orientation, and viewing angle. This makes the movement path R1 and the movement time until the AR object V1 becomes the AR object V3 different from the movement path R2 and the movement time until the AR object V2 becomes the AR object V3. Therefore, the AR object V3 is displayed asynchronously on the mobile terminals 100a and 100b, which makes it difficult to provide the specific scenario based on the AR object V3 to the users U1 and U2 in the same manner.

[0048] The mobile terminal 100 and the information processing device 200 according to the present disclosure perform the information processing described below so that each user can experience the specific scenario in the same manner even using a display device with a limited viewing angle. Specifically, each mobile terminal 100 sends its own position information to the information processing device 200. The position information is calculated based on various sensor information obtained by a sensor 110 described later, according to map matching, etc. For example, map matching is performed based on an image captured by a stereo camera used by the mobile terminal 100, and the position of the mobile terminal 100 is determined by using a gyro sensor, etc., to calculate the position information.

[0049] The information processing device 200 determines whether to display the AR objects V1 and V2 based on the position information of each of the mobile terminals 100a and 100b received in this way. For example, when the users U1 and U2 approach a certain distance (the first position) from a specific door where the appearance range E1 is set, the information processing device 200 determines to display the AR objects V1 and V2. When it is determined to display the AR objects V1 and V2, the information processing device 200 sends a display start trigger to the mobile terminals 100a and 100b. The display start trigger includes, for example, the type of the AR object to be displayed (the type of the character), the appearance position, and the destination position (the second position). The destination position is, for example, the position of the above AR object V3. In addition, the display start trigger may include the time to reach the destination position.

[0050] Examples of the conditions for determining whether to display the AR objects V1 and V2 include, in addition to the above determination based on the user position, the condition that a specific location falls within the user's viewing angle, and the condition of reaching a specific time due to the progress of a multiplayer game scenario. Examples of the specific time include the time of defeating a specific enemy in a multiplayer game, the time of completing a mission, etc.

[0051] Upon receiving the display start trigger from the information processing device 200, the mobile terminals 100a and 100b respectively determine the types and display positions of the AR objects V1 and V2 based on the display start trigger. First, for the display positions of the AR objects V1 and V2, a position adjustable range is preset in the gaming environment (area). The position adjustable range is set as a plane or a space. It is set using, for example, a shape (plane / sphere / cube, etc.), position (x, y, z), and size (x, y, z). In this article, the appearance range E1 shown below is set. Figure 1 The appearance range E1 shown.

[0052] The mobile terminals 100a and 100b determine whether the user's position, orientation, and perspective collide with the appearance range E1, which is the set position adjustable range, based on the display start trigger. The mobile terminals 100a and 100b calculate the positions that intersect with the respective perspectives A1 and A2 of the users U1 and U2 within the appearance range E1. The mobile terminals 100a and 100b determine the calculated respective positions as the display positions of the AR objects V1 and V2. The mobile terminals 100a and 100b display the AR objects V1 and V2 at the so-determined display positions respectively. At this time, the AR objects V1 and V2 may be different from each other in terms of display time.

[0053] When there is no position within the appearance range E1 that intersects with the perspectives A1 and A2 of the users U1 and U2, the mobile terminals 100a and 100b can repeat the calculation of the intersection position for a predetermined period, for example, about several seconds. In this case, when the user turns his / her face towards the appearance range E1 within the predetermined period, the display position of the AR object can be determined. On the other hand, when no intersection position is found within the predetermined period, the collision determination can be terminated, and the AR object can be displayed at the default position. Note that the default position can be, for example, the center of the appearance range E1.

[0054] After displaying the AR objects V1 and V2 within the appearance range E1, the mobile terminals 100a and 100b respectively move the so-displayed AR objects V1 and V2 to the destination positions. For example, the mobile terminal 100a moves the AR object V1 to the position of the AR object V3 through the movement path R1. Similarly, for example, the mobile terminal 100b moves the AR object V2 to the position of the AR object V3 through the movement path R2.

[0055] At this time, each of the mobile terminals 100a and 100b adjusts the moving speed and path of the corresponding one of the AR objects V1 and V2 based on the time to reach the destination position included in the display start trigger, and moves the AR objects V1 and V2. Note that when the display start trigger does not include the time to reach the destination position, each of the mobile terminals 100a and 100b moves the corresponding one of the AR objects V1 and V2 using a default speed or path. In addition, when the mobile terminals 100a and 100b asynchronously move the AR objects V1 and V2 - that is, asynchronously start an animation (moving image) - they can adjust the reproduction speed and number of repetitions of the animation so that the animations enter a synchronized state. In addition, depending on the length of time to reach the destination position, the animation to be reproduced can be changed, for example, to 5 seconds for user U1 and 10 seconds for user U2. When each of the mobile terminals 100a and 100b moves the corresponding one of the AR objects V1 and V2 to the position of the AR object V3, it sends a notification of completion of the movement to the information processing device 200.

[0056] When the information processing device 200 receives the notification of completion of the movement from the mobile terminals 100a and 100b, it sends a notification of state transition of the AR object to the mobile terminals 100a and 100b. Based on the received notification of state transition, the mobile terminals 100a and 100b start the synchronization process for the AR object V3.

[0057] As described above, based on the position information of the mobile terminals 100a and 100b, the information processing device 200 causes the AR objects that have been first displayed at positions slightly different according to the gaze points of the users U1 and U2 to be displayed at the destination position. This enables each user to experience a specific scene in the same way even using a display device with a limited viewing angle. That is, each user can experience a multi-person application using AR technology without missing necessary scenes as much as possible.

[0058] [1-2. Structure of the System According to the First Embodiment]

[0059] Figure 2 and Figure 3 is a diagram showing an example of the information processing system according to the first embodiment. As Figure 2 shown, the information processing system 1 includes a plurality of mobile terminals 100a to 100c and an information processing device 200. The mobile terminals 100a to 100c and the information processing device 200 are connected via, for example, the same network. The mobile terminals 100a to 100c are information processing devices worn by the users U1 to U3 and used by the users U1 to U3 to play multi-person games and the like. The information processing device 200 is an information processing device responsible for controlling the progress of the multi-person game and sending commands or data to the mobile terminals 100a to 100c according to the progress of the scene or the application state. Note thatFigure 3 Similar to the information processing system 2 shown, the mobile terminal 300 of user U4 can also be used as the information processing device 200. In this case, users U1 to U4 are players in a multiplayer game or the like.

[0060] [1-3. Structure of the mobile terminal according to the first embodiment]

[0061] Figure 4 FIG. is a diagram showing an example of the structure of the mobile terminal according to the first embodiment. As Figure 4 shown, the mobile terminal 100 includes a display 101, a speaker 102, a communication section 103, an operation input section 104, a sensor 110, a memory 120, and a controller 130.

[0062] The display 101 is, for example, a binocular optical see-through HMD, and is located in the field of view of the user wearing the mobile terminal 100 to superimpose an AR object on the real space and display the AR object superimposed on the real space. Note that various HMDs such as a monocular optical see-through HMD, a video see-through HMD, a retinal projection HMD, and an immersive HMD can be used as the display 101.

[0063] A description of how to wear the mobile terminal 100 will be given below with reference to Figure 5 FIG. Figure 5 FIG. is a diagram showing an example of how to wear the mobile terminal according to the first embodiment. As Figure 5 shown, when the user wears the mobile terminal 100 like wearing glasses, the display 101 is located in front of the user's eyes. The display 101 includes a right-eye display 101a and a left-eye display 101b, and the right-eye display 101a and the left-eye display 101b each display a corresponding image. In addition, the speaker 102 can be an earphone worn on the user's ear.

[0064] Returning to the reference Figure 4 description. For example, a binaural earphone can be used as the speaker 102. The speaker 102 enables the user to listen to the sound of a multiplayer game and at the same time communicate with another user.

[0065] The communication section 103 is implemented by a communication module or the like that is suitable for, for example, a wireless local area network (LAN) such as Wi-Fi (registered trademark). The communication section 103 is a communication interface responsible for controlling information communication with the information processing device 200. The communication section 103 receives a display start trigger and a notification regarding state transition from the information processing device 200. The communication section 103 outputs the received display start trigger and the notification regarding state transition to the controller 130. In addition, the communication section 103 sends the position information and the notification regarding movement completion input from the controller 130 to the information processing device 200.

[0066] The operation input section 104 is, for example, a button that receives user input. The operation input section 104 outputs the user input to the controller 130 as operation information. In addition, the operation input section 104 may support gesture input by hand recognition using the camera of the sensor 110.

[0067] The sensor 110 includes various sensors that capture the scene around the user wearing the mobile terminal 100. The sensor 110 includes an environmental camera 111, a depth sensor 112, a gyro sensor 113, an acceleration sensor 114, an azimuth sensor 115, and a position sensor 116.

[0068] The environmental camera 111 is, for example, a monochrome stereo camera and captures the scene in front of the mobile terminal 100. The environmental camera 111 uses an imaging element such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor to capture an image. The environmental camera 111 performs photoelectric conversion on the light received by the imaging element and performs analog / digital (A / D) conversion on the result of the photoelectric conversion to generate an image. The environmental camera 111 outputs the captured image to the controller 130, and the captured image is a stereo image. The captured image output from the environmental camera 111 is used for self-position estimation using simultaneous localization and mapping (SLAM) technology. Note that the environmental camera 111 may be a camera equipped with a wide-angle lens or a fish-eye lens.

[0069] The depth sensor 112 is, for example, a monochrome stereo camera similar to the environmental camera 111 and captures the scene in front of the mobile terminal 100. The depth sensor 112 outputs the captured image to the controller 130, and the captured image is a stereo image. The captured image output from the depth sensor 112 is used to measure the distance to the subject in the user's line of sight. Note that a time-of-flight (TOF) sensor may be used as the depth sensor 112.

[0070] The gyro sensor 113 is a sensor that detects the orientation of the mobile terminal 100 - that is, the orientation of the user. For example, a vibrating gyro sensor may be used as the gyro sensor 113.

[0071] The acceleration sensor 114 is a sensor that detects the acceleration in each direction of the mobile terminal 100. For example, a triaxial acceleration sensor such as a piezoresistive acceleration sensor or a capacitive acceleration sensor may be used as the acceleration sensor 114.

[0072] The orientation sensor 115 is a sensor that detects the orientation of the mobile terminal 100. For example, a magnetic sensor can be used as the orientation sensor 115. The gyro sensor 113, the acceleration sensor 114, and the orientation sensor 115 are jointly used as a nine-degree-of-freedom (nine-axis sensor) to output first position information to the controller 130.

[0073] The position sensor 116 is a sensor that detects the position of the mobile terminal 100 - that is, the position of the user. The position sensor 116 is, for example, a global positioning system (GPS) receiver that generates second position information including longitude, latitude, and altitude based on received GPS signals. The position sensor 116 outputs the thus-generated second position information to the controller 16. Note that the second position information is used, for example, to switch maps obtained as a result of dividing each area. When map switching is not required, the position sensor 116 can be removed.

[0074] The memory 120 is implemented by a semiconductor storage element (e.g., random access memory (RAM) or flash memory) or a storage device (e.g., hard disk or optical disc). The memory 120 stores an environmental map for calculating the user's position. The environmental map is, for example, a three-dimensional map using a world coordinate system that is pre-generated using SLAM technology. In addition, the memory 120 temporarily stores captured images or position information obtained from the sensor 110, and stores information (programs and data) for processing in the controller 130.

[0075] The controller 130 is implemented by executing a program stored in an internal storage device on a RAM used as a working area through, for example, a central processing unit (CPU), a microprocessing unit (MPU), etc. Alternatively, the controller 130 can be implemented by an integrated circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0076] The controller 130 includes an environmental information acquisition section 131, a position information calculation section 132, an application execution section 133, and an output controller 134, and implements or executes the functions or actions of information processing to be described below. Note that the internal structure of the controller 130 is not limited to Figure 4 the structure shown, and can be a different structure as long as it executes the information processing to be described later.

[0077] When the environmental information acquisition section 131 receives a captured image as a stereo image input from the depth sensor 112, it calculates the distance (depth) to the subject in the image based on the captured image. The environmental information acquisition section 131 generates a distance image including the calculated distance (depth). The environmental information acquisition section 131 outputs the generated distance image to the application execution section 133. Note that the environmental information acquisition section 131 can calculate the distance (depth) to the subject based on sensor information according to methods such as motion parallax, TOF, or structured light in addition to based on the stereo image.

[0078] A captured image as a stereo image is input from the environmental camera 111 to the position information calculation section 132. In addition, first position information is input from the gyro sensor 113, the acceleration sensor 114, and the azimuth sensor 115 to the position information calculation section 132, and second position information is input from the position sensor 116 to the position information calculation section 132. The position information calculation section 132 selects an environmental map stored in the memory 120 based on the received second position information. The position information calculation section 132 refers to the selected environmental map and performs its own position estimation based on the input captured image according to map matching using SLAM. Note that according to the present embodiment, only a limited number of environmental maps are available, so that the position of the area where the environmental map is located is estimated in the self-position estimation according to map matching.

[0079] The position information calculation section 132 estimates the distance and direction from the position estimated according to map matching based on the input first position information, and calculates the position information about the user. The position information about the user includes information about the position of the area where the user is located and direction information indicating which direction the user is facing. The position information calculation section 132 sends the calculated position information about the user to the information processing device 200 via the communication section 103. In addition, the position information calculation section 132 outputs the calculated position information about the user to the application execution section 133.

[0080] The application execution section 133 executes applications such as multi-player games. The distance image is input from the environmental information acquisition section 131 to the application execution section 133. In addition, the position information about the user is input from the position information calculation section 132 to the application execution section 133.

[0081] The application execution unit 133 outputs data about the AR object and audio information generated based on commands and data, such as a scene, a distance image, and position information about the user received from the information processing device 200, to the output controller 134. For example, when the application execution unit 133 receives a display start trigger from the information processing device 200, it determines the type and display position of the AR object based on the display start trigger, the distance image, and the position information about the user. The application execution unit 133 estimates the distance between the user and the appearance range E1, which is the set position adjustable range, based on the distance image.

[0082] The application execution unit 133 determines whether the position, orientation, and viewing angle of the user collide with the appearance range E1, which is the set position adjustable range. The application execution unit 133 calculates the position where the viewing angle of the user intersects within the appearance range E1. Note that the viewing angle of the user is preset in this text. The application execution unit 133 determines the calculated position as the display position of the AR object. In addition, the application execution unit 133 determines the size of the AR object based on the estimated distance.

[0083] The application execution unit 133 moves the AR object from the display position within the appearance range E1 to the destination position. At this time, the application execution unit 133 adjusts the moving speed or path of the AR object based on the time to reach the destination position, and then moves the AR object. After moving the AR object to the destination position, the application execution unit 133 sends a notification of the completion of the movement to the information processing device 200 via the communication unit 103.

[0084] When the application execution unit 133 receives a notification of the state transition of the AR object from the information processing device 200 via the communication unit 103, it starts the synchronization process of the AR object at the destination position based on the received notification of the state transition.

[0085] When the output controller 134 receives the data about the AR object and the audio information input from the application execution unit 133, it displays the AR object on the display 101 and outputs the audio information to the speaker 102. The output controller 134 displays the data about the AR object input from the application execution unit 133 on the display 101, and outputs the audio information to the speaker 102, for example, according to the progress of the scene or the movement of the AR object based on the display start trigger.

[0086] [1-4. Structure of the information processing device according to the first embodiment]

[0087] Next, a description of the structure of the information processing device 200 will be given with reference to Figure 6 A description of the structure of the information processing device 200 will be given. Figure 6 is a diagram showing an example of the structure of the information processing device according to the first embodiment. AsFigure 6 As shown, the information processing apparatus 200 includes a communication section 201, a memory 210, and a controller 220.

[0088] The communication section 201 is implemented by a communication module suitable for wireless LAN such as Wi-Fi (registered trademark), etc. The communication section 201 is a communication interface responsible for controlling information communication with the mobile terminal 100. The communication section 201 receives location information and a notification of completion of movement from the mobile terminal 100. The communication section 201 outputs the received location information and the notification of completion of movement to the controller 220. In addition, the communication section 201 sends a display start trigger and a notification of state transition input from the controller 220 to the mobile terminal 100.

[0089] The memory 210 is implemented by, for example, a semiconductor storage element (such as a RAM or a flash memory) or a storage device (such as a hard disk or an optical disk). The memory 210 stores scenes for multiplayer games, etc., as well as commands and data suitable for the progress of the scene or the application state of the mobile terminal 100. In addition, the memory 210 stores information (programs and data) for processing in the controller 220.

[0090] The controller 220 is implemented by executing a program stored in an internal storage device on a RAM used as a work area through, for example, a CPU, an MPU, etc. Alternatively, the controller 220 may be implemented by an integrated circuit such as an ASIC or an FPGA. The controller 220 includes a determination section 221 and a display controller 222, and implements or executes the functions or actions of information processing to be described below. Note that the internal structure of the controller 220 is not limited to Figure 6 the structure shown, and may be a different structure as long as it executes the information processing to be described later.

[0091] When receiving the location information from the mobile terminal 100 via the communication section 201, the determination section 221 determines whether to display an AR object based on the received location information. When determining to display the AR object, the determination section 221 instructs the display controller 222 to generate a display start trigger. When determining not to display the AR object, the determination section 221 repeats the determination of whether to display the AR object based on the received location information.

[0092] In addition, a determination unit 221 determines whether the mobile terminal 100b (second mobile terminal) is at the position of the mobile terminal 100a (first mobile terminal) for which a determination has been made as to whether to display an AR object. When the determination unit 221 determines that the mobile terminal 100b is at the position of the mobile terminal 100a, the determination unit 221 instructs the display controller 222 to also generate a display start trigger for the mobile terminal 100b. That is, when the determination unit 221 determines that the plurality of mobile terminals 100 have experienced the same event in the scene, the determination unit 221 instructs the display controller 222 to generate a display start trigger for each mobile terminal 100. In other words, the determination unit 221 starts the action of the AR object at different times for each mobile terminal 100.

[0093] When the display controller 222 receives an instruction to generate a display start trigger from the determination unit 221, the display controller 222 generates a display start trigger for each mobile terminal 100, including the type of the AR object to be displayed (type of the character), the appearance position, the destination position, and the time to reach the destination position. The display controller 222 sends the thus generated display start trigger to the mobile terminal 100 via the communication unit 201.

[0094] When the display controller 222 receives a notification of completion of movement from all the mobile terminals 100 to which the display start trigger has been sent via the communication unit 201, the display controller 222 sends a notification of state transition of the AR object to all the mobile terminals 100 from which the notification of completion of movement has been received. The notification of state transition includes, for example, a notification of start of synchronization, the position and state of the AR object. That is, in each mobile terminal 100 that has received the notification of state transition, the AR object is displayed in a specific scene in the case of synchronization of position, movement, etc.

[0095] [1-5. Information processing flow according to the first embodiment]

[0096] Figure 7 is a sequence diagram showing the processing flow according to the first embodiment. Note that Figure 7 shows an example of the following situation: In the mobile terminals 100a and 100b of the users U1 and U2, the AR objects that have started asynchronous movement are synchronized to stop moving at the end points, and the next scene is displayed using the AR objects synchronized with each other. Note that since the mobile terminals 100a and 100b operate in the same manner, the mobile terminals 100a and 100b are collectively referred to as the mobile terminal 100 below.

[0097] The environmental information acquisition section 131 of the mobile terminal 100 generates a distance image based on the captured image input from the depth sensor 112, and outputs the generated distance image to the application execution section 133. In addition, the position information calculation section 132 selects an environmental map stored in the memory 120 based on the second position information input from the position sensor 116. The position information calculation section 132 refers to the selected environmental map, and performs its own position estimation based on the captured image input from the environmental camera 111 according to map matching. The position information calculation section 132 estimates the distance and direction from the position estimated according to map matching based on the first position information input from the gyro sensor 113, the acceleration sensor 114, and the azimuth sensor 115, and calculates the position information about the user. The position information calculation section 132 starts sending the calculated position information about the user to the information processing device 200 (step S1). In addition, the position information calculation section 132 outputs the calculated position information about the user to the application execution section 133.

[0098] When receiving the position information from the mobile terminal 100, the determination section 221 of the information processing device 200 determines whether to display the AR object based on the received position information (step S2). The determination section 221 instructs the display controller 222 to generate a display start trigger based on the determination result regarding whether to display the AR object.

[0099] When receiving the instruction to generate a display start trigger from the determination section 221, the display controller 222 generates a display start trigger for each mobile terminal 100. The display controller 222 sends the generated display start trigger to each mobile terminal 100 (step S3).

[0100] When receiving the display start trigger from the information processing device 200, the application execution section 133 of the mobile terminal 100 determines the type and display position of the AR object based on the display start trigger, the distance image, and the position information about the user (step S4). The application execution section 133 starts moving the AR object to the destination position (step S5). After moving the AR object to the destination position, the application execution section 133 sends a notification of movement completion to the information processing device 200 (step S6).

[0101] When the display controller 222 of the information processing apparatus 200 receives a notification of movement completion from all the mobile terminals 100 of the users to which a display start trigger has been sent, the display controller 222 sends a notification of state transition of the AR object to the mobile terminal 100 from which the notification of movement completion has been received (step S7). Note that, for example, when the display controller 222 receives a notification of movement completion from the mobile terminals 100 of most of the users to which a display start trigger has been sent, the display controller 222 may send a notification of state transition to the mobile terminals 100 to which a display start trigger has been sent.

[0102] When the application execution section 133 of the mobile terminal 100 receives a notification of state transition of the AR object from the information processing apparatus 200, the application execution section 133 starts synchronization processing of the AR object at the destination location based on the received notification of state transition (step S8). This enables each user to experience a specific scenario in the same manner even when using the mobile terminal 100 which is a display device with a limited viewing angle.

[0103] [1-6. Effects]

[0104] As described above, each user can experience a specific scenario in the same manner even when using the mobile terminal 100 which is a display device with a limited viewing angle. That is, each user can experience a multi-user application using AR technology without missing necessary scenarios as much as possible. This also enables natural synchronization and stop of the movement of the AR object that has started asynchronously among users at the target location. This also enables the AR object to appear according to the current position of each user.

[0105] (2. Second Embodiment)

[0106] Meanwhile, according to the first embodiment, a specific object such as a door in the real space is set as the appearance range, but a wider range can be set as the appearance range, and such a case will be described as the second embodiment. Note that, according to the second embodiment, the mobile terminal 100 and the information processing apparatus 200 having the same structure as the mobile terminal 100 and the information processing apparatus 200 according to the first embodiment are used, and thus a description of the common structure and operation will not be given.

[0107] [2-1. Example of Appearance Range According to Second Embodiment]

[0108] Figure 8 is a diagram showing an example of information processing according to the second embodiment. In Figure 8In the example shown, when users U1 and U2 approach a specific location on the road within the area, AR objects V3 and V4 appear from the appearance range E2. Note that AR object V3 is the AR object displayed on the mobile terminal 100a of user U1, and AR object V4 is the AR object displayed on the mobile terminal 100b of user U2.

[0109] The appearance range E2 is set on the entire wall surface of the building corresponding to one block of the street in the real space. In this case, the information processing device 200 adjusts the height of the AR objects V3 and V4, the start time of the appearance animation, and the time to reach the destination position according to the positions where users U1 and U2 are viewing. As described above, the appearance range E2 can have a wider adjustable range of appearance positions than the appearance range E1 according to the first embodiment. In addition, the appearance range can be set, for example, on the floor of one block of the street in the real space according to the location or scene.

[0110] In other words, according to the second embodiment, the surface in the real space on which the appearance range is set is specifically a wall or a floor in the real space. The surface in the real space can be regarded as a stationary surface in the real space having at least a specific area detected by sensors including the depth sensor 112, etc. The location in the real space is not limited to a smooth surface - that is, a flat surface, and can have irregularities. Alternatively, the surface in the real space can be a curved surface. The surface in the real space according to the second embodiment is an area stably detected by the sensors, so that the surface can be appropriately used as an area for setting AR objects.

[0111] (3. Third Embodiment)

[0112] In addition, when it is desired to make an AR object appear at a specific location, the appearance range can be restricted to the specific location, and such a case will be described as the third embodiment. Note that according to the third embodiment, the mobile terminal 100 and the information processing device 200 having the same structure as the mobile terminal 100 and the information processing device 200 according to the first embodiment are used, and thus the description of the common structure and operation will not be given.

[0113] [3-1. Example of the Appearance Range According to the Third Embodiment]

[0114] Figure 9 is a diagram showing an example of the information processing according to the third embodiment. In Figure 9In the example shown, when the user approaches the door of the building, the AR object V5 appears within the appearance range E3 set on the lighting fixture installed above the door. The appearance range E3 is restricted to a specific location, namely the lighting fixture. In this case, the information processing device 200 adjusts the start time of the appearance animation and the reproduction time of the animation without adjusting the position of the AR object V5. For example, in order to display the AR object V5 on the mobile terminals 100a and 100b of the users U1 and U2, when the users U1 and U2 each approach a predetermined position relative to the appearance range E3, the information processing device 200 starts to make the animation appear. Subsequently, for example, when the AR object V5 is an enemy in a multiplayer game and a battle start animation appears, synchronization starts between the mobile terminals 100a and 100b. As described above, even when the appearance range is restricted, the animations that start asynchronously according to the user position can be synchronized at a predetermined time.

[0115] (4. Modifications of the Embodiment)

[0116] The processing according to each of the above embodiments can be executed in various different modes other than the above embodiments.

[0117] In each of the above embodiments, an example has been given in which the mobile terminal 100 includes a processor such as the controller 130. The mobile terminal 100 can be divided into, for example, a glasses-type interface part, a computing part including the controller 130, and an operation part that receives input operations from the user, etc. In addition, as described according to each embodiment, the mobile terminal 100 including the display 101 having transparency and held in the user's line of sight direction is a so-called AR glasses. The mobile terminal 100 can be a device that communicates with the display 101 used as an external display and performs display control on the display 101. In addition, the mobile terminal 100 can be, for example, a smart phone held by the user in front of his / her eyes.

[0118] In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown above and in the drawings can be changed as needed. For example, the various types of information shown in each figure are not limited to the information shown.

[0119] Furthermore, each component of each device shown in the drawings is a functional concept, and is not necessarily physically configured as shown in the drawings. That is, the specific form of the allocation and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically allocated and integrated into any unit according to various loads, usage conditions, etc. For example, Figure 3 like the mobile terminal 300 shown, the functions of the mobile terminal 100 and the information processing device 200 can be integrated.

[0120] In addition, the above-described embodiments and modifications can be combined as needed to the extent that the processing contents do not conflict with each other.

[0121] (5. Hardware Structure)

[0122] For example, the information device according to each of the above embodiments, such as the mobile terminal 100 or the information processing apparatus 200, is implemented by a computer 1000 having a structure as Figure 10 shown. A description will be given below based on an example of the information processing apparatus 200 according to the first embodiment. Figure 10 FIG. 10 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an I / O interface 1600. The components of the computer 1000 are connected via a bus 1050.

[0123] The CPU 1100 operates according to programs stored in the ROM 1300 or the HDD 1400 to control each component. For example, the CPU 1100 loads a program stored in the ROM 1300 or the HDD 1400 onto the RAM 1200 and executes processing associated with each of the various programs.

[0124] The ROM 1300 stores a boot program, such as a basic input / output system (BIOS) to be executed by the CPU 1100 when the computer 1000 is put into operation, a program suitable for the specific hardware of the computer 1000, and the like.

[0125] The HDD 1400 is a computer-readable recording medium that records programs to be executed by the CPU 1100, data to be used by the programs, and the like in a non-transitory manner. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure, and the information processing program is an example of the program data 1450.

[0126] The communication interface 1500 is an interface for the computer 1000 to establish a connection with an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or sends data generated by the CPU 1100 to another device.

[0127] The I / O interface 1600 is an interface for the computer 1000 to establish a connection with the I / O device 1650. For example, the CPU 1100 of the computer 1000 receives data from an input device such as a keyboard or a mouse via the I / O interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the I / O interface 1600. Further, the I / O interface 1600 can be used as a medium interface for reading programs and the like recorded on a predetermined recording medium (medium). Examples of the medium include optical recording media (such as digital versatile discs (DVDs) or phase change rewritable discs (PDs)), magneto-optical recording media (such as magneto-optical (MO) discs), tape media, magnetic recording media, semiconductor memories, and the like.

[0128] For example, when the computer 1000 is used as the information processing device 200 according to the first embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded on the RAM 1200 to implement functions such as the determination unit 221. In addition, the HDD 1400 stores the information processing program and data according to the present disclosure in the memory 210. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, or alternatively, as another example, the CPU 1100 can obtain such a program from another device through the external network 1550.

[0129] [6. Effects]

[0130] The information processing device 200 includes a control device (controller 220). The control device determines whether the first mobile terminal (mobile terminal 100a) is at a first position in the real space, and based on the determination that the first mobile terminal is at the first position, causes the first mobile terminal to start superimposing a virtual object on the real space according to the first position, and determines whether a second mobile terminal (mobile terminal 100b) different from the first mobile terminal is at the first position, and based on the determination that the second mobile terminal is at the first position, causes the second mobile terminal to start superimposing the virtual object on the real space. Further, the control device causes the first mobile terminal to start a first action of the virtual object and causes the second mobile terminal to start a second action of the virtual object differently at least according to the display start time or the display start position of the virtual object, and causes the first action and the second action to end synchronously.

[0131] This allows the information processing device 200 to enable each user to experience a specific scenario in the same way even using a display device with a limited viewing angle.

[0132] The first action of the virtual object corresponds to a movement to a second position in the real space different from the first position.

[0133] This allows the information processing device 200 to enable each user to experience the multiplayer game application using AR technology without missing necessary scenarios as much as possible.

[0134] The second action of the virtual object corresponds to the movement from a display start position different from the display start position of the first action to a second position.

[0135] This allows the information processing device 200 to naturally synchronize and stop the movement of the virtual object that has started asynchronously among users at the destination position.

[0136] The virtual object corresponds to a path in which the movement path corresponding to the first action and the movement path corresponding to the second action are different from each other.

[0137] This allows the information processing device 200 to cause the virtual object to appear according to the current position of each user.

[0138] The control device synchronizes the first mobile terminal and the second mobile terminal to move the virtual object after the first action and the second action are completed.

[0139] This allows the information processing device 200 to enable each user to experience a specific scenario in the same way.

[0140] The second action of the virtual object corresponds to the reproduction of a moving image in which at least one of the reproduction speed, the number of repetitions, and the length is different from the first action.

[0141] This allows the information processing device 200 to naturally synchronize and stop the movement of the virtual object that has started asynchronously among users at the destination position.

[0142] The display start position is set on a plane in the real space.

[0143] This allows the information processing device 200 to cause the virtual object to appear according to the current position of each user.

[0144] The display start position is set within a range corresponding to an object in the real space.

[0145] This allows the information processing device 200 to cause the virtual object to appear from an object in the real space according to the current position of each user.

[0146] The first action of the virtual object and the second action of the virtual object each correspond to the display of an image at the display start position.

[0147] This allows the information processing device 200 to cause a virtual object whose appearance position is restricted to a predetermined position in the real space to appear for each user.

[0148] Note that the effects described in this document are merely examples and should not be construed restrictively, and other effects may be provided.

[0149] Note that this technology may also have the following configurations.

[0150] (1) An information processing apparatus including a control device configured to:

[0151] Determine whether a first mobile terminal is at a first position in the real space, and

[0152] Based on the determination that the first mobile terminal is at the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and

[0153] Determine whether a second mobile terminal different from the first mobile terminal is at the first position, and

[0154] Based on the determination that the second mobile terminal is at the first position, control the second mobile terminal to start superimposing the virtual object on the real space, where

[0155] The control device:

[0156] Regarding at least one of the display start time of the virtual object and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and

[0157] Synchronize the end of the first action and the end of the second action.

[0158] (2) The information processing apparatus according to (1), wherein

[0159] The first action of the virtual object is a movement to a second position in the real space different from the first position.

[0160] (3) The information processing apparatus according to (2), wherein

[0161] The second action of the virtual object is a movement from a display start position different from the display start position of the first action to the second position.

[0162] (4) The information processing apparatus according to (3), wherein

[0163] The movement path of the virtual object corresponding to the first action and the movement path of the virtual object corresponding to the second action are different paths from each other.

[0164] (5) The information processing apparatus according to (1), wherein

[0165] the control device causes the first mobile terminal and the second mobile terminal to synchronize the actions on the virtual object after the first action and the second action are completed.

[0166] (6) The information processing apparatus according to (2), wherein

[0167] the second action of the virtual object is the reproduction of a moving image in which at least one of the reproduction speed, the number of repetitions, and the length is different from the first action.

[0168] (7) The information processing apparatus according to (1), wherein

[0169] the display start position is set on a surface in the real space.

[0170] (8) The information processing apparatus according to (1), wherein

[0171] the display start position is set within a range corresponding to an object in the real space.

[0172] (9) The information processing apparatus according to (1), wherein

[0173] the first action of the virtual object and the second action of the virtual object are the display of an image at the display start position.

[0174] (10) An information processing method, comprising causing a control device to perform the following processing:

[0175] determining whether a first mobile terminal is at a first position in the real space, and

[0176] based on the determination that the first mobile terminal is at the first position, controlling the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and

[0177] determining whether a second mobile terminal different from the first mobile terminal is at the first position, and

[0178] based on the determination that the second mobile terminal is at the first position, controlling the second mobile terminal to start superimposing the virtual object on the real space, wherein

[0179] the processing performed by the control device includes:

[0180] Regarding at least one of the display start time and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and

[0181] Synchronize the end of the first action and the end of the second action.

[0182] (11) A recording medium that records an information processing program, the information processing program causing a control device to execute the following processing:

[0183] Determine whether a first mobile terminal is at a first position in the real space, and

[0184] Based on the determination that the first mobile terminal is at the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and

[0185] Determine whether a second mobile terminal different from the first mobile terminal is at the first position, and

[0186] Based on the determination that the second mobile terminal is at the first position, control the second mobile terminal to start superimposing the virtual object on the real space, where

[0187] The processing executed by the control device includes:

[0188] Regarding at least one of the display start time and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and

[0189] Synchronize the end of the first action and the end of the second action.

[0190] List of reference numerals

[0191] 100, 100a, 100b Mobile terminal

[0192] 101 Display

[0193] 103 Communication section

[0194] 110 Sensor

[0195] 120 Memory

[0196] 130 Controller

[0197] 131 Environment information acquisition section

[0198] 132 Location Information Calculation Section

[0199] 133 Application Execution Section

[0200] 134 Output Controller

[0201] 200 Information Processing Device

[0202] 201 Communication Section

[0203] 210 Memory

[0204] 220 Controller

[0205] 221 Determination Section

[0206] 222 Display Controller

[0207] Viewpoints A1 and A2

[0208] Appearance Ranges E1 to E3

[0209] AR Objects V1 to V5

[0210] Users U1 to U4

Claims

1. An information processing apparatus, including a control device, the control device being configured to: Determine whether a first mobile terminal is at a first position in the real space, and Based on the determination that the first mobile terminal is at the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and Determine whether a second mobile terminal different from the first mobile terminal is at the first position, and Based on the determination that the second mobile terminal is at the first position, control the second mobile terminal to start superimposing the virtual object on the real space, where The control device: Regarding at least one of the display start time of the virtual object and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and Cause the end of the first action and the end of the second action to end synchronously.

2. The information processing apparatus according to claim 1, wherein The first action of the virtual object is a movement to a second position different from the first position in the real space.

3. The information processing apparatus according to claim 2, wherein The second action of the virtual object is a movement from a display start position different from the display start position of the first action to the second position.

4. The information processing apparatus according to claim 3, wherein The movement path of the virtual object corresponding to the first action and the movement path of the virtual object corresponding to the second action are different paths from each other.

5. The information processing apparatus according to claim 1, wherein The control device causes the actions of the virtual object by the first mobile terminal and the second mobile terminal to be synchronized after the end of the first action and the second action.

6. The information processing apparatus according to claim 2, wherein The second action of the virtual object is the reproduction of a dynamic image in which at least one of the reproduction speed, the number of repetitions, and the length is different from the first action.

7. The information processing apparatus according to claim 1, wherein The display start position is set on a surface in the real space.

8. The information processing apparatus according to claim 1, wherein The display start position is set within a range corresponding to an object in the real space.

9. The information processing apparatus according to claim 1, wherein The first action of the virtual object and the second action of the virtual object are the display of an image at the display start position.

10. An information processing method, including causing a control device to perform the following processing: Determine whether a first mobile terminal is at a first position in the real space, and Based on the determination that the first mobile terminal is at the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and Determine whether a second mobile terminal different from the first mobile terminal is at the first position, and Based on the determination that the second mobile terminal is in the first position, control the second mobile terminal to start superimposing the virtual object on the real space, where the processing executed by the control device includes: Regarding at least one of the display start time of the virtual object and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and synchronize the end of the first action and the end of the second action.

11. A recording medium that records an information processing program, where the information processing program causes a control device to execute the following processing: Determine whether a first mobile terminal is in a first position in the real space, and Based on the determination that the first mobile terminal is in the first position, control the first mobile terminal to start superimposing a virtual object on the real space in response to the first position, and Determine whether a second mobile terminal different from the first mobile terminal is in the first position, and Based on the determination that the second mobile terminal is in the first position, control the second mobile terminal to start superimposing the virtual object on the real space, where the processing executed by the control device includes: Regarding at least one of the display start time of the virtual object and the display start position of the virtual object, perform the following operations differently: cause the first mobile terminal to start a first action of the virtual object and cause the second mobile terminal to start a second action of the virtual object, and synchronize the end of the first action and the end of the second action.

Images