Information processing device and program

Abstract

To let a plurality of character objects operate in a mode corresponding to a combination of a character object which can be displayed on one terminal and a character object which can be displayed on another terminal.SOLUTION: An information processing apparatus as one embodiment of the present invention includes: a determination section which determines, based upon a first character object which can be displayed on a first terminal and a second character object which can be displayed on a second terminal, operation modes of the first character object and the second character object; and a display control section which controls the first terminal so that the first character object and the second character object are displayed over an image captured by the first terminal according to the operation modes.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] In mobile terminals such as smartphones and game machines, a technology of augmented reality (AR: Augmented Reality) (hereinafter referred to as AR technology) has been used. The AR technology is a technology that superimposes an object arranged in a virtual space and an image obtained by imaging the real space with a camera and displays the result on a display. By this technology, an object on the virtual space can be made to appear as if it exists in the real space. The AR technology is disclosed in, for example, Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As an application example of the AR technology, a 3D virtual character object owned by a user is displayed on a mobile terminal so as to exist in the real space. In addition to a user using a character object owned by himself / herself alone, it is desired that a user can use a character object in cooperation with other users.

[0005] One object of the present invention is to provide an information processing apparatus and a program capable of operating a plurality of character objects in a form corresponding to a combination of a character object that can be displayed on one terminal and a character object that can be displayed on another terminal.

Means for Solving the Problems

[0006] According to one embodiment of the present invention, an information processing device is provided which includes: a determination unit that determines the operating modes of a first character object and a second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; and a display control unit that controls the first terminal so as to display the first character object and the second character object in accordance with the operating modes by superimposing them on an image captured by the first terminal.

[0007] The first terminal and the second terminal may satisfy predetermined related conditions.

[0008] The aforementioned related conditions may include the fact that the first terminal and the second terminal are closer than a predetermined distance apart.

[0009] The aforementioned related conditions may include the fact that the same object is included in the image captured by the first terminal and the image captured by the second terminal.

[0010] When neither the first terminal nor the second terminal satisfies the predetermined related conditions, the first character object that can be displayed on the first terminal may be an image based on a part of the operation mode, and the second character object that can be displayed on the second terminal may be an image based on a part of the operation mode.

[0011] In a state where neither the first terminal nor the second terminal satisfies the predetermined related conditions, the first character object that can be displayed on the first terminal may be an image based on a form not included in the operation mode, and the second character object that can be displayed on the second terminal may be an image based on a form not included in the operation mode.

[0012] The second terminal may further be controlled to display the first character object and the second character object superimposed on the image captured by the second terminal in accordance with the operation mode.

[0013] The display control unit may control the first terminal to display the first character object and the second character object using a coordinate system set for the image captured by the first terminal, superimpose the first character object and the second character object onto the image captured by the second terminal with respect to the coordinate system, and control the second terminal to display them in synchronization with the display on the first terminal.

[0014] The determination unit may further determine the operation mode based on the item object if there is an item object that can be displayed together with the first character object at the first terminal.

[0015] The aforementioned operating mode may correspond to an operating mode defined in data that defines the operating modes of a plurality of character objects, including the first character object and the second character object, for at least a portion of the period during which the first character object and the second character object are used.

[0016] Furthermore, according to one embodiment of the present invention, a program is provided for causing a computer to control the first terminal to determine the operating modes of the first character object and the second character object based on a first character object that can be displayed on the first terminal and a second character object that can be displayed on the second terminal, and to display the first character object and the second character object in accordance with the operating modes by superimposing them on an image captured by the first terminal. [Effects of the Invention]

[0017] According to an embodiment of the present invention, it is possible to provide an information processing apparatus and a program capable of operating a plurality of character objects in a form corresponding to a combination of a character object that can be displayed on one terminal and a character object that can be displayed on another terminal.

Brief Description of the Drawings

[0018] [Figure 1] It is a block diagram showing the configuration of the system in the first embodiment. [Figure 2] It is a diagram for explaining animation data in the first embodiment. [Figure 3] It is a diagram for explaining operation data in the first embodiment. [Figure 4] It is a diagram for explaining still data in the first embodiment. [Figure 5] It is a diagram for explaining set correspondence data in the first embodiment. [Figure 6] It is a diagram for explaining user management data in the first embodiment. [Figure 7] It is a diagram for explaining the relationship between the real space and the imaging direction in the first embodiment. [Figure 8] It is a diagram for explaining a display example in the single mode in the first embodiment. [Figure 9] It is a diagram for explaining the communication flow of service provision processing (pre-setting) in the first embodiment. [Figure 10] It is a diagram for explaining the user selection screen in the first embodiment. [Figure 11] It is a diagram for explaining the selection acceptance screen in the first embodiment. [Figure 12] It is a diagram for explaining the communication flow of service provision processing (multi-mode display control) in the first embodiment. [Figure 13] It is a diagram for explaining the playback instruction screen in the first embodiment. [Figure 14]This figure illustrates an example of display in multiple modes in the first embodiment (when image is taken along the first direction AR1). [Figure 15] This figure illustrates an example of display in multiple modes in the first embodiment (when image is taken along the second direction AR2). [Figure 16] This is a block diagram showing the configuration of the display control unit in the first embodiment. [Modes for carrying out the invention]

[0019] The system in one embodiment of the present invention will be described in detail below with reference to the drawings. The embodiments shown below are examples of embodiments of the present invention, and the present invention is not limited to these embodiments.

[0020] <First Embodiment> The system in the first embodiment of the present invention will be described in detail with reference to the drawings.

[0021] [overview] Figure 1 is a block diagram showing the configuration of the system in the first embodiment. The system 1000 includes a server 10 (information processing device) and a communication terminal 50. The server 10 and the communication terminal 50 are connected via a network NW such as the Internet. In this example, the communication terminal 50 is a smartphone. However, the communication terminal 50 may also be a portable device such as a tablet or a portable game console.

[0022] Although Server 10 is shown as a single device in Figure 1, it may consist of multiple devices. Server 10 provides an application program to the communication terminal 50 and provides various services to the communication terminal 50 that executes the application program. This application program is recorded on a recording medium (storage device) that Server 10 can connect to. The various services include the operation services of character objects, which are described below. The various services may also include, for example, SNS (Social Networking Service).

[0023] An application program for receiving services from the server 10 is installed on the communication terminal 50. This program allows the communication terminal 50 to display character objects using AR technology. Furthermore, in this example program, when certain conditions are met, the communication terminal 50 can display additional character objects and perform special actions on the multiple displayed character objects. The configuration of the communication terminal 50 will be described below.

[0024] [Hardware configuration of communication terminal 50] The communication terminal 50 includes a control unit 51, a storage unit 52, a communication unit 53, a position measuring unit 54, a behavior measuring unit 55, an imaging unit 56, a display unit 57, and an operation unit 58. The control unit 51 is an example of a computer that includes an arithmetic processing circuit such as a CPU. The control unit 51 executes application programs stored in the storage unit 52 using the CPU to realize various functions in the communication terminal 50.

[0025] The storage unit 52 includes a storage device and stores application programs downloaded and installed from the server 10 via the network NW. The storage unit 52 also stores various data used when the application programs are executed. This program may be provided recorded on a computer-readable recording medium such as a magnetic recording medium, optical recording medium, magneto-optical recording medium, or semiconductor memory. In this case, the communication terminal 50 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 52 included in the communication terminal 50, or it may be the medium used in the storage unit 52.

[0026] The communication unit 53 includes a communication module and, under the control of the control unit 51, connects to the network NW and transmits and receives information with other devices connected to the network NW. The position measurement unit 54 measures the position of the communication terminal 50 using GPS or other technologies and outputs position information (hereinafter sometimes referred to as position information) to the control unit 51. The behavior measurement unit 55 includes an acceleration sensor and a gyro sensor, etc., and measures the behavior of the communication terminal 50 and outputs behavior information (hereinafter sometimes referred to as behavior information) to the control unit 51. The behavior information includes, for example, information for identifying the amount of movement and rotation of the communication terminal 50. In addition, by using a geomagnetic sensor or the like in combination, information for identifying the orientation of the communication terminal 50 may be generated in the communication terminal 50.

[0027] The imaging unit 56 is a camera including a CCD sensor or a CMOS sensor, and outputs information indicating an image within the imaging range to the control unit 51. The information indicating an image can be either a still image or a moving image. The display unit 57 is a display device such as a liquid crystal display or an organic EL display, and displays various screens in the display area based on the control of the control unit 51. The imaging unit 56 and the display unit 57 are located on opposite sides of each other via the housing of the communication terminal 50. In other words, the positional relationship between the imaging range of the imaging unit 56 and the display area of ​​the display unit 57 is defined such that when a user views the display area of ​​the display unit 57, the imaging range of the imaging unit 56 includes the area beyond the display area. In this example, as will be described later, the display unit 57 can perform a display using AR technology (hereinafter sometimes referred to as AR display) under the control of the control unit 51. More specifically, the display unit 57 can superimpose and display character objects in a virtual space onto the image corresponding to the imaging range of the imaging unit 56.

[0028] In this example, the operation unit 58 includes a touch sensor and outputs information corresponding to the position operated by the user to the control unit 51. This touch sensor is located on the display area of ​​the display unit 57. In other words, the display unit 57 and the operation unit 58 constitute a touch panel.

[0029] [Hardware configuration of Server 10] Server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is an example of a computer that includes an arithmetic processing circuit such as a CPU. The control unit 11 executes control programs stored in the storage unit 12 using the CPU to realize various functions in Server 10. Through the functions realized in the communication terminal 50 and the functions realized in Server 10, Server 10 provides services to the user via the communication terminal 50.

[0030] The storage unit 12 includes a memory device and stores the control program. The storage unit 12 also stores various data used when the control program is executed. This program may be provided recorded on a computer-readable recording medium such as a magnetic recording medium, optical recording medium, magneto-optical recording medium, or semiconductor memory. In this case, the server 10 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 12 included in the server 10, or it may be the medium used in the storage unit 12.

[0031] The communication unit 13 includes a communication module and, under the control of the control unit 11, connects to the network NW to send and receive information with other devices connected to the network NW. In this example, the communication unit 13 also connects to the database 80 to send and receive information. The database 80 contains data related to services provided by the server 10. The communication unit 13 may also connect to the database 80 via the network NW. Furthermore, the data registered in the database 80 may be stored in the storage unit 12. In this case, the database 80 does not need to exist.

[0032] [Data registered in database 80] The data registered in database 80, that is, the data related to the services provided by server 10, includes animation data, motion data, static data, set-related data, and user management data. These data will be described next.

[0033] Figure 2 is a diagram illustrating the animation data in the first embodiment. The animation data is the data that constitutes the animation work. The animation data includes data that defines the content of the video, and includes motion data MdCh1, MdCh2, MdCh3 corresponding to three types of character objects ch1, ch2, and ch3, and background data BGd corresponding to the background BG. The horizontal axis shows the elapsed time T from the start of the video. The background data BGd is data that shows the video in the virtual space other than the character objects, and the content of the video is defined in accordance with the elapsed time T. The character objects are defined by various data. In this example, in order to define the movement of the character objects, data is used for multiple parts of the character (for example, parts corresponding to joints such as shoulders, elbows, and wrists), and data related to the skeleton that connects the multiple parts. The pose of the character object is defined by this data, and the movement form of the character object is defined by changing the position of each part over time. Data that defines the appearance of the character object (body surface, hair, clothing, etc.) is also used. The motion data that defines the movement form of the character object will be described in more detail below.

[0034] The behavior data MdCh1, MdCh2, and MdCh3, corresponding to character objects, define the behavior of each character object in relation to the period in which it appears. For example, character object ch1 appears in periods Sc1a and Sc1b, and the behavior data MdCh1 is defined for that period. Similarly, character object ch2 appears in periods Sc2a and Sc2b, and the behavior data MdCh2 is defined for that period. Also, character object ch3 appears in period Sc3a, and the behavior data MdCh3 is defined for that period.

[0035] Figure 3 illustrates the operation data in the first embodiment. Here, operation data MdCh1 corresponding to the character object ch1 is shown as an example. In operation data MdCh1, the coordinates in virtual space corresponding to multiple parts of the character object (Pa, Pb, ...) are associated with the elapsed time T. At an elapsed time T of "001:21:35", the X, Y, and Z coordinates of part Pa are Xa2, Ya2, and Za2, respectively. The operation mode of the character object in virtual space is defined by the time change of the coordinates of each part. In this way, the operation of the character object is represented by the time change of the coordinates corresponding to each part of the character object. That is, the content of this operation corresponds to the operation mode.

[0036] Let's return to Figure 2 and continue the explanation. For each character object, a specific timing at the elapsed time T is defined. The coordinates of each part at that timing are registered as static data in the database 80. In this example, timing t1 is defined for character object ch1. Static data Sd1, which defines the coordinates of each part of character object ch1 at timing t1, is registered in the database 80. Similarly, timings t4 and t7 are defined for character objects ch2 and ch3, respectively, and static data Sd2 and Sd3 are registered in the database 80.

[0037] Figure 4 illustrates static data in the first embodiment. Here, static data Sd1 corresponding to the character object ch1 is shown as an example. Static data Sd1 associates the coordinates in virtual space with each part Pa, Pb, ... This corresponds to the coordinates of each part when the elapsed time T is t1, i.e., "001:21:35", in the motion data MdCh1 shown in Figure 3.

[0038] Let's return to Figure 2 and continue the explanation. In at least a portion of the overlapping appearance periods of multiple character objects, a specific period at elapsed time T is defined. The operation data of the multiple character objects during that period is registered in database 80 as an operation dataset. In this example, for the combination of character objects ch1 and ch2, a specific period from timing t2 to t3 is defined. The data extracted from the operation data MdCh1 of character object ch1 for that specific period, and the data extracted from the operation data MdCh2 of character object ch2 for that specific period are set together and registered in database 80 as an operation dataset Md12. Note that this specific period may be the entirety, not just a portion, of the overlapping appearance period of character objects ch1 and ch2.

[0039] Similarly, for the combinations of character objects ch1 and ch3, and the combinations of character objects ch2 and ch3, specific periods from t5 to t6 and from t8 to t9 are defined, respectively, and action datasets Md13 and Md23, which contain action data corresponding to these specific periods, are registered in database 80. Furthermore, for the three combinations, namely the combinations of character objects ch1, ch2, and ch3, a specific period from t10 to t11 is defined, and an action dataset Md123 is registered in database 80.

[0040] Figure 5 is a diagram illustrating the set-corresponding data in the first embodiment. The set-corresponding data defines the correspondence between the action dataset and the character object combination. The correspondence is as described above. For example, the combination of character objects ch1 and ch3 corresponds to the action dataset Md13. As described above, the action dataset Md13 is data extracted from the action data MdCh1 and MdCh3 of character objects ch1 and ch3 for a specific period from t5 to t6.

[0041] Next, we will explain user management data. A user can display authorized character objects from among character objects ch1, ch2, and ch3 on the display unit 57 of the communication terminal 50 they operate, using AR technology. User management data is data that defines the character objects permitted for each user. A user and a character object are associated, for example, by user identification information (user ID, etc.) and object identification information (object ID, etc.). In other words, user management data is data that defines the character objects that can be displayed on the user's communication terminal 50. Furthermore, user management data can also be said to be data that defines the character objects owned by a user through the association between user identification information and object identification information.

[0042] Figure 6 illustrates an example of user management data in the first embodiment. User management data is data that defines the character objects that each user U1, U2, ... is permitted to display. For example, user U1 is permitted to display character object ch1. In some cases, such as user U4, multiple character objects ch1, ch2, etc., may be permitted to display. This concludes the explanation of the data registered in database 80.

[0043] [Service provision processing] Next, we will describe the process by which server 10 provides services to the user. As mentioned above, this service is the service that operates character objects. First, we will describe the services provided in relation to this, and then we will describe the character object operation service in detail.

[0044] A viewing service is provided to the user that allows them to watch animated works by executing a predetermined application program on the communication terminal 50. In providing the viewing service, when the communication terminal 50 requests the server 10 to watch an animated work, predetermined data is sent from the server 10 to the communication terminal 50, and based on that data, the communication terminal 50 becomes able to watch the animated work. In this example, the server 10 searches the database 80 for animation data corresponding to the animated work requested by the communication terminal 50 and sends it to the communication terminal 50. On the communication terminal 50, the control unit 51 generates video data based on the animation data and displays an image corresponding to the video data on the display unit 57. In the case of the animation data shown in Figure 2, video data corresponding to the passage of time is generated by superimposing images of character objects ch1, ch2, and ch3 that operate according to the motion data onto an image generated based on the background data BGd.

[0045] When viewing such an animated work, for example, one character object randomly selected from multiple character objects is permitted to be displayed on the user's communication terminal 50. The selected character object is associated with the user as a character object permitted to be displayed and registered in the user management data by the server 10. If the user wants to display multiple character objects, as shown in Figure 6 for user U4, this can be achieved, for example, through billing. In this way, the character objects permitted to be displayed can be displayed on the communication terminal 50 using AR technology. An example of character object display will be briefly explained.

[0046] Figure 7 is a diagram illustrating the relationship between the real space and the imaging direction in the first embodiment. In the following description, the real space RS to be imaged is assumed to be the space shown in Figure 7. The real space RS includes the floor surface F and walls W1 and W2. A marker MR is placed on the floor surface F. This marker MR contains orientation information MA. As imaging directions, a first direction AR1 is defined, which is from the left side of Figure 7 toward the marker MR, and a second direction AR2 is defined, which is from the right side of Figure 7 toward the marker MR.

[0047] Here, it is assumed that a character object is displayed on a communication terminal 50 (hereinafter sometimes referred to as communication terminal 50-1) operated by user U1 while capturing images of the real space RS along the first direction AR1. It is also assumed that user U2 is near user U1. The communication terminal 50 operated by user U2 may hereafter be referred to as communication terminal 50-2. The relationship between communication terminal 50 and the user, that is, the user operating communication terminal 50, is identified by identification information such as the user ID logged in while the application program is running.

[0048] Figure 8 illustrates an example of display in the singular mode (solo mode) in the first embodiment. An application program is executed on the communication terminal 50-1. As a result, a character object is superimposed on the image of the real space RS included in the imaging range of the imaging unit 56 and displayed in the display area DA of the display unit 57. This character object is determined by referring to user management data (Figure 6), and in this case, it is the character object ch1 that user U1 is permitted to display. The form of the character object ch1 is specified according to static data Sd1 (Figure 4). The coordinate system in the virtual space to which static data Sd1 is applied is defined by the marker MR. Specifically, the reference position of the coordinates (origin of the virtual space) is defined by the marker MR, and the X-axis, Y-axis, and Z-axis directions are specified by information MA. At this time, the scale of each axis direction may be defined according to the size of the marker MR. Note that the example shown in Figure 8 shows a display mode in which a character object permitted to be displayed by the user is available for display. This display mode is called "singular mode" in this example.

[0049] The display area DA shown in Figure 8 also displays a toggle button SB1 for switching from single mode (solo mode) to multiple mode (multi-mode). When the user operates the toggle button SB1, a process is executed to switch the display mode to multiple mode. In this example, multiple mode is a display mode in which character objects that are permitted to be displayed by a user operating communication terminal 50 that is near communication terminal 50-1 can be displayed together with the character object ch1 on communication terminal 50-1. Next, the process that provides the service for realizing display in multiple mode will be described.

[0050] Figure 9 is a diagram illustrating the communication flow of the service provision process (pre-configuration) in the first embodiment. First, when user U1's communication terminal 50-1 and user U2's communication terminal 50-2 execute the application program, they periodically send location information to the server 10. (Step S100). Here, it is assumed that the communication terminal 50-1 is displaying in singular mode as described above. In the following description of the communication flow, the processing of the server 10 is performed under the control of the control unit 11, and the processing of the communication terminal 50 is performed under the control of the control unit 51.

[0051] When user U1 operates the switch button SB1 on communication terminal 50-1, communication terminal 50-1 receives an instruction to switch to multiple modes (step S110) and notifies server 10 of the mode switch (step S120). Upon receiving this notification, server 10 searches for users operating communication terminals located within a distance shorter than a predetermined distance from communication terminal 50-1 (step S130). In this example, communication terminal 50-2 is identified as a communication terminal close to communication terminal 50-1, and user U2 of this communication terminal 50-2 is identified. Although not shown in Figure 9, in this example, users U3 and U5 are also identified. Server 10 sends data to communication terminal 50-1 for displaying the user selection screen (step S140). Upon receiving this data, communication terminal 50-1 displays the user selection screen on the display unit 57 (step S150). The user selection screen presents the users identified by the search and allows the user (user U1) to select a user to combine with themselves. This user selection determines which character object is displayed alongside character object ch1 in multiple modes.

[0052] Figure 10 illustrates the user selection screen in the first embodiment. The user selection screen shown in Figure 10 is an example displayed in the display area DA of the display unit 57 of the communication terminal 50-1. The user selection screen displays selection buttons U2B, U3B, and U5B for selecting identified users U2, U3, and U5, and a confirmation button USB for confirming the selected user. When the user operates the selection button U2B, the display changes to one different from the unselected user, as shown in Figure 10. Then, when the user operates the confirmation button USB, the selected user is confirmed.

[0053] Returning to Figure 9, let's continue the explanation. Communication terminal 50-1 sends data indicating the selected user to server 10 (step S160). Here, we assume that user U2 has been determined on the user selection screen. Server 10 sends data to communication terminal 50-2, which corresponds to user U2 determined by communication terminal 50-1, to display an acceptance request screen (step S170). When communication terminal 50-2 receives this data, it displays the acceptance request screen on the display unit 57 (step S180). The acceptance request screen is a screen for responding whether or not to accept being designated as a target for a combination by another user.

[0054] Figure 11 is a diagram illustrating the selection / acceptance screen in the first embodiment. The selection / acceptance screen shown in Figure 11 is an example displayed in the display area DA of the display unit 57 of the communication terminal 50-2. The selection / acceptance screen displays that the user has been selected as a target for combination by another user (in this case, user U1), and buttons for indicating whether to accept or reject it, namely a Yes button YB1 for acceptance and a No button NB1 for rejection.

[0055] When user U2 operates the No button NB1 on the selection acceptance screen, communication terminal 50-2 sends a rejection response to server 10. Server 10 notifies communication terminal 50-1 that user U2 has rejected the combination. Upon receiving this notification, communication terminal 50-1 displays that user U2 has rejected the combination on display unit 57 and returns the display mode to single mode.

[0056] On the other hand, when user U2 operates the Yes button YB1 on the selection acceptance screen, communication terminal 50-2 sends an acceptance response to server 10. As a result, server 10 starts control on communication terminal 50-1 to enable display in multiple modes.

[0057] Figure 12 is a diagram illustrating the communication flow of the service provision process (multiple mode display control) in the first embodiment. Server 10 searches for an action dataset corresponding to a combination of users (a combination of user U1 and user U2) (step S210). Specifically, the process is executed as follows. First, Server 10 refers to the user management data to identify the character object corresponding to the combined users. In this example, when user U1 and user U2 are combined, character objects ch1 and ch2 are identified. Next, Server 10 refers to the set correspondence data to search for an action dataset corresponding to the combination of character objects. In this example, when character objects ch1 and ch2 are combined, action dataset Md12 is identified.

[0058] Server 10 transmits the identified action dataset Md12 and data related to character objects ch1 and ch2 to communication terminals 50-1 and 50-2 (step S220). If communication terminal 50-1 already has data related to character object ch1, then it is sufficient to transmit only the action dataset Md12 and data related to character object ch2. Similarly, if communication terminal 50-2 already has data related to character object ch2, then it is sufficient to transmit only the action dataset Md12 and data related to character object ch1.

[0059] When communication terminal 50-1 has finished receiving the operation dataset Md12, etc., from server 10, it displays a playback start screen (step S230). The playback start screen is a screen for starting playback in multiple modes on communication terminal 50, in this example, communication terminal 50-1, which has been switched to multiple modes.

[0060] Figure 13 is a diagram illustrating the playback instruction screen in the first embodiment. The playback start screen shown in Figure 13 is an example of what is displayed in the display area DA of the display unit 57 of the communication terminal 50-1. The playback start screen displays a message prompting playback in multiple modes, and buttons for indicating whether to start or postpone playback, namely a Yes button YB2 to instruct playback to start and a Later button NB2 to postpone playback.

[0061] When user U1 operates the Later button NB2 on the playback start screen, the playback start screen is displayed again after a predetermined time, and during that time, the communication terminal 50-1 can perform other processing.

[0062] On the other hand, when user U1 operates the Yes button YB2 on the playback start screen, communication terminal 50-1 sends a playback start instruction to server 10 (step S240). Upon receiving this instruction, server 10 sends a synchronization signal to communication terminals 50-1, 50-1 (step S250). Upon receiving the synchronization signal, communication terminal 50-1 displays the character objects ch1 and ch2 in the display area DA of the display unit 57 in accordance with the operation dataset Md12. The time progression of the operation of character objects ch1 and ch2 is controlled by the synchronization signal.

[0063] Figure 14 is a diagram illustrating an example of display in multiple modes in the first embodiment (when imaging is performed along the first direction AR1). Character objects ch1 and ch2 are displayed in the display area DA of the display unit 57 of the communication terminal 50-1. In the example shown in Figure 14, a predetermined time has elapsed since the playback of the operation of character objects ch1 and ch2 started in multiple mode display. The elapsed time is indicated by the seek bar SK. Furthermore, a switch button SB2 for switching from multiple mode to single mode is displayed in the display area DA. The user can change the playback position (the playback portion in the operation data) by operating the seek bar SK, or end the multiple mode by operating the switch button SB2.

[0064] Since character objects ch1 and ch2 operate according to the motion dataset Md12, the operation during the period from t2 to t3 in the animation data shown in Figure 2 is reproduced. As a result, a portion of the animation is reproduced by AR display. Here, in the display area DA, character objects ch1 and ch2 are superimposed on the image of the real space RS which is included in the imaging range of the imaging unit 56. The coordinate system in the virtual space to which the motion dataset Md12 is applied is the same as in the singular mode. That is, the reference position of the coordinates (the origin of the virtual space) is defined by the marker MR, and the X-axis, Y-axis, and Z-axis directions are specified by the information MA.

[0065] In this example, the background is an image of the real-world RS, and the background BG in the animation data is not used. Alternatively, a background BG processed as a semi-transparent image may be superimposed on the image of the real-world RS.

[0066] Here, the display in multiple modes may be implemented only on the communication terminal 50-1, but the same display as on communication terminal 50-1 may also be implemented on the communication terminal 50-2 of the user U2 that is the target of the combination. In this case, the operation of character objects ch1 and ch2 may be synchronized by a synchronization signal. By using a synchronization signal, the operation of character objects ch1 and ch2 can be displayed at the same elapsed time on communication terminal 50-1 and communication terminal 50-2.

[0067] Figure 15 illustrates an example of display in multiple modes in the first embodiment (when imaging is performed along the second direction AR2). Figure 14 shows the display content when communication terminal 50-1 images the real space RS along the first direction AR1. On the other hand, Figure 15 shows the display content when communication terminal 50-2 images the real space RS along the second direction AR2. When viewed from the second direction AR2, the orientation of the marker MR is different. Since both displays using a common coordinate system, the X-axis, Y-axis, and Z-axis directions in the virtual space relative to the screen in the display example shown in Figure 15 are different from the display example shown in Figure 14. As a result, the appearance of character objects ch1 and ch2 differs between communication terminal 50-1 and communication terminal 50-2. However, in this example, the passage of time is synchronized between communication terminal 50-1 and communication terminal 50-2 by a synchronization signal, so the operation of character objects ch1 and ch2 is the same.

[0068] In the example described above, the combination of users U1 and U2 allows character objects ch1 and ch2 to be operated together and displayed on the display unit 57. This combination is not limited to two, but may be three or more. In the example of user management data shown in Figure 6, the combination of users U1, U2, and U3 allows character objects ch1, ch2, and ch3 to be operated together and displayed. Similarly, the combination of users U3 and U4 also allows character objects ch1, ch2, and ch3 to be operated together and displayed. Since user U4 is permitted to display both character objects ch1 and ch2, user U4's communication terminal 50 may display one of them in single mode, or both in multiple mode.

[0069] Thus, even if each user is permitted to display only a few character objects, they can display multiple character objects together by cooperating with other users. In this case, by combining multiple character objects, it is possible not only to display them, but also to display them on the display unit 57 with actions that could not be achieved with a single character object. Therefore, by combining character objects with other character objects, a wide variety of expressions can be provided to the user, thus preventing the user from losing interest in character objects.

[0070] [Configuration of the display control unit used in the service provision process] Next, the configuration of the function (display control unit) for realizing the process shown in Figure 12 will be explained using Figure 16.

[0071] Figure 16 is a block diagram showing the configuration of the display control unit in the first embodiment. Figure 16 shows an example in which the server 10 and the communication terminal 50 cooperate to realize the display control unit 700. The operation determination unit 600 is realized in the server 10. The operation determination unit 600 has the function of performing the processing of step S210 shown in Figure 12. That is, the operation determination unit 600 refers to user management data to identify character objects that are permitted to be displayed for multiple users, and refers to set correspondence data to identify an operation set corresponding to a combination of character objects. In this way, the operation determination unit 600 determines the operation of multiple character objects.

[0072] The display control unit 700 includes a data supply unit 710, an operation control unit 730, and a display overlay unit 750. The data supply unit 710 reads operation data included in the operation dataset and data related to multiple character objects to which the operation data is applied from the database 80 and provides them to the operation control unit 730. The operation control unit 730 applies the operation data to each character object and generates an image of the character object in motion.

[0073] The display superimposition unit 750 superimposes the image generated by the motion control unit 730 onto the image within the imaging range of the imaging unit 56 and displays it on the display unit 57. At this time, the X-axis, Y-axis, and Z-axis directions in the virtual space may be corrected based on the attitude of the communication terminal 50, etc., using the behavior information from the behavior measurement unit 55.

[0074] In the first embodiment, the functions of the data provision unit 710 are implemented in the server 10, and the functions of the operation control unit 730 and the display superimposition unit 750 are implemented in the communication terminal 50. The server 10 controls the operation of character objects displayed on the communication terminal 50 by providing the communication terminal 50 with the functions of the data provision unit 710, i.e., operation datasets, etc.

[0075] <Second Embodiment> In the display control unit 700, the functions of the data provision unit 710 and the operation control unit 730 may be implemented in the server 10. In this case, the server 10 will not send an operation dataset or the like to the communication terminal 50, but will send image data of the character object in motion to the communication terminal 50.

[0076] <Third Embodiment> In the display control unit 700, all functions, namely the functions of the data provision unit 710, the operation control unit 730, and the display overlay unit 750, may be implemented in the server 10. In this case, the server 10 acquires the image captured by the communication terminal 50 and uses it in the display overlay unit 750.

[0077] <Fourth Embodiment> In the display control unit 700, all functions, namely the functions of the data provision unit 710, the operation control unit 730, and the display superimposition unit 750, may be implemented in the communication terminal 50. In this case, the communication terminal 50 connects to the database 80 via the network NW in order to implement the functions of the data provision unit 710.

[0078] <Variation> Although one embodiment of the present invention has been described above, each of the embodiments described above can be applied by combining or substituting them with one another. Furthermore, each of the embodiments described above can also be implemented with the following modifications.

[0079] (1) In the embodiments described above, the character object, static data, and motion data were data extracted and generated from animation data, but the original animation data is not required. In other words, each of them may be data generated separately for the purpose of realizing the data provision service.

[0080] (2) The process for granting permission to display a character object is not limited to the method of the above embodiment. The display may be granted through a lottery system known as gacha, or through a code distributed at an event, etc.

[0081] (3) In the embodiments described above, the users to be combined were selected based on the distance between them, but they may also be combined by another method if they satisfy predetermined related conditions. Users who are close together may be identified, for example, as users of a communication terminal 50 that is photographing an object determined to be the same by image recognition, or as users of a communication terminal 50 that can communicate by short-range wireless communication. The above-mentioned object may be, for example, the marker MR shown in the embodiments described above, or any other object that is included in the imaging range and can be extracted by image analysis. That is, the predetermined related conditions may be having specific common information obtained by image analysis of the imaging range. Note that the users to be combined may not necessarily be close together, and may be users who are linked as friends on an SNS, for example.

[0082] (4) In the embodiments described above, marker MR was used to identify the coordinate system in the virtual space in order to display character objects, but the method is not limited to using marker MR. For example, 3D information of the real space RS may be obtained by using V-SLAM (Visual Simultaneous Localization and Mapping) technology and associated with the coordinate system in the virtual space. A part of the analyzed 3D information may be recognized as a feature part, and the feature part may be replaced by marker MR.

[0083] (5) In the embodiments described above, the action dataset was determined by the combination of character objects, but the action dataset may be determined using another element. This may be used to specify that the action dataset will differ even for the same combination of character objects. For example, the item objects used by a particular character object may be specified. This may be used to specify set-associated data such that the action dataset identified differs depending on whether or not an item object exists as part of the combination. Specifically, the action dataset applied in multiple modes will differ depending on whether or not a user who is permitted to display character object ch1 is also permitted to display the item objects used by character object ch1 (item objects associated with character object ch1).

[0084] (6) In the embodiments described above, the timing of the static data corresponding to each character object is defined as a timing that does not overlap with the appearance period of other character objects, but this is not limited to this. That is, the timing of the static data corresponding to a character object may be defined as a timing that overlaps with the appearance period of other character objects, or it may be defined as a timing that overlaps with the period used for the motion data set. The static data Sd1 may be defined, for example, as the coordinates of each part at any timing from t2 to t3 in the period that overlaps with the appearance period of character object ch2.

[0085] As a result, the static data Sd1 is an extraction of a portion of the motion pattern of the character object ch1 defined in the motion dataset Md12. Therefore, static data can be efficiently generated from motion data even without overall data such as animation data, and it can also motivate the user to want to see the motion before and after the character object ch1 is displayed in singular mode. On the other hand, as in the embodiment described above, the static data Sd1 can also be in a form that is not included in the motion pattern of the character object ch1 defined in the motion dataset Md12. In this case, the static data Sd1 needs to be generated as data of a different motion pattern than that of the motion dataset Md12, but the user can enjoy a completely different display of the character object ch1 in multiple modes compared to the singular mode.

[0086] (7) In the embodiments described above, character objects displayed in singular mode were given still data which resulted in still images, but motion data which results in moving images may also be given. Even in this case, different motion data should be applied to the character objects in singular mode and in multiple mode.

[0087] (8) Although the embodiments described above illustrate an example of applying the present invention to the control of the movement of a character object, the present invention is applicable not only to character objects but also to the control of the movement of objects having a three-dimensional shape. [Explanation of Symbols]

[0088] 10…Server, 11…Control Unit, 12…Storage Unit, 13…Communication Unit, 50, 50-1, 50-2…Communication Terminal, 51…Control Unit, 52…Storage Unit, 53…Communication Unit, 54…Position Measurement Unit, 55…Behavior Measurement Unit, 56…Imaging Unit, 57…Display Unit, 58…Operation Unit, 80…Database, 600…Operation Determination Unit, 700…Display Control Unit, 710…Data Provision Unit, 730…Operation Control Unit, 750…Display Overlay Unit, 1000…System

Claims

1. The processor comprises, From animation data that defines the movement patterns of multiple character objects, motion data for a specific period is extracted, and a motion dataset corresponding to the combination of character objects is generated. The system stores set correspondence data that defines the correspondence between the aforementioned action dataset and the combination of the aforementioned character object. In response to a request from the user terminal, the system refers to the set-corresponding data and provides the corresponding operation dataset to the user terminal. Information processing device.

2. The processor extracts motion data for a specific period from animation data that defines the motion patterns of multiple character objects, and generates a motion dataset corresponding to the combination of character objects. The processor stores set correspondence data that defines the correspondence between the operation dataset and the character object combination. The processor, in response to a request from the user terminal, refers to the set-corresponding data and provides the corresponding operation dataset to the user terminal. Information processing methods.

3. The processor extracts motion data for a specific period from animation data that defines the motion patterns of multiple character objects, and generates motion datasets corresponding to combinations of character objects. The processor is instructed to store set correspondence data that defines the correspondence between the operation dataset and the character object combination. The processor, in response to a request from the user terminal, refers to the set-corresponding data and provides the corresponding operation dataset to the user terminal. program.

4. Equipped with a server and terminal, The aforementioned server, From animation data that defines the movement patterns of multiple character objects, motion data for a specific period is extracted, and a motion dataset corresponding to the combination of character objects is generated. The system stores set correspondence data that defines the correspondence between the aforementioned action dataset and the combination of the aforementioned character object. In response to a request from the user terminal, the system refers to the set-corresponding data and provides the corresponding operation dataset to the terminal. system.

Images