Information processing apparatus and program

Abstract

To let a plurality of character objects operate in a mode corresponding to a combination of a character object which can be displayed on one terminal and a character object which can be displayed on another terminal.SOLUTION: An information processing apparatus as one embodiment of the present invention includes: a determination section which determines, based upon a first character object which can be displayed on a first terminal and a second character object which can be displayed on a second terminal, operation modes of the first character object and the second character object; and a display control section which controls the first terminal so that the first character object and the second character object are displayed over an image captured by the first terminal according to the operation modes.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Augmented reality (AR) technology (hereinafter referred to as AR technology) has been used in mobile devices such as smartphones and game consoles. AR technology is a technology that displays on a display an image of a virtual space superimposed on an object placed in the virtual space and an image obtained by capturing an image of the real space with a camera, and this technology makes it possible to make an object in the virtual space appear to exist in the real space. AR technology is disclosed, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-41126 Summary of the Invention [Problem to be solved by the invention]

[0004] As an example of the application of AR technology, a 3D virtual character object owned by a user is displayed on a mobile device as if it exists in real space. In addition to using a character object owned by a user alone, it is also desirable for a user to be able to use the character object in collaboration with other users.

[0005] One object of the present invention is to provide an information processing device and program that can operate multiple character objects in a manner that corresponds to a combination of character objects that can be displayed on one terminal and character objects that can be displayed on another terminal. [Means for solving the problem]

[0006] According to one embodiment of the present invention, there is provided an information processing device including: a determination unit that determines the motion forms of a first character object and a second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; and a display control unit that controls the first terminal to display the first character object and the second character object according to the motion forms, superimposed on an image captured on the first terminal.

[0007] The first terminal and the second terminal may satisfy a predetermined association condition.

[0008] The association condition may include that the first terminal and the second terminal are closer than a predetermined distance.

[0009] The association condition may include that the same object is included in the image captured by the first terminal and the image captured by the second terminal.

[0010] When the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal may be an image based on a part of the movement form, and the second character object that can be displayed on the second terminal may be an image based on a part of the movement form.

[0011] When the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal may be an image based on a form that is not included in the action form, and the second character object that can be displayed on the second terminal may be an image based on a form that is not included in the action form.

[0012] The method may further include controlling the second terminal to display the first character object and the second character object according to the action form, superimposed on an image captured by the second terminal.

[0013] The display control unit may control the first terminal to display the first character object and the second character object using a coordinate system set for an image captured on the first terminal, and may control the second terminal to superimpose the first character object and the second character object on an image captured on the second terminal based on the coordinate system and display them in synchronization with the display on the first terminal.

[0014] When there is an item object that can be displayed together with the first character object on the first terminal, the determination unit may further determine the action form based on the item object.

[0015] The movement form may correspond to a movement form defined in data defining the movement forms of a plurality of character objects including the first character object and the second character object, during at least a portion of the period during which the first character object and the second character object are used.

[0016] Furthermore, according to one embodiment of the present invention, a program is provided for causing a computer to control a first terminal to determine the motion forms of a first character object and a second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal, and to display the first character object and the second character object in accordance with the motion forms by superimposing them on an image captured on the first terminal. [Effects of the Invention]

[0017] According to one embodiment of the present invention, an information processing device and program can be provided that can operate multiple character objects in a form that corresponds to a combination of character objects that can be displayed on one terminal and character objects that can be displayed on another terminal. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a configuration of a system in a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating animation data in the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating operation data in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating still data in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating set correspondence data in the first embodiment. [Figure 6] FIG. 3 is a diagram illustrating user management data in the first embodiment. [Figure 7] 3A and 3B are diagrams illustrating the relationship between a real space and an imaging direction in the first embodiment. [Figure 8] FIG. 4 is a diagram illustrating a display example in a single mode in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating a communication flow of a service provision process (presetting) in the first embodiment. [Figure 10] FIG. 2 is a diagram illustrating a user selection screen in the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating a selection acceptance screen in the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating a communication flow of the service providing process (multiple mode display control) in the first embodiment. [Figure 13] FIG. 4 is a diagram illustrating a playback instruction screen in the first embodiment. [Figure 14]10A and 10B are diagrams illustrating a display example in a multiple mode in the first embodiment (when images are taken along a first direction AR1). [Figure 15] 10 is a diagram illustrating a display example in a multiple mode in the first embodiment (when images are taken along the second direction AR2). FIG. [Figure 16] FIG. 2 is a block diagram showing the configuration of a display control unit in the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] A system according to an embodiment of the present invention will be described in detail below with reference to the drawings. The embodiment described below is an example of an embodiment of the present invention, and the present invention is not limited to these embodiments.

[0020] First Embodiment A system according to a first embodiment of the present invention will be described in detail with reference to the drawings.

[0021] [overview] 1 is a block diagram showing the configuration of a system in the first embodiment. The system 1000 includes a server 10 (information processing device) and a communication terminal 50. The server 10 and the communication terminal 50 are connected via a network NW such as the Internet. In this example, the communication terminal 50 is a smartphone. Note that the communication terminal 50 may also be a portable device such as a tablet terminal or a portable game console.

[0022] Although the server 10 is shown as one device in FIG. 1, it may be composed of multiple devices. The server 10 provides application programs to the communication terminals 50 and various services to the communication terminals 50 that execute the application programs. The application programs are recorded on a recording medium (storage device) to which the server 10 can be connected. The various services include a character object operation service, which will be described below. The various services may also include, for example, an SNS (social networking service).

[0023] An application program for receiving services from the server 10 is installed in the communication terminal 50. This program allows character objects to be displayed on the communication terminal 50 using AR technology. In this example, the program further allows other character objects to be displayed on the communication terminal 50 when a predetermined condition is met, and allows the displayed character objects to perform special actions. The configuration of the communication terminal 50 will be described below.

[0024] [Hardware configuration of communication terminal 50] The communication terminal 50 includes a control unit 51, a storage unit 52, a communication unit 53, a position measurement unit 54, a behavior measurement unit 55, an imaging unit 56, a display unit 57, and an operation unit 58. The control unit 51 is an example of a computer including an arithmetic processing circuit such as a CPU. The control unit 51 executes application programs stored in the storage unit 52 using the CPU, thereby realizing various functions in the communication terminal 50.

[0025] The storage unit 52 includes a storage device and stores an application program downloaded from the server 10 via the network NW and installed. The storage unit 52 also stores various data used when the application program is executed. The program may be provided in a state recorded on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the communication terminal 50 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 52 included in the communication terminal 50, or may be a medium used for the storage unit 52.

[0026] The communication unit 53 includes a communication module, connects to the network NW under the control of the control unit 51, and transmits and receives information to and from other devices connected to the network NW. The position measurement unit 54 measures the position of the communication terminal 50 using technology such as GPS, and outputs information indicating the position (hereinafter, sometimes referred to as position information) to the control unit 51. The behavior measurement unit 55 includes an acceleration sensor, a gyro sensor, etc., and measures the behavior of the communication terminal 50, and outputs information indicating the behavior (hereinafter, sometimes referred to as behavior information) to the control unit 51. The behavior information includes, for example, information for identifying the amount of movement and rotation of the communication terminal 50. Note that, by using a geomagnetic sensor or the like in combination, information for identifying the orientation of the communication terminal 50 may also be generated in the communication terminal 50.

[0027] The imaging unit 56 is a camera including a CCD sensor or a CMOS sensor, and outputs information indicating an image in the imaging range to the control unit 51. The information indicating the image is either a still image or a video. The display unit 57 is a display device such as a liquid crystal display or an organic EL display, and displays various screens in the display area under the control of the control unit 51. The imaging unit 56 and the display unit 57 are disposed on opposite sides of each other across the housing of the communication terminal 50. In other words, the positional relationship between the imaging range of the imaging unit 56 and the display area of ​​the display unit 57 is defined so that when a user views the display area of ​​the display unit 57, the imaging range of the imaging unit 56 includes an area beyond the display area. In this example, as will be described later, under the control of the control unit 51, the display unit 57 can display using AR technology (hereinafter, sometimes referred to as AR display). More specifically, the display unit 57 can display a character object in a virtual space superimposed on an image corresponding to the imaging range of the imaging unit 56.

[0028] In this example, operation unit 58 includes a touch sensor, and outputs information corresponding to the position operated by the user to control unit 51. This touch sensor is provided on the display area of ​​display unit 57. In other words, display unit 57 and operation unit 58 form a touch panel.

[0029] [Server 10 hardware configuration] The server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is an example of a computer including an arithmetic processing circuit such as a CPU. The control unit 11 executes a control program stored in the storage unit 12 using the CPU to realize various functions in the server 10. In this way, the server 10 provides services to the user via the communication terminal 50 using the functions realized in the communication terminal 50 and the functions realized in the server 10.

[0030] The storage unit 12 includes a storage device and stores a control program. The storage unit 12 also stores various data used when the control program is executed. The program may be provided in a state recorded on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the server 10 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 12 included in the server 10, or may be a medium used in the storage unit 12.

[0031] The communication unit 13 includes a communication module, and connects to the network NW under the control of the control unit 11 to transmit and receive information to and from other devices connected to the network NW. In this example, the communication unit 13 also connects to a database 80 to transmit and receive information. Data related to services provided by the server 10 is registered in the database 80. The communication unit 13 may be connected to the database 80 via the network NW. The data registered in the database 80 may also be stored in the storage unit 12. In this case, the database 80 may not exist.

[0032] [Data registered in Database 80] The data registered in the database 80, that is, the data related to the services provided by the server 10, includes animation data, motion data, still data, set correspondence data, and user management data. Next, these data will be described.

[0033] FIG. 2 is a diagram illustrating animation data in the first embodiment. Animation data is data that constitutes an animation work. The animation data includes data that defines the content of the video, including movement data MdCh1, MdCh2, and MdCh3 corresponding to three types of character objects ch1, ch2, and ch3, and background data BGd corresponding to the background BG. The horizontal axis represents the elapsed time T from the start of the video. The background data BGd is data representing video in a virtual space other than the character objects, and defines the content of the video in accordance with the elapsed time T. Character objects are defined by various data. In this example, to define the movement of a character object, data related to multiple body parts of the character (e.g., body parts corresponding to joints such as shoulders, elbows, and wrists) and the skeleton connecting the multiple body parts are used. These data define the pose of the character object, and the movement form of the character object is defined by changing the position of each body part over time. Data defining the appearance of the character object (body surface, hair, clothing, etc.) is also used. The movement data that defines the movement form of a character object will be described in more detail below.

[0034] The motion data MdCh1, MdCh2, and MdCh3 corresponding to the character objects define the motions of the character objects corresponding to the periods in which the respective character objects appear. For example, character object ch1 appears in periods Sc1a and Sc1b, and motion data MdCh1 is defined for those periods. Similarly, character object ch2 appears in periods Sc2a and Sc2b, and motion data MdCh2 is defined for those periods. Furthermore, character object ch3 appears in period Sc3a, and motion data MdCh3 is defined for that period.

[0035] FIG. 3 is a diagram illustrating movement data in the first embodiment. Here, movement data MdCh1 corresponding to character object ch1 is illustrated. In movement data MdCh1, coordinates in virtual space corresponding to multiple body parts (Pa, Pb, ...) of the character object are associated with elapsed time T. When elapsed time T is "001:21:35", the X, Y, and Z coordinates of body part Pa are Xa2, Ya2, and Za2, respectively. The movement form of the character object in virtual space is defined by the coordinates of each body part changing over time. In this way, the movement of the character object is expressed by the coordinates corresponding to each body part of the character object changing over time. In other words, the content of this movement corresponds to the movement form.

[0036] Returning to FIG. 2, the explanation continues. A specific timing in elapsed time T is defined for each character object. The coordinates of each body part at that timing are registered in database 80 as still data. In this example, timing t1 is defined for character object ch1. Still data Sd1 defining the coordinates of each body part of character object ch1 at timing t1 is registered in database 80. Similarly, timings t4 and t7 are defined for character objects ch2 and ch3, respectively, and still data Sd2 and Sd3 are registered in database 80.

[0037] FIG. 4 is a diagram illustrating still data in the first embodiment. Here, still data Sd1 corresponding to character object ch1 is shown as an example. In the still data Sd1, coordinates in the virtual space are associated with each of the body parts Pa, Pb, .... These correspond to the coordinates of each body part when the elapsed time T is t1, i.e., "001:21:35", in the motion data MdCh1 shown in FIG. 3.

[0038] Returning to FIG. 2, the explanation continues. A specific period of elapsed time T is defined for at least a portion of the overlapping appearance periods of multiple character objects. The movement data of the multiple character objects for that period is registered in database 80 as a movement data set. In this example, a specific period from timing t2 to t3 is defined for the combination of character objects ch1 and ch2. Data extracted from the specific period in the movement data MdCh1 of character object ch1 and data extracted from the specific period in the movement data MdCh2 of character object ch2 are registered as a set in database 80 as movement data set Md12. Note that this specific period may be the entire period in which the appearance periods of character objects ch1 and ch2 overlap, rather than just a portion of it.

[0039] Similarly, for the combination of character objects ch1 and ch3, and the combination of character objects ch2 and ch3, specific periods from t5 to t6 and from t8 to t9 are defined, respectively, and action data sets Md13 and Md23 containing action data corresponding to these specific periods are registered in the database 80. Furthermore, for the three combinations, namely the combination of character objects ch1, ch2, and ch3, a specific period from t10 to t11 is defined, and action data set Md123 is registered in the database 80.

[0040] 5 is a diagram illustrating set correspondence data in the first embodiment. The set correspondence data defines the correspondence between combinations of action data sets and character objects. The correspondence is as described above. For example, the combination of character objects ch1 and ch3 corresponds to action data set Md13. As described above, action data set Md13 is data obtained by extracting data for a specific period from t5 to t6 from the action data MdCh1 and MdCh3 for character objects ch1 and ch3.

[0041] Next, the user management data will be described. A user can display, using AR technology, character objects that are permitted for each user from among character objects ch1, ch2, and ch3 on the display unit 57 of the communication terminal 50 that the user operates. The user management data is data that defines the character objects permitted for each user. A user and a character object are associated with each other by, for example, user identification information (such as a user ID) and object identification information (such as an object ID). In other words, the user management data is data that defines character objects that can be displayed on the user's communication terminal 50. Furthermore, the user management data can also be said to be data that defines character objects owned by a user by associating user identification information with object identification information.

[0042] FIG. 6 is a diagram illustrating an example of user management data in the first embodiment. The user management data is data that specifies the character objects that each user U1, U2, ... is permitted to display. For example, user U1 is permitted to display character object ch1. In addition, there are cases where a user, like user U4, is permitted to display multiple character objects ch1 and ch2. This concludes the description of the data registered in database 80.

[0043] [Service provision processing] Next, a process in which the server 10 provides a service to a user will be described. As described above, this service is a service for moving a character object. First, the services provided in relation to this will be described, and then the character object movement service will be described in detail.

[0044] A viewing service that allows a user to view an animation work by executing a predetermined application program on communication terminal 50 is provided to the user. In providing the viewing service, when communication terminal 50 requests viewing of an animation work from server 10, predetermined data is transmitted from server 10 to communication terminal 50, and the animation work becomes viewable on communication terminal 50 based on the data. In this example, server 10 searches database 80 for animation data corresponding to the animation work requested by communication terminal 50 and transmits the data to communication terminal 50. In communication terminal 50, control unit 51 generates video data based on the animation data and displays an image corresponding to the video data on display unit 57. In the case of the animation data shown in FIG. 2, video data corresponding to the passage of time is generated by superimposing images of character objects ch1, ch2, and ch3 that move according to the action data on an image generated based on background data BGd.

[0045] When a user watches such an animation work, for example, one character object randomly selected from a plurality of character objects is permitted to be displayed on the user's communication terminal 50. The character object selected in this manner is associated with the user as a character object permitted to be displayed, and is registered in the user management data by the server 10. If a user wants to be able to display multiple character objects, as in the case of user U4 shown in FIG. 6, this can be achieved, for example, by paying a fee. In this way, the character object permitted to be displayed can be displayed on the communication terminal 50 using AR technology. A brief description will be given of an example of displaying a character object.

[0046] FIG. 7 is a diagram illustrating the relationship between the real space and the imaging direction in the first embodiment. In the following description, the space shown in FIG. 7 is assumed as the real space RS to be imaged. The real space RS includes a floor surface F and wall surfaces W1 and W2. A marker MR is placed on the floor surface F. This marker MR includes information MA indicating its orientation. As the imaging direction, a first direction AR1 directed from the left side of FIG. 7 toward the marker MR and a second direction AR2 directed from the right side of FIG. 7 toward the marker MR are defined.

[0047] Here, it is assumed that a communication terminal 50 (hereinafter sometimes referred to as communication terminal 50-1) operated by a user U1 displays a character object while capturing an image of the real space RS along a first direction AR1. It is also assumed that a user U2 is present near the user U1. The communication terminal 50 operated by the user U2 may be referred to as communication terminal 50-2 below. Note that the relationship between the communication terminal 50 and the user, i.e., the user operating the communication terminal 50, is identified by identification information such as a user ID logged in when an application program is being executed.

[0048] FIG. 8 is a diagram illustrating a display example in the single mode (solo mode) in the first embodiment. An application program is executed in communication terminal 50-1. As a result, a character object is displayed in display area DA of display unit 57, superimposed on an image of real space RS included in the imaging range of imaging unit 56. This character object is determined with reference to user management data (FIG. 6). In this case, it is character object ch1 that user U1 is permitted to display. The form of character object ch1 is specified according to still data Sd1 (FIG. 4). The coordinate system in the virtual space to which still data Sd1 is applied is defined by marker MR. Specifically, the position serving as the reference for the coordinates (the origin of the virtual space) is defined by marker MR, and the X-axis, Y-axis, and Z-axis directions are specified by information MA. At this time, the scale of each axis direction may be defined according to the size of marker MR. Note that the example shown in FIG. 8 illustrates a display mode in which character objects that the user is permitted to display can be displayed. This display mode is referred to as the "single mode" in this example.

[0049] The display area DA shown in Fig. 8 further displays a switch button SB1 for switching from a single mode (solo mode) to a multiple mode (multi mode). When a user operates the switch button SB1, a process for switching the display mode to the multiple mode is executed. In this example, the multiple mode is a display mode in which character objects that are permitted to be displayed by a user operating a communication terminal 50 near the communication terminal 50-1 can be displayed together with the character object ch1 on the communication terminal 50-1. Next, a process for providing a service for realizing a display in the multiple mode will be described.

[0050] 9 is a diagram illustrating a communication flow of the service provision process (pre-setting) in the first embodiment. First, when the communication terminal 50-1 of the user U1 and the communication terminal 50-2 of the user U2 execute an application program, they periodically transmit location information to the server 10. (Step S100). Here, it is assumed that the communication terminal 50-1 is displaying the single mode as described above. In the following description of the communication flow, the processing of the server 10 is executed under the control of the control unit 11, and the processing of the communication terminal 50 is executed under the control of the control unit 51.

[0051] When user U1 operates the switching button SB1 on communication terminal 50-1, communication terminal 50-1 receives an instruction to switch to the multiplexing mode (step S110) and notifies server 10 of the mode switch (step S120). Upon receiving this notification, server 10 searches for users operating communication terminals located within a predetermined distance from communication terminal 50-1 (step S130). In this example, communication terminal 50-2 is identified as the communication terminal closest to communication terminal 50-1, and user U2 of this communication terminal 50-2 is identified. Although not shown in FIG. 9, in this example, users U3 and U5 are also identified. Server 10 transmits data for displaying a user selection screen to communication terminal 50-1 (step S140). Upon receiving this data, communication terminal 50-1 displays the user selection screen on display unit 57 (step S150). The user selection screen presents users identified by the search and allows the user to select a user to be paired with user U1 from among the users. This user selection determines the character objects that are displayed together with character object ch1 in the multiple mode.

[0052] Fig. 10 is a diagram illustrating a user selection screen in the first embodiment. The user selection screen shown in Fig. 10 is an example displayed in the display area DA of the display unit 57 of the communication terminal 50-1. The user selection screen displays selection buttons U2B, U3B, and U5B for selecting identified users U2, U3, and U5, and a confirmation button USB for confirming the selected users. When the user operates the selection button U2B, the display changes to one different from that of unselected users, as shown in Fig. 10. Then, when the user operates the confirmation button USB, the selected user is confirmed.

[0053] Returning to FIG. 9, the explanation will be continued. The communication terminal 50-1 transmits data indicating the selected user to the server 10 (step S160). Here, it is assumed that the user U2 has been selected on the user selection screen. The server 10 transmits data for displaying an acceptance request screen to the communication terminal 50-2 corresponding to the user U2 selected on the communication terminal 50-1 (step S170). Upon receiving this data, the communication terminal 50-2 displays the acceptance request screen on the display unit 57 (step S180). The acceptance request screen is a screen for responding whether or not to accept a combination when the communication terminal 50-2 has been selected by another user as a combination target.

[0054] Fig. 11 is a diagram illustrating a selection acceptance screen in the first embodiment. The selection acceptance screen shown in Fig. 11 is an example displayed in the display area DA of the display unit 57 of the communication terminal 50-2. The selection acceptance screen displays that another user (user U1 in this case) has selected the user as a combination target, and displays buttons for indicating whether to accept or reject the selection, i.e., a Yes button YB1 for acceptance and a No button NB1 for rejection.

[0055] When user U2 operates the No button NB1 on the selection acceptance screen, communication terminal 50-2 transmits a rejection response to server 10. Server 10 notifies communication terminal 50-1 that user U2 has rejected the combination. Upon receiving this notification, communication terminal 50-1 displays on display unit 57 that user U2 has rejected the combination, and returns the display mode to single mode.

[0056] On the other hand, when the user U2 operates the Yes button YB1 on the selection acceptance screen, the communication terminal 50-2 transmits an acceptance response to the server 10. In response to this, the server 10 starts control for displaying in multiple modes on the communication terminal 50-1.

[0057] FIG. 12 is a diagram illustrating the communication flow of the service provision process (multiple mode display control) in the first embodiment. The server 10 searches for an action data set corresponding to the user combination (combination of user U1 and user U2) (step S210). Specifically, the process is executed as follows. First, the server 10 refers to the user management data to identify the character objects corresponding to the combined users. As in this example, when user U1 and user U2 are combined, character objects ch1 and ch2 are identified. Next, the server 10 refers to the set correspondence data to search for an action data set corresponding to the combination of character objects. As in this example, in the case of the combination of character objects ch1 and ch2, action data set Md12 is identified.

[0058] Server 10 transmits the identified action data set Md12 and data related to character objects ch1 and ch2 to communication terminals 50-1 and 50-2 (step S220). If communication terminal 50-1 already holds data related to character object ch1, it only needs to transmit the action data set Md12 and the data related to character object ch2. Similarly, if communication terminal 50-2 already holds data related to character object ch2, it only needs to transmit the data related to action data set Md12 and character object ch1.

[0059] When the communication terminal 50-1 has completed receiving the operation data set Md12 etc. from the server 10, it displays a playback start screen (step S230). The playback start screen is a screen for starting playback in the multiple mode on the communication terminal 50 that has switched to the multiple mode, in this example, the communication terminal 50-1.

[0060] Fig. 13 is a diagram illustrating a playback instruction screen in the first embodiment. The playback start screen shown in Fig. 13 is an example displayed in the display area DA of the display unit 57 of the communication terminal 50-1. The playback start screen displays a message encouraging playback in multiple modes and buttons for indicating whether to start playback or put playback on hold, i.e., a Yes button YB2 for instructing the start of playback and a Later button NB2 for putting playback on hold.

[0061] When the user U1 operates the Later button NB2 on the playback start screen, the playback start screen is displayed again after a predetermined time, and during the time until that time, another process can be executed on the communication terminal 50-1.

[0062] On the other hand, when the user U1 operates the Yes button YB2 on the playback start screen, the communication terminal 50-1 transmits a playback start instruction to the server 10 (step S240). Upon receiving this instruction, the server 10 transmits a synchronization signal to the communication terminals 50-1, 50-2 (step S250). Upon receiving the synchronization signal, the communication terminal 50-1 displays the character objects ch1 and ch2 in the display area DA of the display unit 57 so that they move in accordance with the movement data set Md12. The time course of the movements of the character objects ch1 and ch2 is controlled by the synchronization signal.

[0063] FIG. 14 is a diagram illustrating a display example in the multiple mode in the first embodiment (when an image is captured along the first direction AR1). Character objects ch1 and ch2 are displayed in the display area DA of the display unit 57 of the communication terminal 50-1. The example shown in FIG. 14 shows a state in which a predetermined time has elapsed since playback of the actions of the character objects ch1 and ch2 was started in the multiple mode display. The elapsed time is indicated by a seek bar SK. Furthermore, a switch button SB2 for switching from the multiple mode to the single mode is displayed in the display area DA. The user can operate the seek bar SK to change the playback position (the playback portion within the action data) or can operate the switch button SB2 to end the multiple mode.

[0064] Because the character objects ch1 and ch2 move according to the movement data set Md12, the movements from t2 to t3 in the animation data shown in FIG. 2 are reproduced. This reproduces a portion of the animation work through AR display. Here, in the display area DA, the character objects ch1 and ch2 are displayed superimposed on an image of the real space RS included in the imaging range of the imaging unit 56. The coordinate system in the virtual space to which the movement data set Md12 is applied is the same as in the single mode. That is, the position serving as the reference for the coordinates (the origin of the virtual space) is defined by the marker MR, and the X-axis, Y-axis, and Z-axis directions are specified by the information MA.

[0065] In this example, the background is an image of the real space RS, and the background BG in the animation data is not used. Note that the background BG processed into a semi-transparent image may be further superimposed on the image of the real space RS.

[0066] Here, the display in multiple modes may be realized only on communication terminal 50-1, but the same display as that on communication terminal 50-1 may also be realized on communication terminal 50-2 of user U2, which is the combined target. In this case, the movements of character objects ch1 and ch2 may be synchronized by a synchronization signal. By using the synchronization signal, the movements of character objects ch1 and ch2 can be displayed at the same elapsed time on communication terminal 50-1 and communication terminal 50-2.

[0067] FIG. 15 is a diagram illustrating a display example in multiple modes (when an image is captured along the second direction AR2) in the first embodiment. FIG. 14 illustrates the display content when the communication terminal 50-1 captures an image of the real space RS along the first direction AR1. On the other hand, FIG. 15 illustrates the display content when the communication terminal 50-2 captures an image of the real space RS along the second direction AR2. When viewed from the second direction AR2, the orientation of the marker MR is different. Because both are displayed using a common coordinate system, in the display example shown in FIG. 15, the X-axis, Y-axis, and Z-axis directions in the virtual space relative to the screen are different from those in the display example shown in FIG. 14. As a result, the appearance of the character objects ch1 and ch2 differs between the communication terminal 50-1 and the communication terminal 50-2. However, in this example, the passage of time is synchronized between the communication terminal 50-1 and the communication terminal 50-2 by a synchronization signal, and therefore the behavior of the character objects ch1 and ch2 is the same.

[0068] In the above example, depending on the combination of users U1 and U2, character objects ch1 and ch2 can be displayed on the display unit 57 while moving together. This combination is not limited to two, and may be three or more. In the example of user management data shown in FIG. 6, depending on the combination of users U1, U2, and U3, character objects ch1, ch2, and ch3 can be displayed while moving together. Similarly, depending on the combination of users U3 and U4, character objects ch1, ch2, and ch3 can be displayed while moving together. Note that user U4 is permitted to display both character objects ch1 and ch2, and therefore, on the communication terminal 50 of user U4, either one of them can be displayed in single mode, or both can be displayed in multiple mode.

[0069] In this way, even if each user has only a few character objects that they are permitted to display, they can cooperate with other users to display multiple character objects together. In this case, by combining multiple character objects, they can not only be simply displayed, but also be made to perform actions that could not be achieved with a single character object and displayed on the display unit 57. Therefore, by combining character objects with other character objects, a variety of expressions can be provided to the user, and it is possible to prevent the user from losing interest in the character objects.

[0070] [Configuration of the display control unit used in the service provision process] Next, the functional (display control unit) configuration for realizing the processing shown in FIG. 12 will be described with reference to FIG.

[0071] Fig. 16 is a block diagram showing the configuration of the display control unit in the first embodiment. Fig. 16 shows an example in which the display control unit 700 is realized by cooperation between the server 10 and the communication terminal 50. The action determination unit 600 is realized in the server 10. The action determination unit 600 has a function of performing the processing of step S210 shown in Fig. 12. That is, the action determination unit 600 refers to the user management data to identify character objects that are permitted to be displayed for multiple users, and refers to the set correspondence data to identify an action data set corresponding to a combination of character objects. In this way, the action determination unit 600 determines the actions of the multiple character objects.

[0072] The display control unit 700 includes a data providing unit 710, a movement control unit 730, and a display superimposing unit 750. The data providing unit 710 reads out, from the database 80, movement data included in the movement data set and data related to a plurality of character objects to which the movement data is to be applied, and provides this to the movement control unit 730. The movement control unit 730 applies the movement data to each character object to generate an image in which the character object is moved.

[0073] The display superimposing unit 750 superimposes the image generated by the operation control unit 730 on the image in the imaging range of the imaging unit 56, and displays the image on the display unit 57. At this time, the X-axis direction, the Y-axis direction, and the Z-axis direction in the virtual space may be corrected based on the attitude of the communication terminal 50, etc., using behavior information from the behavior measurement unit 55.

[0074] In the first embodiment, of the display control unit 700, the function of the data providing unit 710 is realized in the server 10, and the functions of the action control unit 730 and the display superimposing unit 750 are realized in the communication terminal 50. The server 10 controls the action of the character object displayed on the communication terminal 50 by providing the function of the data providing unit 710, that is, the action data set and the like, to the communication terminal 50.

[0075] Second Embodiment Of the display control unit 700, the functions of the data providing unit 710 and the action control unit 730 may be realized in the server 10. In this case, the server 10 will transmit to the communication terminal 50 image data of the character object in action, rather than transmitting an action data set or the like to the communication terminal 50.

[0076] Third Embodiment Of the display control unit 700, all functions, i.e., the functions of the data providing unit 710, the operation control unit 730, and the display superimposing unit 750, may be realized in the server 10. In this case, the server 10 acquires an image captured by the communication terminal 50 and uses the image in the display superimposing unit 750.

[0077] <Fourth embodiment> Of the display control unit 700, all functions, i.e., the functions of the data providing unit 710, the operation control unit 730, and the display superimposing unit 750, may be realized in the communication terminal 50. In this case, the communication terminal 50 connects to the database 80 via the network NW to realize the functions of the data providing unit 710.

[0078] <Modification> Although one embodiment of the present invention has been described above, the above-described embodiments can be applied by combining or replacing each other. Furthermore, the above-described embodiments can also be implemented by modifying them as follows.

[0079] (1) In the above-described embodiment, the character object, still data, and motion data were generated by extracting them from animation data, but the original animation data may not be necessary. In other words, each of the data may be generated separately for the purpose of implementing the data provision service.

[0080] (2) The process for obtaining permission to display a character object is not limited to the method described in the above embodiment. Display may be permitted through a lottery system known as gacha, or through a code distributed at an event or the like.

[0081] (3) In the above-described embodiment, users to be combined can be selected based on the distance between the users. However, users may also be combined if they satisfy a predetermined association condition by another method. Users to be combined who are close to each other may be identified, for example, as users of communication terminals 50 capturing an image of an object determined to be the same by image recognition, or as users of communication terminals 50 with which communication is possible via short-range wireless communication. The object may be, for example, the marker MR described in the above-described embodiment, or any other object that is included in the image capture range and can be extracted by image analysis. In other words, the predetermined association condition may be that the users have specific common information as determined by image analysis of the image capture range. Note that users to be combined may be users who are not necessarily close to each other, and may be users who are associated as friends on a social networking site, for example.

[0082] (4) In the above-described embodiment, a coordinate system in a virtual space is identified using a marker MR to display a character object. However, the present invention is not limited to using a marker MR. For example, three-dimensional information of the real space RS may be acquired and associated with a coordinate system in a virtual space by using a V-SLAM (Visual Simultaneous Localization and Mapping) technique. A part of the analyzed three-dimensional information may be recognized as a characteristic part, and the characteristic part may be replaced with a marker MR.

[0083] (5) In the above-described embodiment, the action data set is determined based on the combination of character objects. However, the action data set may be determined using other factors. As a result, different action data sets may be specified even for the same combination of character objects. For example, an item object used by a specific character object may be specified for that character object. As a result, set correspondence data may be specified so that the action data set specified differs depending on whether or not an item object exists as a combination target. Specifically, the action data set applied in multiple modes differs depending on whether or not a user permitted to display character object ch1 is also permitted to display an item object used by character object ch1 (an item object associated with character object ch1).

[0084] (6) In the above-described embodiment, the timing of the still data corresponding to each character object is defined as a timing that does not overlap with the appearance period of other character objects, but this is not limited to this. That is, the timing of the still data corresponding to a character object may be defined as a timing that overlaps with the appearance period of other character objects, or may be defined as a timing that overlaps with the period used for the motion data set. For example, the still data Sd1 may be defined as the coordinates of each part at any timing between t2 and t3, when the appearance period of the character object ch2 overlaps with that of the character object ch2.

[0085] As a result, the still data Sd1 is an extracted portion of the motion form of the character object ch1 defined in the motion data set Md12. Therefore, even if there is no overall data such as animation data, still data can be efficiently generated from the motion data, and the user can be motivated to see the previous and subsequent motions of the character object ch1 displayed in single mode. On the other hand, as in the above-described embodiment, the still data Sd1 can also be a form that is not included in the motion form of the character object ch1 defined in the motion data set Md12. In this case, the still data Sd1 needs to be generated as data of a motion form separate from the motion data set Md12, but the user can enjoy a display in multiple mode that is completely different from the character object ch1 in single mode.

[0086] (7) In the above-described embodiment, still image data is applied to the character object displayed in the single mode, but motion data that forms a moving image may also be applied. Even in this case, different motion data may be applied to the character object in the single mode and the multiple mode.

[0087] (8) The above-described embodiment describes an example in which the present invention is applied to the movement control of a character object, but the present invention can be similarly applied to the movement control of not only character objects but also objects having three-dimensional shapes. [Explanation of symbols]

[0088] 10...server, 11...control unit, 12...storage unit, 13...communication unit, 50, 50-1, 50-2...communication terminal, 51...control unit, 52...storage unit, 53...communication unit, 54...position measurement unit, 55...behavior measurement unit, 56...imaging unit, 57...display unit, 58...operation unit, 80...database, 600...action determination unit, 700...display control unit, 710...data providing unit, 730...action control unit, 750...display superimposition unit, 1000...system

Claims

1. a determination unit that determines motion forms of the first character object and the second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; a display control unit that controls the first terminal to display the first character object and the second character object according to the action form so as to be superimposed on an image captured by the first terminal; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the first terminal and the second terminal satisfy a predetermined association condition.

3. The information processing device according to claim 2 , wherein the association condition includes that the first terminal and the second terminal are closer than a predetermined distance.

4. The information processing device according to claim 2 , wherein the association condition includes that the same object is included in the image captured by the first terminal and the image captured by the second terminal.

5. 5. An information processing device according to claim 2, wherein, when the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal is an image based on a part of the action form, and the second character object that can be displayed on the second terminal is an image based on a part of the action form.

6. 5. An information processing device according to claim 2, wherein, when the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal is an image based on a form that is not included in the action form, and the second character object that can be displayed on the second terminal is an image based on a form that is not included in the action form.

7. 7. The information processing device according to claim 1, further comprising controlling the second terminal so as to display the first character object and the second character object according to the action form superimposed on an image captured by the second terminal.

8. 8. The information processing device according to claim 1, wherein, when an item object that can be displayed together with the first character object exists on the first terminal, the determination unit further determines the action form based on the item object.

9. 9. The information processing device according to claim 1, wherein the movement form corresponds to a movement form defined for at least a portion of a period during which the first character object and the second character object are used, among data defining movement forms of a plurality of character objects including the first character object and the second character object.

10. determining motion forms of the first character object and the second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; controlling the first terminal so as to display the first character object and the second character object in accordance with the action form superimposed on an image captured by the first terminal; A program that causes a computer to execute the following.

Images