Information processing apparatus and program
The information processing device synchronizes character objects across terminals based on proximity and shared targets, addressing the limitations of existing AR technologies to enhance collaborative user interactions.
Patent Information
- Application Number
- JP2024151660
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2039-03-25
AI Technical Summary
Existing AR technologies do not effectively allow multiple character objects to be displayed and synchronized across different terminals, limiting collaborative interactions between users.
An information processing device and program that determines a motion mode for character objects displayed on multiple terminals, synchronizing their movements based on predetermined conditions such as proximity and shared targets, using a common coordinate system to superimpose and display character objects on captured images.
Enables multiple character objects to be displayed and synchronized across terminals, enhancing collaborative experiences by allowing more expressive and engaging interactions.
Smart Images

Figure 2024163234000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device and a program. [Background technology]
[0002] Augmented reality (AR) technology (hereinafter referred to as AR technology) has been used in mobile terminals such as smartphones and game consoles. AR technology is a technology that displays on a display an image obtained by capturing an image of real space with a camera and superimposing an object placed in a virtual space, and this technology makes it possible to make an object in a virtual space appear as if it exists in the real space. The AR technology is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2015-41126 A Summary of the Invention [Problem to be solved by the invention]
[0004] As an application example of AR technology, a 3D virtual character object owned by a user is displayed on a mobile terminal as if it exists in the real world. In addition to using a character object owned by a user alone, it is desired that a user be able to use the character object in cooperation with other users.
[0005] One object of the present invention is to provide an information processing device and program that can operate multiple character objects in a manner that corresponds to a combination of character objects that can be displayed on one terminal and character objects that can be displayed on another terminal. [Means for solving the problem]
[0006] According to one embodiment of the present invention, there is provided an information processing device including: a determination unit that determines an action form of a first character object and a second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; and a display control unit that controls the first terminal to display the first character object and the second character object according to the action form, superimposed on an image captured on the first terminal.
[0007] The first terminal and the second terminal may satisfy a predetermined association condition.
[0008] The association condition may include that the first terminal and the second terminal are closer than a predetermined distance.
[0009] The association condition may include that the same object is included in the image captured by the first terminal and the image captured by the second terminal.
[0010] When the first terminal and the second terminal do not satisfy the specified association condition, the first character object displayable on the first terminal may be an image based on a part of the movement form, and the second character object displayable on the second terminal may be an image based on a part of the movement form.
[0011] In a state in which the first terminal and the second terminal do not satisfy the specified association condition, the first character object displayable on the first terminal may be an image based on a form not included in the operation form, and the second character object displayable on the second terminal may be an image based on a form not included in the operation form.
[0012] The method may further include controlling the second terminal to display the first character object and the second character object according to the action form, superimposed on an image captured by the second terminal.
[0013] The display control unit may control the first terminal to display the first character object and the second character object using a coordinate system set for an image captured on the first terminal, and may control the second terminal to superimpose the first character object and the second character object on an image captured on the second terminal based on the coordinate system and display them in synchronization with the display on the first terminal.
[0014] When there is an item object that can be displayed together with the first character object on the first terminal, the determination unit may further determine the action form based on the item object.
[0015] The movement form may correspond to a movement form defined in data defining movement forms of a plurality of character objects including the first character object and the second character object, during at least a portion of a period during which the first character object and the second character object are used.
[0016] Furthermore, according to one embodiment of the present invention, a program is provided for causing a computer to control a first terminal to determine motion forms of a first character object and a second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal, and to display the first character object and the second character object in accordance with the motion forms by superimposing them on an image captured on the first terminal. Effect of the Invention
[0017] According to one embodiment of the present invention, an information processing device and program can be provided that can operate multiple character objects in a form that corresponds to a combination of character objects that can be displayed on one terminal and character objects that can be displayed on another terminal. [Brief description of the drawings]
[0018] [Figure 1] 1 is a block diagram showing a configuration of a system in a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating animation data in the first embodiment. [Diagram 3] FIG. 4 is a diagram illustrating operation data in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating still data in the first embodiment. [Diagram 5] FIG. 4 is a diagram illustrating set correspondence data in the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating user management data in the first embodiment. [Figure 7] 4A to 4C are diagrams illustrating the relationship between a real space and an imaging direction in the first embodiment. [Figure 8] FIG. 4 is a diagram illustrating a display example in a single mode in the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating a communication flow of a service providing process (presetting) in the first embodiment. [Figure 10] FIG. 4 is a diagram illustrating a user selection screen in the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating a selection acceptance screen in the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating a communication flow of the service providing process (multiple mode display control) in the first embodiment. [Figure 13] FIG. 4 is a diagram illustrating a playback instruction screen in the first embodiment. [Figure 14]10A and 10B are diagrams illustrating a display example in a multiple mode in the first embodiment (when images are taken along a first direction AR1). [Figure 15] 13 is a diagram for explaining a display example in a multiple mode in the first embodiment (when imaging is performed along a second direction AR2). FIG. [Figure 16] FIG. 2 is a block diagram showing a configuration of a display control unit in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Hereinafter, a system according to an embodiment of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example of an embodiment of the present invention, and the present invention is not limited to these embodiments.
[0020] First Embodiment A system according to a first embodiment of the present invention will be described in detail with reference to the drawings.
[0021] [overview] 1 is a block diagram showing a system configuration in the first embodiment. The system 1000 includes a server 10 (information processing device) and a communication terminal 50. The server 10 and the communication terminal 50 are connected via a network NW such as the Internet. In this example, the communication terminal 50 is a smartphone. The communication terminal 50 may also be a portable device such as a tablet terminal or a portable game machine.
[0022] Although the server 10 is illustrated as one device in FIG. 1, it may be composed of multiple devices. The server 10 provides application programs to the communication terminal 50, and provides various services to the communication terminal 50 that executes the application programs. The application programs are recorded in a recording medium (storage device) to which the server 10 can be connected. The various services include a character object operation service described below. The various services may include, for example, an SNS (social networking service).
[0023] An application program for receiving services from the server 10 is installed in the communication terminal 50. This program allows the communication terminal 50 to display character objects using AR technology. In this example program, when a predetermined condition is satisfied, other character objects can be further displayed in the communication terminal 50, and the displayed character objects can be made to perform special actions. The configuration of the communication terminal 50 will be described below.
[0024] [Hardware configuration of communication terminal 50] The communication terminal 50 includes a control unit 51, a storage unit 52, a communication unit 53, a position measurement unit 54, a behavior measurement unit 55, an imaging unit 56, a display unit 57, and an operation unit 58. The control unit 51 is an example of a computer including an arithmetic processing circuit such as a CPU. The control unit 51 executes an application program stored in the storage unit 52 by the CPU, and realizes various functions in the communication terminal 50.
[0025] The storage unit 52 includes a storage device, and stores an application program that is downloaded from the server 10 via the network NW and installed. The storage unit 52 also stores various data used when the application program is executed. The program may be provided in a state in which it is recorded on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the communication terminal 50 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 52 included in the communication terminal 50, or may be a medium used for the storage unit 52.
[0026] The communication unit 53 includes a communication module, and is connected to the network NW under the control of the control unit 51 to transmit and receive information to and from other devices connected to the network NW. The position measurement unit 54 measures the position of the communication terminal 50 using technology such as GPS, and outputs information indicating the position (hereinafter, sometimes referred to as position information) to the control unit 51. The behavior measurement unit 55 includes an acceleration sensor, a gyro sensor, and the like, and measures the behavior of the communication terminal 50, and outputs information indicating the behavior (hereinafter, sometimes referred to as behavior information) to the control unit 51. The behavior information includes, for example, information for identifying the amount of movement and the amount of rotation of the communication terminal 50. Note that information for identifying the orientation of the communication terminal 50 may be generated in the communication terminal 50 by using a geomagnetic sensor or the like in combination.
[0027] The imaging unit 56 is a camera including a CCD sensor or a CMOS sensor, and outputs information indicating an image in the imaging range to the control unit 51. The information indicating the image is either a still image or a moving image. The display unit 57 is a display device such as a liquid crystal display or an organic EL display, and displays various screens in the display area based on the control of the control unit 51. The imaging unit 56 and the display unit 57 are arranged on opposite sides of each other via the housing of the communication terminal 50. In other words, the positional relationship between the imaging range of the imaging unit 56 and the display area of the display unit 57 is specified so that when the user visually recognizes the display area of the display unit 57, the imaging range of the imaging unit 56 includes an area beyond the display area. In this example, the display unit 57 can display using AR technology (hereinafter, sometimes referred to as AR display) under the control of the control unit 51 as described later. More specifically, the display unit 57 can display a character object in a virtual space by superimposing it on an image corresponding to the imaging range of the imaging unit 56.
[0028] In this example, operation unit 58 includes a touch sensor, and outputs information corresponding to the position operated by the user to control unit 51. This touch sensor is provided on the display area of display unit 57. In other words, display unit 57 and operation unit 58 form a touch panel.
[0029] [Server 10 hardware configuration] The server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is an example of a computer including an arithmetic processing circuit such as a CPU. The control unit 11 executes a control program stored in the storage unit 12 by the CPU to realize various functions in the server 10. In this way, the server 10 provides services to the user via the communication terminal 50 by the functions realized in the communication terminal 50 and the functions realized in the server 10.
[0030] The storage unit 12 includes a storage device and stores a control program. The storage unit 12 also stores various data used when the control program is executed. The program may be provided in a state recorded on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the server 10 only needs to include an interface for connecting the recording medium. Here, the storage medium may be defined as a medium separate from the storage unit 12 included in the server 10, or may be a medium used in the storage unit 12.
[0031] The communication unit 13 includes a communication module, and is connected to the network NW under the control of the control unit 11 to transmit and receive information to and from other devices connected to the network NW. In this example, the communication unit 13 is also connected to a database 80 to transmit and receive information. Data related to services provided by the server 10 is registered in the database 80. The communication unit 13 may be connected to the database 80 via the network NW. The data registered in the database 80 may be stored in the storage unit 12. In this case, the database 80 may not exist.
[0032] [Data registered in Database 80] The data registered in the database 80, that is, the data related to the services provided by the server 10, includes animation data, motion data, still data, set corresponding data, and user management data. Next, these data will be described.
[0033] FIG. 2 is a diagram for explaining animation data in the first embodiment. Animation data is data constituting an animation work. Animation data includes data defining the contents of a moving image, and includes motion data MdCh1, MdCh2, MdCh3 corresponding to three types of character objects ch1, ch2, ch3, and background data BGd corresponding to the background BG. The horizontal axis indicates the elapsed time T from the start of the moving image. The background data BGd is data showing a moving image in a virtual space other than the character object, and the contents of the moving image are defined corresponding to the elapsed time T. A character object is defined by various data. In this example, in order to define the movement of a character object, data on a plurality of parts of the character (for example, parts corresponding to joints such as shoulders, elbows, and wrists) and a skeleton connecting the plurality of parts are used. These data define the pose of the character object, and the movement form of the character object is defined by changing the position of each part over time. Data defining the appearance of the character object (body surface, hair, clothing, etc.) is also used. The movement data defining the movement form of a character object will be described in more detail.
[0034] The motion data MdCh1, MdCh2, and MdCh3 corresponding to the character objects define the motion of the character objects corresponding to the period in which the respective character objects appear. For example, character object ch1 appears in period Sc1a and period Sc1b, and motion data MdCh1 is defined for that period. Similarly, character object ch2 appears in period Sc2a and period Sc2b, and motion data MdCh2 is defined for that period. Character object ch3 appears in period Sc3a, and motion data MdCh3 is defined for that period.
[0035] FIG. 3 is a diagram for explaining the motion data in the first embodiment. Here, motion data MdCh1 corresponding to character object ch1 is illustrated. In the motion data MdCh1, coordinates in the virtual space corresponding to a plurality of parts (Pa, Pb, ...) of the character object are associated with elapsed time T. When the elapsed time T is "001:21:35", the X coordinate, Y coordinate, and Z coordinate of part Pa are Xa2, Ya2, and Za2, respectively. The motion form of the character object in the virtual space is defined by the time change of the coordinates of each part. In this way, the motion of the character object is expressed by the time change of the coordinates corresponding to each part of the character object. In other words, the content of this motion corresponds to the motion form.
[0036] Returning to FIG. 2, the explanation continues. A specific timing in elapsed time T is defined for each character object. The coordinates of each body part at that timing are registered in database 80 as still data. In this example, timing t1 is defined for character object ch1. Still data Sd1 defining the coordinates of each body part of character object ch1 at timing t1 is registered in database 80. Similarly, timings t4 and t7 are defined for character objects ch2 and ch3, respectively, and still data Sd2 and Sd3 are registered in database 80.
[0037] 4 is a diagram for explaining still data in the first embodiment. Here, still data Sd1 corresponding to character object ch1 is illustrated. The still data Sd1 is associated with coordinates in the virtual space corresponding to each part Pa, Pb, .... This corresponds to the coordinates of each part when the elapsed time T is t1, i.e., "001:21:35", in the motion data MdCh1 shown in FIG. 3.
[0038] Returning to FIG. 2, the explanation will be continued. In at least a part of the overlapping portion of the appearance periods of the multiple character objects, a specific period in the elapsed time T is defined. The motion data of the multiple character objects in that period is registered in the database 80 as a motion data set. In this example, a specific period from timing t2 to t3 is defined for the combination of character objects ch1 and ch2. Data obtained by extracting a portion of the specific period in the motion data MdCh1 of character object ch1 and data obtained by extracting a portion of the specific period in the motion data MdCh2 of character object ch2 are registered in the database 80 as a motion data set Md12. Note that this specific period may be the entirety of the overlapping period of the appearance periods of the character objects ch1 and ch2, rather than a part of it.
[0039] Similarly, for the combination of character objects ch1, ch3, and the combination of character objects ch2, ch3, specific periods from t5 to t6 and from t8 to t9 are defined, respectively, and motion data sets Md13 and Md23 containing motion data corresponding to the specific periods are registered in the database 80. Furthermore, for the three combinations, namely the combination of character objects ch1, ch2, and ch3, specific periods from t10 to t11 are defined, and motion data set Md123 is registered in the database 80.
[0040] 5 is a diagram for explaining the set correspondence data in the first embodiment. The set correspondence data specifies the mutual correspondence between the combination of the action data set and the character object. The correspondence is as described above. For example, the combination of the character objects ch1 and ch3 corresponds to the action data set Md13. As described above, the action data set Md13 is data obtained by extracting data in a specific period from t5 to t6 from the action data MdCh1 and MdCh3 for the character objects ch1 and ch3.
[0041] Next, the user management data will be described. The user can display, by AR technology, character objects that are permitted among character objects ch1, ch2, and ch3 on display unit 57 of communication terminal 50 operated by the user. The user management data is data that defines the character objects permitted for each user. The user and the character objects are associated with each other by, for example, user identification information (user ID, etc.) and object identification information (object ID, etc.). In other words, the user management data is data that defines the character objects that can be displayed on the user's communication terminal 50. Furthermore, the user management data can also be said to be data that defines the character objects owned by the user by associating the user identification information with the object identification information.
[0042] FIG. 6 is a diagram for explaining an example of user management data in the first embodiment. The user management data is data that specifies the character objects that each user U1, U2, ... is permitted to display. For example, user U1 is permitted to display character object ch1. Also, there are cases where user U4 is permitted to display multiple character objects ch1 and ch2. This concludes the explanation of the data registered in database 80.
[0043] [Service provision processing] Next, a process in which the server 10 provides a service to a user will be described. As described above, this service is a service for moving a character object. First, a related service will be described, and then the character object movement service will be described in detail.
[0044] A viewing service that allows a user to view an animation work by executing a predetermined application program on the communication terminal 50 is provided to the user. In providing the viewing service, when the communication terminal 50 requests the server 10 to view an animation work, predetermined data is transmitted from the server 10 to the communication terminal 50, and the animation work can be viewed on the communication terminal 50 based on the data. In this example, the server 10 searches the database 80 for animation data corresponding to the animation work requested by the communication terminal 50 and transmits it to the communication terminal 50. In the communication terminal 50, the control unit 51 generates video data based on the animation data and displays an image corresponding to the video data on the display unit 57. In the case of the animation data shown in FIG. 2, video data corresponding to the passage of time is generated by superimposing images of the character objects ch1, ch2, and ch3 that move according to the action data on an image generated based on the background data BGd.
[0045] When such an animation work is viewed, for example, one character object randomly selected from a plurality of character objects is permitted to be displayed on the communication terminal 50 of the user. The character object selected in this manner is associated with the user as a character object permitted to be displayed, and is registered in the user management data by the server 10. When a user wants to be able to display a plurality of character objects, as in the case of user U4 shown in FIG. 6, this can be realized, for example, by charging. In this way, the character object permitted to be displayed can be displayed on the communication terminal 50 using AR technology. A brief description will be given of a display example of the character object.
[0046] FIG. 7 is a diagram for explaining the relationship between the real space and the imaging direction in the first embodiment. In the following explanation, the space shown in FIG. 7 is assumed as the real space RS to be imaged. The real space RS includes a floor surface F and walls W1 and W2. A marker MR is placed on the floor surface F. This marker MR includes information MA indicating an orientation. As the imaging direction, a first direction AR1 from the left side of FIG. 7 toward the marker MR and a second direction AR2 from the right side of FIG. 7 toward the marker MR are defined.
[0047] Here, it is assumed that a communication terminal 50 (hereinafter, sometimes referred to as communication terminal 50-1) operated by a user U1 displays a character object while capturing an image of the real space RS along a first direction AR1. It is also assumed that a user U2 is present near the user U1. The communication terminal 50 operated by the user U2 may be referred to as communication terminal 50-2 below. Note that the relationship between the communication terminal 50 and the user, i.e., the user operating the communication terminal 50, is specified by identification information such as a user ID logged in when an application program is being executed.
[0048] FIG. 8 is a diagram for explaining a display example in a single mode (solo mode) in the first embodiment. An application program is executed in the communication terminal 50-1. As a result, in the display area DA of the display unit 57, a character object is displayed superimposed on an image of the real space RS included in the imaging range of the imaging unit 56. This character object is determined with reference to the user management data (FIG. 6), and in this case, it is the character object ch1 permitted to be displayed by the user U1. In addition, the form of the character object ch1 is specified according to the still data Sd1 (FIG. 4). The coordinate system in the virtual space to which the still data Sd1 is applied is defined by the marker MR. Specifically, the position serving as the reference of the coordinates (the origin of the virtual space) is defined by the marker MR, and the X-axis direction, the Y-axis direction, and the Z-axis direction are specified by the information MA. At this time, the scale of each axis direction may be defined according to the size of the marker MR. Note that the example shown in FIG. 8 shows a display mode in which a character object permitted to be displayed by the user can be displayed. This display mode is called a "single mode" in this example.
[0049] The display area DA shown in FIG. 8 further displays a switching button SB1 for switching from a single mode (solo mode) to a multiple mode (multi mode). When a user operates the switching button SB1, a process for switching the display mode to the multiple mode is executed. In this example, the multiple mode is a display mode in which a character object that is permitted to be displayed by a user operating a communication terminal 50 near the communication terminal 50-1 can be displayed together with the character object ch1 on the communication terminal 50-1. Next, a process for providing a service for realizing a display in the multiple mode will be described.
[0050] 9 is a diagram for explaining a communication flow of the service provision process (pre-setting) in the first embodiment. First, when the communication terminal 50-1 of the user U1 and the communication terminal 50-2 of the user U2 execute an application program, they periodically transmit position information to the server 10. (Step S100). Here, it is assumed that the communication terminal 50-1 is displaying in the single mode as described above. In the following description of the communication flow, the process of the server 10 is executed under the control of the control unit 11, and the process of the communication terminal 50 is executed under the control of the control unit 51.
[0051] When the user U1 operates the switching button SB1 on the communication terminal 50-1, the communication terminal 50-1 accepts an instruction to switch to the multiple mode (step S110) and notifies the server 10 of the mode switching (step S120). When the server 10 receives this notification, it searches for a user who operates a communication terminal located within a range shorter than a predetermined distance from the communication terminal 50-1 (step S130). In this example, the communication terminal 50-2 is identified as a communication terminal close to the communication terminal 50-1, and the user U2 of this communication terminal 50-2 is identified. Although not shown in FIG. 9, in this example, it is also assumed that the user U3 and the user U5 are identified. The server 10 transmits data for displaying a user selection screen to the communication terminal 50-1 (step S140). When the communication terminal 50-1 receives this data, it displays the user selection screen on the display unit 57 (step S150). The user selection screen is a screen that presents users identified by the search and allows the user to select a user to be combined with the user (user U1) from among them. This user selection determines the character objects that are displayed together with character object ch1 in the multiple mode.
[0052] Fig. 10 is a diagram for explaining a user selection screen in the first embodiment. The user selection screen shown in Fig. 10 is an example displayed in the display area DA of the display unit 57 in the communication terminal 50-1. The user selection screen displays selection buttons U2B, U3B, and U5B for selecting the identified users U2, U3, and U5, and a decision button USB for deciding the selected users. When the user operates the selection button U2B, the display changes to a different display from that of unselected users, as shown in Fig. 10. Then, when the user operates the decision button USB, the selected user is decided.
[0053] Returning to FIG. 9, the explanation will be continued. The communication terminal 50-1 transmits data indicating the selected user to the server 10 (step S160). Here, it is assumed that the user U2 has been determined on the user selection screen. The server 10 transmits data for displaying an acceptance request screen to the communication terminal 50-2 corresponding to the user U2 determined on the communication terminal 50-1 (step S170). Upon receiving this data, the communication terminal 50-2 displays the acceptance request screen on the display unit 57 (step S180). The acceptance request screen is a screen for responding whether or not to accept a combination that has been designated by another user.
[0054] Fig. 11 is a diagram illustrating a selection acceptance screen in the first embodiment. The selection acceptance screen shown in Fig. 11 is an example displayed in the display area DA of the display unit 57 in the communication terminal 50-2. The selection acceptance screen displays that the user has been selected as a combination target by another user (here, user U1), and buttons for indicating whether to accept or reject the selection, that is, a Yes button YB1 for acceptance and a No button NB1 for rejection.
[0055] When the user U2 operates the No button NB1 on the selection acceptance screen, the communication terminal 50-2 transmits a rejection response to the server 10. The server 10 notifies the communication terminal 50-1 that the user U2 has rejected the combination. When the communication terminal 50-1 receives this notification, the display unit 57 displays that the user U2 has rejected the combination, and the display mode returns to the single mode.
[0056] On the other hand, when the user U2 operates the Yes button YB1 on the selection acceptance screen, the communication terminal 50-2 transmits an acceptance response to the server 10. In response to this, the server 10 starts control for displaying in multiple modes on the communication terminal 50-1.
[0057] FIG. 12 is a diagram for explaining a communication flow of the service provision process (multiple mode display control) in the first embodiment. The server 10 searches for an action data set corresponding to a combination of users (combination of user U1 and user U2) (step S210). Specifically, the process is executed as follows. First, the server 10 refers to the user management data to identify a character object corresponding to the combined users. As in this example, when user U1 and user U2 are combined, character objects ch1 and ch2 are identified. Next, the server 10 refers to the set correspondence data to search for an action data set corresponding to the combination of character objects. As in this example, in the case of a combination of character objects ch1 and ch2, the action data set Md12 is identified.
[0058] The server 10 transmits the identified action data set Md12 and data related to the character objects ch1 and ch2 to the communication terminals 50-1 and 50-2 (step S220). If the communication terminal 50-1 already holds data related to the character object ch1, the action data set Md12 and data related to the character object ch2 may be transmitted. Similarly, if the communication terminal 50-2 already holds data related to the character object ch2, the action data set Md12 and data related to the character object ch1 may be transmitted.
[0059] When the communication terminal 50-1 has completed receiving the operation data set Md12 and the like from the server 10, it displays a playback start screen (step S230). The playback start screen is a screen for starting playback in the multiple mode on the communication terminal 50 that has switched to the multiple mode, that is, on the communication terminal 50-1 in this example.
[0060] Fig. 13 is a diagram for explaining a playback instruction screen in the first embodiment. The playback start screen shown in Fig. 13 is an example displayed in the display area DA of the display unit 57 in the communication terminal 50-1. The playback start screen displays a description encouraging playback in multiple modes, and buttons for indicating an intention to start or hold playback, that is, a Yes button YB2 for instructing the start of playback and a Later button NB2 for holding playback.
[0061] When the user U1 operates the Later button NB2 on the playback start screen, the playback start screen is displayed again after a predetermined time, and during the time until that time, another process can be executed on the communication terminal 50-1.
[0062] On the other hand, when the user U1 operates the Yes button YB2 on the playback start screen, the communication terminal 50-1 transmits a playback start instruction to the server 10 (step S240). Upon receiving this instruction, the server 10 transmits a synchronization signal to the communication terminals 50-1, 50-1 (step S250). Upon receiving the synchronization signal, the communication terminal 50-1 displays the character objects ch1 and ch2 in the display area DA of the display unit 57 so that they move in accordance with the movement data set Md12. The time course of the movements of the character objects ch1 and ch2 is controlled by the synchronization signal.
[0063] FIG. 14 is a diagram for explaining a display example in the multiple mode in the first embodiment (when an image is captured along the first direction AR1). Character objects ch1 and ch2 are displayed in the display area DA of the display unit 57 in the communication terminal 50-1. The example shown in FIG. 14 shows a state in which a predetermined time has elapsed since the playback of the actions of the character objects ch1 and ch2 was started by displaying in the multiple mode. The elapsed time is indicated by a seek bar SK. Furthermore, a switch button SB2 for switching from the multiple mode to the single mode is displayed in the display area DA. The user can operate the seek bar SK to change the playback position (the playback portion in the action data) or operate the switch button SB2 to end the multiple mode.
[0064] Since the character objects ch1 and ch2 are moving according to the action data set Md12, the action from the period t2 to t3 in the animation data shown in FIG. 2 is reproduced. This reproduces a part of the animation work by AR display. Here, in the display area DA, the character objects ch1 and ch2 are displayed superimposed on the image of the real space RS included in the imaging range of the imaging unit 56. The coordinate system in the virtual space to which the action data set Md12 is applied is the same as in the single mode. That is, the position serving as the reference for the coordinates (the origin of the virtual space) is specified by the marker MR, and the X-axis, Y-axis, and Z-axis directions are specified by the information MA.
[0065] In this example, the background is an image of the real space RS, and the background BG in the animation data is not used. Note that the background BG that has been processed into a semi-transparent image may be further superimposed on the image of the real space RS.
[0066] Here, the display in the multiple modes may be realized only in the communication terminal 50-1, but the same display as that in the communication terminal 50-1 may also be realized in the communication terminal 50-2 of the user U2 that is the target of the combination. In this case, the actions of the character objects ch1 and ch2 may be synchronized by a synchronization signal. By using the synchronization signal, the actions of the character objects ch1 and ch2 can be displayed at the same elapsed time in the communication terminals 50-1 and 50-2.
[0067] FIG. 15 is a diagram for explaining a display example in a multiple mode in the first embodiment (when imaging along the second direction AR2). FIG. 14 shows the display content when the communication terminal 50-1 images the real space RS along the first direction AR1. On the other hand, FIG. 15 shows the display content when the communication terminal 50-2 images the real space RS along the second direction AR2. When viewed from the second direction AR2, the orientation of the marker MR is different. Since both are displayed using a common coordinate system, in the display example shown in FIG. 15, the X-axis direction, the Y-axis direction, and the Z-axis direction in the virtual space relative to the screen are different from those in the display example shown in FIG. 14. As a result, the appearance of the character objects ch1 and ch2 differs between the communication terminal 50-1 and the communication terminal 50-2. However, in this example, the passage of time is synchronized between the communication terminal 50-1 and the communication terminal 50-2 by the synchronization signal, so that the behavior of the character objects ch1 and ch2 is the same.
[0068] In the above example, the character objects ch1 and ch2 can be displayed on the display unit 57 while moving together depending on the combination of users U1 and U2. The number of combinations is not limited to two, and may be three or more. In the example of user management data shown in FIG. 6, the character objects ch1, ch2, and ch3 can be displayed while moving together depending on the combination of users U1, U2, and U3. Similarly, the character objects ch1, ch2, and ch3 can be displayed while moving together depending on the combination of users U3 and U4. Note that, since the user U4 is permitted to display both the character objects ch1 and ch2, the communication terminal 50 of the user U4 may display either one of them in a single mode, or both in a multiple mode.
[0069] In this way, even if each user has only a few character objects permitted to be displayed, multiple character objects can be displayed together by cooperating with other users. In this case, by combining multiple character objects, it is possible to display them on the display unit 57 not only by simply displaying them, but also by making them perform actions that could not be realized by a single character object. Therefore, by combining character objects with other character objects, a variety of expressions can be provided to the user, so that the user does not lose interest in the character objects.
[0070] [Configuration of the display control unit used in the service provision process] Next, the functional (display control unit) configuration for realizing the processing shown in FIG. 12 will be described with reference to FIG.
[0071] Fig. 16 is a block diagram showing the configuration of the display control unit in the first embodiment. Fig. 16 shows an example in which the server 10 and the communication terminal 50 cooperate to realize the display control unit 700. The action determination unit 600 is realized in the server 10. The action determination unit 600 has a function of performing the process of step S210 shown in Fig. 12. That is, the action determination unit 600 refers to the user management data to specify character objects permitted to be displayed for a plurality of users, and refers to the set correspondence data to specify an action data set corresponding to a combination of the character objects. In this way, the action determination unit 600 determines the actions of a plurality of character objects.
[0072] The display control unit 700 includes a data providing unit 710, a movement control unit 730, and a display superimposing unit 750. The data providing unit 710 reads out from the database 80 the movement data included in the movement data set and data related to a plurality of character objects to which the movement data is to be applied, and provides the data to the movement control unit 730. The movement control unit 730 applies the movement data to each character object to generate an image in which the character object is moved.
[0073] The display superimposition unit 750 superimposes the image generated by the operation control unit 730 on the image in the imaging range of the imaging unit 56, and displays the image on the display unit 57. At this time, the X-axis direction, the Y-axis direction, and the Z-axis direction in the virtual space may be corrected according to the attitude of the communication terminal 50 using behavior information from the behavior measurement unit 55.
[0074] In the first embodiment, of the display control unit 700, the function of the data providing unit 710 is realized in the server 10, and the functions of the action control unit 730 and the display superimposing unit 750 are realized in the communication terminal 50. The server 10 controls the action of the character object displayed on the communication terminal 50 by providing the communication terminal 50 with the function of the data providing unit 710, that is, an action data set and the like.
[0075] <Second embodiment> Of the display control unit 700, the functions of the data providing unit 710 and the operation control unit 730 may be realized in the server 10. In this case, the server 10 will transmit to the communication terminal 50 data of an image in which a character object is operated, rather than transmitting an operation data set or the like to the communication terminal 50.
[0076] <Third embodiment> Of the display control unit 700, all functions, that is, the functions of the data providing unit 710, the operation control unit 730, and the display superimposing unit 750 may be realized in the server 10. In this case, the server 10 acquires an image captured by the communication terminal 50 and uses the image in the display superimposing unit 750.
[0077] <Fourth embodiment> Of the display control unit 700, all functions, that is, the functions of the data providing unit 710, the operation control unit 730, and the display superimposing unit 750, may be realized in the communication terminal 50. In this case, the communication terminal 50 connects to the database 80 via the network NW to realize the functions of the data providing unit 710.
[0078] <Modification> Although one embodiment of the present invention has been described above, the above-described embodiments can be combined with or substituted for each other. Also, the above-described embodiments can be modified as follows.
[0079] (1) In the above-described embodiment, the character object, still data, and motion data are generated by extracting them from animation data, but the original animation data may not be necessary. In other words, each of the data may be generated separately for the purpose of implementing the data provision service.
[0080] (2) The process for allowing display of a character object is not limited to the method of the above embodiment. Display may be allowed by a lottery system known as gacha, or by a code distributed at an event or the like.
[0081] (3) In the above-described embodiment, the users to be combined can be selected based on the distance between the users, but they may be combined when a predetermined association condition is satisfied by another method. The users to be combined who are close to each other may be identified as users of the communication terminals 50 who are photographing an object determined to be the same by image recognition, or may be identified as users of the communication terminals 50 who can communicate by short-distance wireless communication. The above-described object may be, for example, the marker MR shown in the above-described embodiment, or any other object that is included in the imaging range and can be extracted by image analysis. That is, the predetermined association condition may be that the users have specific common information by image analysis of the imaging range. The users to be combined may be users who are not necessarily close to each other, and may be users who are associated as friends in an SNS, for example.
[0082] (4) In the above-described embodiment, the marker MR is used to identify the coordinate system in the virtual space in order to display the character object, but the present invention is not limited to using the marker MR. For example, three-dimensional information of the real space RS may be acquired and associated with the coordinate system in the virtual space by using a V-SLAM (Visual Simultaneous Localization and Mapping) technique. A part of the analyzed three-dimensional information may be recognized as a characteristic part, and the characteristic part may be replaced with the marker MR.
[0083] (5) In the above-described embodiment, the action data set is determined by the combination of the character objects, but the action data set may be determined using another element. As a result, even if the combination of the same character objects is performed, the action data set may be specified to be different. For example, an item object used by a specific character object is specified. As a result, the set correspondence data may be specified so that the action data set specified is different depending on whether or not an item object exists as a combination target. Specifically, the action data set applied in the multiple modes is different depending on whether or not a user permitted to display the character object ch1 is also permitted to display the item object used by the character object ch1 (the item object associated with the character object ch1).
[0084] (6) In the above embodiment, the timing of the still data corresponding to each character object is specified as a timing when the appearance period does not overlap with other character objects, but this is not limited to the above. That is, the timing of the still data corresponding to the character object may be specified as a timing when the appearance period overlaps with other character objects, or may be specified as a timing when the appearance period overlaps with the period used for the motion data set. For example, the still data Sd1 may be specified as the coordinates of each part at any timing during the period from t2 to t3 when the appearance period overlaps with the character object ch2.
[0085] As a result, the still data Sd1 is an extracted part of the motion form of the character object ch1 defined in the motion data set Md12. Therefore, even if there is no overall data such as animation data, the still data can be efficiently generated from the motion data, and the user can be motivated to see the motions before and after the character object ch1 displayed in the single mode. On the other hand, as in the above-mentioned embodiment, the still data Sd1 can be a form that is not included in the motion form of the character object ch1 defined in the motion data set Md12. In this case, the still data Sd1 needs to be generated as data of a motion form different from the motion data set Md12, but the user can enjoy a display in the multiple mode that is completely different from the character object ch1 in the single mode.
[0086] (7) In the above embodiment, still data that becomes a still image is applied to the character object displayed in the single mode, but action data that becomes a moving image may be applied. Even in this case, different action data may be applied to the character object in the single mode and the multiple mode.
[0087] (8) The above-described embodiment describes an example of applying the present invention to the movement control of a character object, but the present invention can be similarly applied to the movement control of not only character objects but also objects having three-dimensional shapes. [Explanation of symbols]
[0088] 10...server, 11...control unit, 12...storage unit, 13...communication unit, 50, 50-1, 50-2...communication terminal, 51...control unit, 52...storage unit, 53...communication unit, 54...position measurement unit, 55...behavior measurement unit, 56...imaging unit, 57...display unit, 58...operation unit, 80...database, 600...action determination unit, 700...display control unit, 710...data provision unit, 730...action control unit, 750...display superimposition unit, 1000...system
Claims
1. a determination unit that determines motion forms of the first character object and the second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; a display control unit that controls the first terminal so as to display the first character object and the second character object according to the action form so as to be superimposed on an image captured by the first terminal; An information processing device comprising:
2. The information processing device according to claim 1 , wherein the first terminal and the second terminal satisfy a predetermined association condition.
3. The information processing device according to claim 2 , wherein the association condition includes a condition that the first terminal and the second terminal are closer than a predetermined distance.
4. The information processing device according to claim 2 , wherein the association condition includes that the same object is included in the image captured by the first terminal and the image captured by the second terminal.
5. 5. An information processing device according to claim 2, wherein, when the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal is an image based on a part of the movement form, and the second character object that can be displayed on the second terminal is an image based on a part of the movement form.
6. 5. An information processing device according to claim 2, wherein, when the first terminal and the second terminal do not satisfy the specified association condition, the first character object that can be displayed on the first terminal is an image based on a form that is not included in the action form, and the second character object that can be displayed on the second terminal is an image based on a form that is not included in the action form.
7. 7. The information processing device according to claim 1, further comprising: controlling the second terminal so as to display the first character object and the second character object according to the action form superimposed on an image captured by the second terminal.
8. The information processing device according to claim 1 , wherein, when an item object that can be displayed together with the first character object exists on the first terminal, the determination unit further determines the motion form based on the item object.
9. 9. The information processing device according to claim 1, wherein the motion form corresponds to a motion form defined in data defining motion forms of a plurality of character objects including the first character object and the second character object, during at least a portion of a period during which the first character object and the second character object are used.
10. determining motion forms of the first character object and the second character object based on a first character object that can be displayed on a first terminal and a second character object that can be displayed on a second terminal; Controlling the first terminal so as to display the first character object and the second character object in accordance with the motion form superimposed on an image captured by the first terminal. A program for causing a computer to execute the following.
Citation Information
Patent Citations
Portable movement support device
JP2014120006A
Information processing device, system, information processing method and program
WO2014080678A1
Information processing device and information processing method
JP2015041126A
