System and method

Abstract

To provide a system for controlling a virtual character.SOLUTION: A system including at least one computer device includes: first input reception means that receives input of movement information related to movement of a virtual character; second input reception means that receives input of operation information related to an operation of the virtual character; and virtual character movement means that causing the virtual character to move in accordance with the received movement information and to perform an operation in accordance with the received operation information in a virtual three-dimensional space.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a system and a method.

Background Art

[0002] In recent years, there has been a trend of enjoying a virtual space called the metaverse using a virtual character as one's own avatar. In the metaverse space, a user can freely move the avatar.

[0003] Also, in the metaverse space, a user can also cause the avatar to perform a performance such as dancing. The movement data to be executed by the avatar can be obtained, for example, by the user actually performing a movement in the real world and digitizing the movement by motion capture technology.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide a system for controlling a virtual character.

Means for Solving the Problems

[0005] According to the present invention, the above object is achieved by [1] A system including at least one computer device, the system including: a first input receiving means for receiving an input of movement information regarding the movement of a virtual character; a second input receiving means for receiving an input of action information regarding the action of the virtual character; and a virtual character action means for causing the virtual character to perform a movement according to the received movement information and an action according to the received action information in a virtual three-dimensional space; [2] The system according to [1], further including a display means for controlling to display on a display screen a state in which the virtual character performs an action and a movement. [3] The system according to [1] or [2], wherein the first input receiving means receives an input of movement information obtained by an operation on a user terminal operated by a user; [4] The system according to any one of [1] to [3], wherein the second input receiving means receives an input of motion information obtained by imaging a person's motion; [5] The system according to any one of [1] to [4], wherein the movement information includes position coordinates in a virtual three-dimensional space associated with time, and the motion information includes information regarding the motion of a virtual character viewed from the horizontal direction; [6] A method executed in a system including at least one computer device, the method including: a first input receiving step of receiving an input of movement information regarding the movement of a virtual character; a second input receiving step of receiving an input of motion information regarding the motion of the virtual character; and a virtual character motion step of causing the virtual character to execute a movement corresponding to the received movement information and execute a motion corresponding to the received motion information in a virtual three-dimensional space; [7] A system including a server device and a first user terminal connectable to the server device, and optionally including a second user terminal connectable to the server device, wherein the first user terminal includes: first input receiving means for receiving an input of movement information regarding the movement of a virtual character; and second input receiving means for receiving an input of motion information regarding the motion of the virtual character; the server device includes: virtual character motion means for causing the virtual character to execute a movement corresponding to the movement information and execute a motion corresponding to the motion information in a virtual three-dimensional space; generation means for generating rendering information for perspective-transforming and displaying the motion and movement of the virtual character on a display screen; and transmission means for transmitting the generated rendering information to the first user terminal or the second user terminal; and by transmitting the generated rendering information to the first user terminal or the second user terminal, the first user terminal or the second user terminal displays, on the display screen, a state in which the virtual character executes a movement corresponding to the movement information and executes a motion corresponding to the motion information in a virtual three-dimensional space based on the rendering information; [8] A method executed in a system including a server device and a first user terminal connectable to the server device, and optionally including a second user terminal connectable to the server device, the method comprising: a first input receiving step in which the first user terminal receives an input of movement information regarding the movement of a virtual character; and a second input receiving step in which the first user terminal receives an input of action information regarding the action of the virtual character; a virtual character action step in which the server device causes the virtual character to perform a movement according to the movement information and an action according to the action information in a virtual three-dimensional space; a generation step in which the server device generates drawing information for displaying, on a display screen, the actions and movements of the virtual character through perspective transformation; and a transmission step in which the server device transmits the generated drawing information to the first user terminal or the second user terminal, wherein, by transmitting the generated drawing information to the first user terminal or the second user terminal, the first user terminal or the second user terminal displays, on the display screen, a state in which the virtual character performs a movement according to the movement information and an action according to the action information in the virtual three-dimensional space based on the drawing information; can be achieved by.

Effect of the Invention

[0006] According to the present invention, a system for controlling a virtual character can be provided.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described. However, the present invention is not limited to the following embodiments as long as it does not contravene the gist of the present invention. The order of each process constituting the flowchart described below may be in any order as long as there is no contradiction or inconsistency in the process content. Also, as long as there is no contradiction or inconsistency in the process content, a part of each process constituting the flowchart may be omitted, or a new process may be added to each process constituting the flowchart. Further, the device that is the subject of executing each process constituting the flowchart can be changed to another device as long as it does not contravene the gist of the present invention. In that case, the process content can be changed so that there is no contradiction or inconsistency in the process content.

[0009] FIG. 1 is a block diagram showing the configuration of a system according to an embodiment of the present invention. In FIG. 1, system 1 includes a first user terminal 2, a second user terminal 3, and a server device 4. The first user terminal 2, the second user terminal 3, and the server device 4 may be communicatively connected to each other via a communication network 5. In system 1, any one of the first user terminal 2, the second user terminal 3, and the server device 4 may function as an information processing device. When any one of the first user terminal 2, the second user terminal 3, and the server device 4 functions as an information processing device, information transmission and reception are executed between the first user terminal 2, the second user terminal 3, and the server device 4 as necessary. Note that the number of user terminals may be three or more. Also, the server device 4 may be distributed and function among a plurality of computer devices. For example, instead of the server device 4, a distributed ledger technology such as a blockchain may be used. Alternatively, system 1 may include only the first user terminal 2 and may be configured without the second user terminal 3 and the server device 4. That is, system 1 only needs to include at least one computer device.

[0010] The first user terminal 2 may be operated by a user (hereinafter referred to as the first user) who causes a virtual character to perform in the metaverse space. By using system 1, the first user can create a performance to be executed by the virtual character. Also, by using system 1, the first user can cause the created performance to be executed by the virtual character. Hereinafter, the virtual character operated by the first user in the metaverse space is referred to as the first avatar. Here, the first user causes the first avatar to perform.

[0011] The second user terminal 3 may be operated by a user (hereinafter referred to as the second user) who appreciates the performance of the first avatar in the metaverse space. By using the system 1, the second user can appreciate the performance of the first avatar through a virtual character that becomes their own avatar. Hereinafter, the virtual character operated by the second user in the metaverse space is referred to as the second avatar.

[0012] The first user terminal 2 and the second user terminal 3 may have a dedicated application (hereinafter also referred to as the dedicated app) for using the system 1 installed thereon. By launching the dedicated app, the first user and the second user can display the metaverse space on the first user terminal 2 and the second user terminal 3. The first user can operate the first avatar within the metaverse space by inputting operations to the first user terminal 2. Also, the second user can operate the second avatar within the metaverse space by inputting operations to the second user terminal 3.

[0013] The server device 3 may provide the metaverse space to the user. The metaverse space may be a virtual three-dimensional space. Also, the server device 3 may control the production and execution of performances by virtual characters in the metaverse space.

[0014] FIG. 2 is a block diagram showing the hardware configuration of the user terminal according to the embodiment of the present invention. The first user terminal 2 includes a control unit 21, a RAM 22, a storage unit 23, an input unit 24, a display unit 25, and a communication interface 26, which are respectively connected by a bus.

[0015] The control unit 21 is composed of a CPU and a ROM. The control unit 21 executes a program stored in the storage unit 23 to control the first user terminal 2. The RAM 22 is a work area for the control unit 21. The storage unit 23 is a storage area for storing programs and data. That is, the storage unit 23 functions as a recording medium storing a program. The control unit 21 performs arithmetic processing based on the program and data read from the RAM 22, and the data input at the input unit 24.

[0016] The display unit 25 has a display screen. The control unit 21 outputs a video signal for displaying an image on the display screen according to the result of the arithmetic processing. Here, the display screen of the display unit 25 may be a touch panel provided with a touch sensor. In this case, the touch panel functions as the input unit 24.

[0017] The communication interface 26 can be connected to the communication network 5 wirelessly or by wire, and can transmit and receive data with other computer devices via the communication network 5. The data received via the communication interface 26 is loaded into the RAM 22, and arithmetic processing is performed by the control unit 21.

[0018] Also, although not shown, the first user terminal 2 preferably includes an imaging unit such as a camera. The imaging unit may be connected to other functional units by a bus.

[0019] Regarding the hardware configuration of the second user terminal 3, the description of the hardware configuration of the first user terminal 2 can be adopted within the necessary range.

[0020] Figure 3 is a block diagram showing the hardware configuration of the server device according to the embodiment of the present invention. The server device 4 includes at least a control unit 41, a RAM 42, a storage unit 43, and a communication interface 44, which are respectively connected by an internal bus.

[0021] The control unit 41 is composed of a CPU and a ROM, executes the program stored in the storage unit 43, and controls the server device 4. Further, the control unit 41 is provided with an internal timer for measuring time. The RAM 42 is a work area of the control unit 41. The storage unit 43 is a storage area for storing programs and data. That is, the storage unit 43 functions as a recording medium storing the program. The control unit 41 reads the program and data from the RAM 42, and performs program execution processing based on information received from the first user terminal 2 and / or the second user terminal 3, etc.

[0022] Further, the program may be stored in a recording medium such as a CD-ROM. In this case, the program stored in the recording medium may be installed in the first user terminal 2, the second user terminal 3, and / or the server device 4 to execute a predetermined function.

[0023] Alternatively, the program may be distributed from a computer device external to the system. In this case, the program distributed from the computer device external to the system may be installed in the first user terminal 2, the second user terminal 3, and / or the server device 4 to execute a predetermined function.

[0024] Hereinafter, an example will be described in which, on a dedicated application, the first user creates a performance to be executed by the first avatar and causes the first avatar to execute the performance. Further, an example will be described in which the second user and the first user appreciate the performance.

[0025] Users who use the dedicated app can launch the dedicated app on their own operating terminals and log in to System 1. When a user logs in to System 1 for the first time, the user may also perform user registration. In user registration, System 1 may store information about the user, such as the user's name and email address. By performing user registration, an account may be assigned to the user from System 1. In System 1, information about the user may be stored in association with the account assigned to the user. When the user logs in to System 1, inputs such as an email address and password may be required. The password may be set by the user.

[0026] By logging in to System 1, the user can operate their avatar in the metaverse space. The user may be able to change the appearance of their avatar, such as gender, hairstyle, facial features, and clothing.

[0027] In the metaverse space, objects such as virtual characters (the user's avatar) that can be operated by the user, virtual characters controlled by System 1, backgrounds (sky, sea, ground, etc.), buildings, and vehicles may be drawn. The objects drawn in the metaverse space are projected onto the display screen of the user terminal by performing a perspective transformation of the metaverse space using a virtual camera. In the metaverse space, one virtual camera may be assigned to one user (account).

[0028] In the metaverse space, a virtual camera is placed, and the image projected on the virtual screen becomes the image displayed on the display screen. The position of this virtual camera is the viewpoint, the direction of the virtual camera is the visual axis, and the area formed by the four straight lines connecting the viewpoint and the four corners of the vertices of the virtual screen is the field of view. The user may be able to change the position of the virtual camera, that is, the user's viewpoint, by operating and inputting to the user terminal. For example, the user may place the virtual camera at a position where the back of their own avatar is reflected, or at a position corresponding to the viewpoint (head) of their own avatar.

[0029] The method of operating and moving an avatar in the metaverse space is not particularly limited, and any method can be adopted. For example, when the avatar is composed of a virtual skeleton and polygons associated with the virtual skeleton, the avatar may be moved by moving the virtual skeleton. The polygons constituting the avatar are drawn so as to move in accordance with the movement of the virtual skeleton. The degree to which the virtual skeleton is moved may be specified based on the operation input to the user terminal. The position of the virtual skeleton and the position of the polygons may be represented by the world coordinate system or the local coordinate system.

[0030] [Performance production process] The first user may be able to start the production of a performance by pressing a button or the like that requests the start of the production of the performance, which is displayed on the display screen of the first user terminal 2. Note that "performance" refers to the acting performed by a virtual character. "Performance" may include dance, song, drama, etc. "Performance" preferably includes the movement of the virtual character and the actions of the virtual character.

[0031] Figure 4 is a flowchart of the performance production process according to the embodiment of the present invention.

[0032] First, in the first user terminal 2, input of movement information regarding the movement of the virtual character is accepted (step S101). The input movement information is transmitted from the first user terminal 2 to the server device 4 (step S102). In the server device 4, the transmitted movement information is received (step S103).

[0033] The movement information in step S101 may be any information regarding the movement of the virtual character and is not particularly limited. The movement information may be information regarding a change in the position of the virtual character. The change in position may be associated with time. The position of the virtual character may be represented by position coordinates in a virtual three-dimensional space. For example, the movement information may include position coordinates in a virtual three-dimensional space associated with time.

[0034] Also, the movement information may be obtained by an operation on the first user terminal 2. For example, the movement information may be obtained by inputting position coordinates in a virtual three-dimensional space in the first user terminal 2. The movement information is preferably input without using motion capture technology.

[0035] FIG. 5 is a diagram showing an example of an execution screen according to an embodiment of the present invention. In step S101, a movement information input screen as shown in FIG. 5 may be displayed on the display screen of the first user terminal 2 as the execution screen 100.

[0036] On the movement information input screen shown in FIG. 5, a top view 101 of the stage, grid lines 102, icons 103 (103a to 103c) indicating the positions where the virtual character moves, information 104 (104a to 104c) regarding the time when the virtual character moves, trajectories 105 (105a and 105b) of the movement of the virtual character, a play button 106, a seek bar 107, a stage change button 108, and an OK button 109 are displayed.

[0037] The top view 101 of the stage may be the top view of the stage on which the virtual character performs. Note that the top surface of the stage refers to the surface on the stage where the performer steps onto the stage. The top surface of the stage may be a plane parallel to the ground. In FIG. 5, a rectangular top view is displayed.

[0038] The first user may be able to change the stage on which the virtual character performs by pressing the stage change button 108. The shape of the top view of the stage is not particularly limited and can be designed as appropriate. The shape of the top view of the stage may be, for example, rectangular, square, circular, elliptical, or star-shaped.

[0039] In FIG. 5, the top view 101 of the stage is provided with grid lines 102 in a substantially square grid pattern vertically and horizontally. Note that in FIG. 5, the reference signs of the grid lines 102 are omitted except for some parts. By providing the grid lines 102 on the top view 101 of the stage, it becomes easy to grasp the positions on the top view 101 of the stage. The shape of the grid lines 102 is not particularly limited and can be designed as appropriate. The shape of the grid lines 102 may be, for example, a square grid pattern, a rectangular grid pattern, a regular triangular honeycomb pattern, a regular hexagonal honeycomb pattern, a horizontal grid line pattern, or a vertical grid line pattern. Alternatively, the grid lines 102 may not be provided on the top view 101 of the stage.

[0040] The first user may be able to input the position where the virtual character moves by selecting an arbitrary location on the top view 101 of the stage, such as by tapping. The position coordinates of the selected location may be input as movement information. The position coordinates may be represented by a world coordinate system or a local coordinate system.

[0041] At the location selected by the first user, an icon 103 with a circle mark as shown in FIG. 5 may be displayed. The shape of the icon is not particularly limited and can be designed as appropriate. In FIG. 5, the number 1 is attached to the icon 103a, the number 2 is attached to the icon 103b, and the number 3 is attached to the icon 103c. The numbers attached to the icon 103 may represent the order in which the virtual character moves. The order in which the virtual character moves may be input by the first user.

[0042] In FIG. 5, between the icon 103a indicating the position where the virtual character moves first and the icon 103b indicating the position where it moves second, they are connected by a movement trajectory 105a. Also, between the icon 103b indicating the position where the virtual character moves second and the icon 103c indicating the position where it moves third, they are connected by a movement trajectory 105b. In FIG. 5, the movement trajectory 105 of the virtual character is a straight line. That is, the virtual character will move straight between each icon 103. The movement trajectory 105 of the virtual character is not particularly limited and can be designed as appropriate. The movement trajectory 105 of the virtual character may, for example, be a straight line, draw an arc, or meander. The movement trajectory 105 of the virtual character may be inputtable by the first user.

[0043] The first user may be able to input the time for the virtual character to move in association with the position where the virtual character moves. For example, the first user may be able to input the time for the virtual character to stay at the selected position and the time for the virtual character to move between the selected positions. For example, the first user may be able to input the time for the virtual character to stay at the position indicated by each icon 103 by selecting the information 104 regarding the time displayed in the vicinity of each icon 103, such as by tapping. By inputting the time for the virtual character to stay at the position indicated by each icon 103, the time for the virtual character to move between each icon 103 is specified. Also, by inputting the time for the virtual character to stay at the position indicated by each icon 103, the order in which the virtual character moves may be specified.

[0044] In FIG. 5, the information 104a regarding time is displayed as "00:00 → 00:02", the information 104b regarding time is displayed as "00:04 → 00:06", and the information 104c regarding time is displayed as "00:07 → 00:10". Also, the information 104a to 104c regarding time is displayed in the vicinity of the icons 103a to 103c, respectively. Therefore, it is possible to grasp that the time for the virtual character to stay at the position indicated by the icon 103a is 2 seconds from the start of the performance, and then it moves to the position indicated by the icon 103b over 2 seconds. Also, it is possible to grasp that the time for the virtual character to stay at the position indicated by the icon 103b is 2 seconds, and then it moves to the position indicated by the icon 103c over 1 second. Furthermore, it is possible to grasp that the time for the virtual character to stay at the position indicated by the icon 103c is 3 seconds.

[0045] The first user may be able to preview the movement of the virtual character on the stage according to the input movement information by pressing the play button 106. For example, when the play button 106 is pressed, the virtual character may be displayed on the top view 101 of the stage. Also, according to the time indicated in the time-related information 104, the movement of the virtual character walking from above the icon 103a, via the icon 103b, to above the icon 103c may be displayed.

[0046] While previewing, the first user may be able to grasp the playback status of the preview. For example, the round slider on the seek bar 107 may be displayed as moving from the left end to the right end of the seek bar 107 along with the playback of the preview. Also, for example, information regarding the playback time of the preview may be displayed near the seek bar 107. In FIG. 5, the start time of the preview, "00:00", is displayed at the lower part of the left end of the seek bar 107, and the end time of the preview, "00:10", is displayed at the lower part of the right end of the seek bar 107. That is, the total playback time of the preview is 10 seconds. While the preview is being played back, the playback time of the preview (the time elapsed since the start of the preview) may be displayed at the lower part of the round slider on the seek bar 107.

[0047] If there is no problem with the movement of the virtual character, the first user can confirm the input movement information by pressing the OK button 109. When the first user presses the OK button 109, the movement information may be transmitted to the server device 4 in step S102.

[0048] In the above description, as an example of the movement information, the position coordinates in the top view of the stage, that is, the coordinates on the xy plane in the virtual three-dimensional space are input. However, as the movement information, it may be possible to input the coordinates on the z plane in the virtual three-dimensional space. By including the position coordinates on the z plane in the virtual three-dimensional space in the movement information, for example, effects such as the virtual character flying can be incorporated into the performance.

[0049] Also, in the above description, an example is given in which the first user inputs the position and time at which the virtual character moves. However, the mode of inputting the movement information is not particularly limited and can be designed as appropriate. For example, the first user may be able to select arbitrary movement information from among the templates of the movement information in which the position and / or time at which the virtual character moves are determined in advance.

[0050] Returning to the description of FIG. 4. After the movement information is transmitted to the server device 4, at the first user terminal 2, input of operation information regarding the operation of the virtual character is accepted (step S104). The input operation information is transmitted from the first user terminal 2 to the server device 4 (step S105). At the server device 4, the transmitted operation information is received (step S106).

[0051] The operation information in step S104 may be any information regarding the operation of the virtual character and is not particularly limited. The information regarding the operation of the virtual character may be information regarding the movement of the virtual skeleton of the virtual character. The information regarding the movement of the virtual skeleton may include information regarding the position coordinates of the virtual skeleton. The information regarding the movement of the virtual character may be associated with time.

[0052] Also, the motion information may be obtained by imaging the motion of the first user. The motion of the first user may be the motion of the entire body of the first user or the motion of a part of the body of the first user. The part of the body may be the upper body, an arm, the head, the lower body, or a leg. The motion information may include information regarding the motion of the virtual character viewed from the horizontal direction.

[0053] FIG. 6 is a diagram showing an example of an execution screen according to an embodiment of the present invention. In step S104, on the display screen of the first user terminal 2, as the execution screen 200, a motion information input screen as shown in FIG. 6 may be displayed.

[0054] On the motion information input screen shown in FIG. 6, an image 201 of the real world, an image 202 of the virtual three-dimensional space, a first user 203, a virtual character 204, a shooting start / end button 205, and a shooting time 206 are displayed.

[0055] The image 201 of the real world may be an image of the real world reflected by the camera of the first user terminal 2. The image 201 of the real world is preferably an image of the posture of the first user reflected from the horizontal direction.

[0056] The motion of the first user 203 displayed in the image 201 of the real world may be reflected in real time on the virtual character 204 displayed in the image 202 of the virtual three-dimensional space using motion capture technology. For example, by extracting the silhouette of the first user 203 displayed in the image 201 of the real world to create a skeleton model and obtaining data such as joint angles in the motion of the first user 203, the motion of the first user 203 can be reflected on the virtual character 204.

[0057] Together with the first user 203, by displaying the virtual character 204 that reflects the actions of the first user 203, the first user can input action information while grasping how the virtual character would execute the actions of the first user. The first user 203 and the virtual character 204 may be displayed side by side vertically or horizontally on the display screen of the first user terminal 2.

[0058] By pressing the shooting start / end button 205 displayed on the action information input screen, the shooting of the actions of the first user in the real world may be started or ended. The data capturing the actions of the first user may be input as action information in step S104. The data capturing the actions of the first user may be associated with time (shooting time).

[0059] During shooting, the time elapsed since the start of shooting may be displayed as the shooting time 206. Also, after shooting is completed, the total time of the captured moving images may be displayed as the shooting time 206.

[0060] If the shooting start / end button 205 is pressed during shooting and shooting is ended, the action information may be transmitted to the server device 4 in step S105.

[0061] In the above, an example of inputting action information by imaging the actions of the first user has been given. However, the mode of inputting action information is not particularly limited and can be designed as appropriate. For example, the action information may be obtained by digitizing the actual actions of the first user in the real world using any motion capture technology. Motion capture technology may include, for example, video-based motion capture, optical motion capture, inertial sensor-based motion capture, mechanical motion capture, magnetic motion capture, etc.

[0062] In addition to movement information and action information, the first user may be able to input performance information regarding the performance to be executed by the virtual character. The performance information may include, for example, expression information regarding the expression of the virtual character, voice information regarding the voice of the virtual character, music information regarding the music to be played during the performance, production information regarding the production on the stage during the performance, effect information regarding the effects to be generated on the virtual character during the performance, and the like.

[0063] The mode of inputting the performance information is not particularly limited and can be designed as appropriate. The expression information may be input, for example, by imaging the face of the first user in the real world with a camera and capturing the expression of the first user. The voice information may be input, for example, by acquiring the voice of the first user in the real world with a microphone, or electronic voice data that reads out arbitrary character information may be input. The music information may be input, for example, by inputting part or all of the music data stored in the first user terminal 2 or the server device 4. The production information may be input, for example, by the first user selecting a predetermined production. The effect information may be input, for example, by the first user selecting a predetermined effect. The types of selectable productions and the types of effects may be stored in the system 1 in advance.

[0064] Each performance information may be managed as a track constituting the performance in the system 1. By processing a plurality of pieces of information input as tracks in an overlapping manner, the performance can be executed.

[0065] Return to the description of FIG. 4. After the first user inputs performance information including movement information and action information, the virtual character can be made to perform. At the first user terminal 2, an input of a request to make the virtual character perform (hereinafter also referred to as an execution request) is accepted (step S107). The input execution request is transmitted from the first user terminal 2 to the server device 4 (step S108). At the server device 4, the transmitted execution request is received (step S109). The server device 4 makes the virtual character execute movement according to the movement information and execute an action according to the action information (step S110).

[0066] Next, the server device 4 generates drawing information for perspective-transforming the actions and movements of the virtual character and displaying them on the display screen (step S111). The generated drawing information is transmitted from the server device 4 to the first user terminal 2 (step S112). At the first user terminal 2, the transmitted drawing information is received (step S113). At the first user terminal 2, based on the received drawing information, a state in which the virtual character executes actions and movements is displayed (step S114).

[0067] At the first user terminal 2, an input of a request to store the movement information and action information (hereinafter also referred to as a storage request) is accepted (step S115). The input storage request is transmitted from the first user terminal 2 to the server device 4 (step S116). At the server device 4, the transmitted storage request is received (step S117). At the server device 4, the movement information and action information are stored (step S118), and the performance production process ends.

[0068] FIG. 7 is a diagram showing an example of an execution screen according to an embodiment of the present invention. In step S107, on the display screen of the first user terminal 2, as an execution screen 300, a performance confirmation screen as shown in FIG. 7 may be displayed.

[0069] On the performance confirmation screen shown in FIG. 7, performance information 301, track input status 302 (302a to 302g), performance start time 303, performance end time 304, performance display column 305, play button 306, seek bar 307, play time 308, performance time 309, and save button 310 are displayed.

[0070] The performance information 301 may be information regarding the performance to be executed by the virtual character. In FIG. 7, as the performance information 301, the track input status 302, performance start time 303, and performance end time 304 are displayed.

[0071] The track input status 302 may indicate the input status of each performance information as a track. In FIG. 7, as the track input status 302, the input status 302a of movement information, the input status 302b of action information, the input status 302c of expression information, the input status 302d of voice information, the input status 302e of music information, the input status 302f of production information, and the input status 302g of effect information are displayed.

[0072] In FIG. 7, the track input status 302 is shown in the shape of a bar graph. The part where the bar graph is shown by a solid line indicates that the performance information has been input, and the part where the bar graph is shown by a broken line indicates that the performance information has not been input.

[0073] Also, the position within the bar graph may correspond to the time of the performance. For example, the left end of the bar graph may indicate the input status at the performance start time, the central part of the bar graph may indicate the input status during the performance, and the right end of the bar graph may indicate the input status at the performance end time. Note that the longer the elapsed time from the start of the performance, the more the input status is shown on the right side of the bar graph.

[0074] For example, since the bar graph of the input status 302a of movement information is displayed in a solid line, it can be understood that movement information is being input throughout the performance time. Also, since the number 1 is displayed at the left end of the bar graph, the number 2 is displayed in the center of the bar graph, and the number 3 is displayed at the right end of the bar graph, it can be understood that the first position is input for some time from the start time of the performance, the second position is input around the middle of the performance, and the third position is input at the end time of the performance. Note that the first position, the second position, and the third position correspond to the first position, the second position, and the third position shown in FIG. 5.

[0075] Also, in FIG. 7, since the bar graphs of the input status 302b of operation information, the input status 302c of expression information, and the input status 302e of music information are displayed in solid lines, it can be understood that operation information, expression information, and music information are being input throughout the performance time. Also, in FIG. 7, since the bar graph of the input status 302d of voice information is displayed in a dashed line, it can be understood that voice information is not being input throughout the performance time.

[0076] Also, in FIG. 7, in the bar graph of the input status 302f of production information, a part of the left end and the right end is displayed in a solid line, and the central part is displayed in a dashed line. Therefore, it can be understood that production information is input in the parts corresponding to several seconds after the start of the performance and several seconds before the end, and production information is not input in the part corresponding to several seconds in the middle of the performance. Also, since the character "FIRE" is displayed in the solid line part at the left end of the bar graph and the character "FLOWER" is displayed in the solid line part at the right end of the bar graph, it can be understood that a flame production is drawn on the stage for several seconds after the start of the performance, and a flower flurry production is drawn on the stage for several seconds before the end of the performance.

[0077] Also, in FIG. 7, in the bar graph of the input status 302g of the effect information, a part of the left end and the right end are displayed as broken lines, and the central part is displayed as a solid line. Therefore, it can be grasped that no effect information is input in the parts corresponding to several seconds after the start of the performance and several seconds before the end, and effect information is input in the part corresponding to several seconds in the middle of the performance. Further, since the character "SPARKLING" is displayed on the solid line part in the center of the bar graph, it can be grasped that a virtual character is drawn with a glowing effect for several seconds in the middle of the performance.

[0078] The first user may be able to confirm, on the performance confirmation screen, the performance drawn according to the input performance information. For example, when the playback button 306 is pressed by the first user, an execution request may be transmitted in step S108. Then, in step S110, the performance is executed in the virtual three-dimensional space according to the performance information 301. That is, in step S110, the virtual character is made to execute a movement according to the movement information and an action according to the action information.

[0079] Note that when the movement information does not include the movement of the legs of the virtual character at the timing when the position of the virtual character changes based on the movement information, the server device 4 may cause the virtual character to execute a walking action.

[0080] Here, the movement of the virtual character according to the movement information may be regarded as a movement, and the movement of the virtual character according to the action information may be regarded as an action. For example, when the action information includes information regarding a movement that changes the position of the virtual character, the position of the virtual character changes according to the action information, and the change in position may also be regarded as an action.

[0081] "Performance is executed according to performance information 301" may include causing a virtual character to execute movement according to movement information, causing a virtual character to execute an action according to action information, causing a virtual character to execute an expression according to expression information, playing the voice of a virtual character according to voice information, playing music according to music information, rendering an on-stage effect according to effect information, and rendering an effect of a virtual character according to effect information. When performing a performance, it is preferable that each performance information is processed so that the times included in the respective performance information are synchronized.

[0082] When the performance is executed, in step S111, drawing information for perspective-transforming the performance and displaying it on the display screen is generated. At this time, the position of the virtual camera for perspective-transforming the performance is not particularly limited and can be designed as appropriate. For example, the position of the virtual camera may be a position where the entire stage can be viewed, or a position where a part of the stage can be viewed. Also, the position of the virtual camera may be a position where the stage can be viewed from the front, a position where the stage can be viewed from the back, or a position where the stage can be viewed from the side.

[0083] In step S114, based on the generated drawing information, the state in which the performance is executed, that is, the state in which the virtual character executes actions and movements, is displayed. The state in which the performance is executed may be displayed, for example, in the performance display column 305. In FIG. 7, the entire stage where the performance is being executed is displayed in the performance display column 305 as viewed from the front.

[0084] While the performance is being played, the first user may be able to grasp the playback status of the performance. For example, the circular slider on the seek bar 307 may be displayed as moving from the left end to the right end of the seek bar 307 as the performance is played. Also, for example, information regarding the playback time of the performance may be displayed near the seek bar 307. In FIG. 7, information "00:10 / 00:10" is displayed below the left end of the seek bar 307. Here, the information displayed before the slash indicates the playback time 308 (the time elapsed since the start of the performance playback), and it is assumed that the information displayed after the slash indicates the performance time 309 (the total time of the performance).

[0085] When there is no problem with the way the performance is executed, the first user can store the input performance information by pressing the save button 310. When the first user presses the save button 310, in step S115, an input of a storage request may be received, and in step S116, the storage request may be transmitted to the server device 4.

[0086] In step S118, the performance information may be stored in the storage unit 43 of the server device 4. The movement information, action information, expression information, voice information, music information, production information, effect information, etc. included in the performance information may be stored in association with each other as information regarding one performance. The performance information may be stored in association with a number or name for identifying the performance.

[0087] In the above description, an example where the virtual character that performs the performance is the first avatar has been given. However, the virtual character that performs the performance may be a virtual character different from the first avatar. Also, when the virtual character that performs the performance is the first avatar, the appearance of the first avatar while performing the performance may be the same as the normal appearance of the first avatar, or may be different from the normal appearance of the first avatar. The appearance of the virtual character that performs the performance may be set by the operation of the first user.

[0088] [Performance Execution Processing] The performance produced in this way can be presented in the metaverse space. Hereinafter, in the metaverse space, a mode in which the first user causes a virtual character to perform a performance and the second user and the first user appreciate the performance will be described.

[0089] FIG. 8 is a flowchart of the performance execution process according to an embodiment of the present invention.

[0090] First, at the first user terminal 2, an input of a performance execution request is received (step S201). The input execution request is transmitted from the first user terminal 2 to the server device 4 (step S202). At the server device 4, the transmitted execution request is received (step S203). The server device 4 causes the virtual character to perform a movement according to the movement information and perform an action according to the action information (step S204).

[0091] The server device 4 generates second drawing information for displaying the actions and movements of the virtual character on the display screen of the second user terminal 3 through perspective transformation (step S205). The generated second drawing information is transmitted from the server device 4 to the second user terminal 3 (step S206). At the second user terminal 3, the transmitted second drawing information is received (step S207). At the second user terminal 3, based on the received second drawing information, the state of the virtual character performing actions and movements is displayed (step S208), and the performance execution process ends.

[0092] Also, after step S204, separately from steps S205 to S208, the server device 4 generates first drawing information for displaying the actions and movements of the virtual character on the display screen of the first user terminal 2 through perspective transformation (step S209). The generated first drawing information is transmitted from the server device 4 to the first user terminal 2 (step S210). At the first user terminal 2, the transmitted first drawing information is received (step S211). At the first user terminal 2, based on the received first drawing information, the state of the virtual character performing actions and movements is displayed (step S212), and the performance execution process ends.

[0093] The first user may input a performance execution request in step S201 by pressing a button or the like that requests the start of performance execution and is displayed on the display screen of the first user terminal 2. Also, in step S201, information for specifying the performance to be executed may be input together with the execution request. For example, the first user may select a performance to be executed by the virtual character and input an execution request for that performance.

[0094] Note that the execution request may be input by an operation of a user other than the first user. Alternatively, when the performance information includes the performance execution start time, for example, the performance may be executed when a predetermined time is reached even if the execution request is not input by the user.

[0095] Regarding the execution of the performance in step S204, the description of the execution of the performance in the performance production process can be adopted within the necessary scope. In step S204, it is only necessary to cause the virtual character to execute a movement according to the movement information and an action according to the action information.

[0096] In step S205, the position of the virtual camera that performs perspective transformation on the performance may be the position set as the viewpoint of the second user. The second user may arrange the virtual camera at a position where the back of his or her avatar is reflected, or may arrange it at a position corresponding to the viewpoint (head) of his or her avatar. The position of the virtual camera assigned to the second user may be changeable by the operation of the second user.

[0097] In step S208, the second user terminal 3 displays, based on the second drawing information, how the virtual character executes a movement according to the movement information and an action according to the action information.

[0098] FIG. 9 is a diagram showing an example of an execution screen according to an embodiment of the present invention. In step S208, a performance viewing screen as shown in FIG. 9 may be displayed as the execution screen 400 on the display screen of the second user terminal 3.

[0099] On the performance viewing screen shown in FIG. 9, a first avatar 401, a stage 402, and a second avatar 403 are displayed.

[0100] The first avatar 401 may be the avatar of the first user who is performing the performance. Also, the stage 402 may be the stage selected by the first user as the stage for performing the performance. The second avatar 403 may be the avatar of the second user who is viewing the performance.

[0101] In the performance appreciation screen shown in FIG. 9, the virtual camera is placed at a position where the back of the second avatar 403 is reflected. The second user can appreciate the performance by the first avatar from the perspective of the second user capturing the metaverse space.

[0102] In step S209, the position of the virtual camera that performs perspective transformation on the performance may be the position set as the perspective of the first user, or the position set for appreciating the performance from the outside. The position set as the perspective of the first user may be, for example, a position where the back of the first avatar is reflected, or a position corresponding to the perspective (head) of the first avatar. The position set for appreciating the performance from the outside may be, for example, a position where the entire stage can be viewed, or a position where a part of the stage can be viewed. Also, the position set for appreciating the performance from the outside may be a position where the stage can be viewed from the front, a position where the stage can be viewed from the back, or a position where the stage can be viewed from the side. The first user may be able to change the position of the virtual camera while the performance is being executed.

[0103] In step S212, the first user terminal 2 displays, based on the first drawing information, how the virtual character executes movements according to the movement information and executes actions according to the action information. The first user can appreciate the performance by the first avatar from the perspective of the first user capturing the metaverse space or from the perspective for appreciating the performance from the outside.

[0104] Note that in the above, an example where both the second user and the first user appreciate the performance has been given, but either the second user or the first user may appreciate the performance, or a user other than the second user and the first user may appreciate the performance.

[0105] When both the movement information and the action information to be executed by the virtual character are input by motion capture, in order for the user to actually perform, a space with the same area as the stage is required. Also, in order to photograph how the user actually performs, a plurality of cameras are required. On the other hand, as described above, when the movement information is input by position coordinates and the action information is input by motion capture, all that is required is a user terminal with one camera and a space within the range that can be photographed by the camera. Therefore, it becomes easier for the user to produce a performance.

[0106] As described above, a system including at least one computer device can provide a system for controlling a virtual character by including a first input reception unit that receives an input of movement information related to the movement of the virtual character, a second input reception unit that receives an input of action information related to the action of the virtual character, and a virtual character action unit that causes the virtual character to execute a movement corresponding to the received movement information and an action corresponding to the received action information in a virtual three-dimensional space.

[0107] Also, as described above, by providing a display unit that controls the system to display on the display screen how the virtual character executes actions and movements, it is possible to confirm how the virtual character executes actions and movements.

[0108] Also, as described above, by the first input reception unit receiving an input of movement information obtained by an operation on the user terminal operated by the user, the movement information can be input by an operation on the user terminal.

[0109] Also, as described above, by the second input reception unit receiving an input of action information obtained by imaging a person's action, the action actually performed by the person can be executed by the virtual character.

[0110] Also, in this way, since the movement information includes the position coordinates in the virtual 3D space associated with time and the action information includes the information regarding the actions of the virtual character viewed from the horizontal direction, the virtual character can be made to move based on the position coordinates. Also, the actions to be performed by the virtual character become easier to understand.

[0111] Also, in a system that may include a server device and a first user terminal capable of connecting to the server device and may further include a second user terminal capable of connecting to the server device, the first user terminal includes a first input reception means for receiving an input of movement information regarding the movement of the virtual character and a second input reception means for receiving an input of action information regarding the actions of the virtual character. The server device includes a virtual character action means for causing the virtual character to execute a movement according to the movement information and an action according to the action information in the virtual 3D space, a generation means for generating drawing information for perspective-transforming the actions and movements of the virtual character and displaying them on the display screen, and a transmission means for transmitting the generated drawing information to the first user terminal or the second user terminal. By the server device transmitting the generated drawing information to the first user terminal or the second user terminal, the first user terminal or the second user terminal can display on the display screen the state in which the virtual character executes a movement according to the movement information and an action according to the action information in the virtual 3D space based on the drawing information, thereby allowing the performance of the virtual character created by the first user to be confirmed by the first user terminal or the second user terminal.

Explanation of Signs

[0112] 1 System 2 First User Terminal 3 Second User Terminal 4 Server Device 5 Communication Network 21 Control Unit 22 RAM 23 Storage Unit 24 Input Unit 25 Display Unit 26 Communication Interface 41 Control Unit 42 RAM 43 Storage Unit 44 Communication Interface 100 Execution Screen 101 Top View of the Stage 102 Grid Lines 103 Icons 104 Information Regarding Time 105 Trajectory of Movement 106 Play Button 107 Seek Bar 108 Stage Change Button 109 OK Button 200 Execution Screen 201 Image of the Real World 202 Image of the Virtual 3D Space 203 First User 204 Virtual Character 205 Shooting Start / End Button 206 Shooting Time 300 Execution Screen 301 Performance Information 302 Track Input Status 303 Performance Start Time 304 Performance End Time 305 Performance Display Column 306 Play Button 307 Seek Bar 308 Playback Time 309 Performance Time 310 Save Button 400 Execution Screen 401 First Avatar 402 Stage 403 Second Avatar

Claims

1. A system comprising at least one computer device, comprising: first input receiving means for receiving an input of movement information regarding the movement of a virtual character; second input receiving means for receiving an input of action information regarding the action of a virtual character; virtual character action means for causing the virtual character to execute a movement according to the received movement information and an action according to the received action information in a virtual three-dimensional space A system comprising:

2. display means for controlling to display on a display screen how the virtual character executes actions and movements The system according to claim 1, comprising:

3. The system according to claim 1 or 2, wherein the first input receiving means receives an input of movement information obtained by an operation on a user terminal operated by a user.

4. The system according to claim 1 or 2, wherein the second input receiving means receives an input of action information obtained by imaging a person's action.

5. The movement information includes position coordinates of a virtual three-dimensional space associated with time, The system according to claim 1 or 2, wherein the action information includes information regarding the action of the virtual character viewed from the horizontal direction.

6. A method executed in a system comprising at least one computer device, comprising: a first input receiving step of receiving an input of movement information regarding the movement of a virtual character; a second input receiving step of receiving an input of action information regarding the action of a virtual character; a virtual character action step of causing the virtual character to execute a movement according to the received movement information and an action according to the received action information in a virtual three-dimensional space A method comprising:

7. A system that may comprise a server device and a first user terminal connectable to the server device, and may further comprise a second user terminal connectable to the server device, comprising: The first user terminal: first input receiving means for receiving an input of movement information regarding the movement of a virtual character; second input receiving means for receiving an input of action information regarding the action of a virtual character Comprising: The server device: virtual character action means for causing the virtual character to execute a movement according to the movement information and an action according to the action information in a virtual three-dimensional space; generation means for generating rendering information for perspective-transforming the actions and movements of the virtual character and displaying them on a display screen transmission means for transmitting the generated drawing information to the first user terminal or the second user terminal comprising a system in which the server device transmits the generated drawing information to the first user terminal or the second user terminal, so that the first user terminal or the second user terminal displays, on the display screen, a state in which a virtual character executes a movement according to the movement information and an action according to the action information in a virtual three-dimensional space based on the drawing information.

8. A method executed in a system including a server device, a first user terminal connectable to the server device, and optionally a second user terminal connectable to the server device, comprising: a first input reception step in which the first user terminal receives an input of movement information regarding the movement of a virtual character; a second input reception step in which the first user terminal receives an input of action information regarding the action of the virtual character; a virtual character action step in which the server device causes the virtual character to execute a movement according to the movement information and an action according to the action information in a virtual three-dimensional space; a generation step in which the server device generates drawing information for displaying the action and movement of the virtual character on the display screen by performing a perspective transformation; a transmission step in which the server device transmits the generated drawing information to the first user terminal or the second user terminal; wherein, by the server device transmitting the generated drawing information to the first user terminal or the second user terminal, the first user terminal or the second user terminal displays, on the display screen, a state in which the virtual character executes a movement according to the movement information and an action according to the action information in a virtual three-dimensional space based on the drawing information. ​ ​ ​ ​

Images