System and method

Abstract

The present invention provides a system for controlling a virtual character.　The system comprises at least one computer device, said system including: a first input reception means for receiving input of movement information pertaining to a movement of a virtual character; a second input reception means for receiving input of action information pertaining to an action of the virtual character; and a virtual character action means for, in a virtual three-dimensional space, causing the virtual character to execute a movement corresponding to the received movement information and causing the virtual character to execute an action corresponding to the received action information.

Description

Systems and methods

[0001] The present invention relates to a system and a method.

[0002] In recent years, users have been enjoying a virtual space called the Metaverse, using virtual characters as their own avatars. Within the Metaverse space, users can freely move their avatars.

[0003] Additionally, within the Metaverse space, users can have their avatars perform dances and other performances. Data on the movements that the avatars perform can be obtained, for example, by having the user actually perform the movements in the real world and then using motion capture technology to digitize the movements.

[0004] SUMMARY OF THE INVENTION It is an object of the present invention to provide a system for controlling a virtual character.

[0005] According to the present invention, the above object can be achieved by providing: [1] a system including at least one computer device, the system including first input receiving means for receiving input of movement information related to the movement of a virtual character, second input receiving means for receiving input of movement information related to the movement of the virtual character, and virtual character operating means for causing the virtual character to move in a virtual three-dimensional space in accordance with the received movement information and to perform an action in accordance with the received action information; [2] the system according to [1], including display means for controlling to display on a display screen the state in which the virtual character performs an action and a movement; [3] the system according to [1] or [2], wherein the first input receiving means receives input of movement information obtained by operating a user terminal operated by a user; [4] the system according to any of [1] to [3], wherein the second input receiving means receives input of action information obtained by capturing an image of a person's action; [5] the system according to any of [1] to [4], wherein the movement information includes position coordinates in the virtual three-dimensional space associated with time, and the action information includes information regarding the action of the virtual character as viewed from a horizontal direction; [6] A method executed in a system including at least one computer device, the method comprising: a first input receiving step of receiving input of movement information related to movement of a virtual character; a second input receiving step of receiving input of action information related to action of the virtual character; and a virtual character action step of making the virtual character move in a virtual three-dimensional space in accordance with the received movement information and also make the virtual character act in accordance with the received action information;[7] A system comprising a server device, a first user terminal connectable to the server device, and optionally a second user terminal connectable to the server device, wherein the first user terminal comprises first input receiving means for receiving input of movement information relating to movement of a virtual character, and second input receiving means for receiving input of action information relating to action of the virtual character, wherein the server device comprises virtual character action means for causing the virtual character to move in a virtual three-dimensional space in accordance with the movement information and to act in accordance with the action information, a generation means for generating drawing information for performing perspective transformation on the action and movement of the virtual character and displaying the same on a display screen, and a transmission means for transmitting the generated drawing information to the first user terminal or the second user terminal, wherein the server device transmits the generated drawing information to the first user terminal or the second user terminal, and the first user terminal or the second user terminal displays on a display screen an image of the virtual character moving in the virtual three-dimensional space in accordance with the movement information and acting in accordance with the action information based on the drawing information; [8] A method executed in a system including a server device and a first user terminal connectable to the server device, and optionally including a second user terminal connectable to the server device, wherein the first user terminal has a first input receiving step of receiving input of movement information related to the movement of a virtual character, and a second input receiving step of receiving input of operation information related to the operation of the virtual character, and the server device has a virtual character operation step of causing the virtual character to move in a virtual three-dimensional space in accordance with the movement information and to perform an operation in accordance with the operation information, a generation step of generating drawing information for performing perspective transformation on the operation and movement of the virtual character and displaying the generated drawing information on a display screen, and a transmission step of transmitting the generated drawing information to the first user terminal or the second user terminal, wherein the server device transmits the generated drawing information to the first user terminal or the second user terminal, and the first user terminal or the second user terminal displays on the display screen an image of the virtual character moving in the virtual three-dimensional space in accordance with the movement information and performing an operation in accordance with the operation information based on the drawing information;

[0006] According to the present invention, a system for controlling a virtual character can be provided.

[0007] FIG. 1 is a block diagram showing a configuration of a system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a hardware configuration of a user terminal according to an embodiment of the present invention. FIG. 3 is a block diagram showing a hardware configuration of a server device according to an embodiment of the present invention. FIG. 4 is a flowchart of a performance production process according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of an execution screen according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of an execution screen according to an embodiment of the present invention. FIG. 7 is a flowchart of a performance execution process according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of an execution screen according to an embodiment of the present invention.

[0008] The following describes embodiments of the present invention, but the present invention is not limited to the following embodiments as long as they do not violate the spirit of the present invention. The order of each process constituting the flowcharts described below is random as long as no contradictions or inconsistencies occur in the process content. Furthermore, it is possible to omit some of the processes constituting the flowcharts or add new processes to each process constituting the flowcharts as long as they do not violate the spirit of the present invention. Furthermore, the device that executes each process constituting the flowcharts can be changed to another device as long as it does not violate the spirit of the present invention. In this case, the process content can be changed so as not to cause contradictions or inconsistencies in the process content.

[0009] FIG. 1 is a block diagram showing the configuration of a system according to an embodiment of the present invention. In FIG. 1, the system 1 includes a first user terminal 2, a second user terminal 3, and a server device 4. The first user terminal 2, the second user terminal 3, and the server device 4 may be communicatively connected to each other via a communication network 5. In the system 1, any of the first user terminal 2, the second user terminal 3, and the server device 4 may function as an information processing device. When any of the first user terminal 2, the second user terminal 3, and the server device 4 functions as an information processing device, information is transmitted and received between the first user terminal 2, the second user terminal 3, and the server device 4 as necessary. The number of user terminals may be three or more. The server device 4 may function in a distributed manner across multiple computer devices. For example, a distributed ledger technology such as a blockchain may be used instead of the server device 4. Alternatively, the system 1 may be configured only with the first user terminal 2, without including the second user terminal 3 and the server device 4. In other words, the system 1 only needs to include at least one computer device.

[0010] The first user terminal 2 may be operated by a user (hereinafter referred to as the first user) who causes a virtual character to perform a performance in the metaverse space. The first user can create a performance to be performed by the virtual character by using the system 1. Furthermore, the first user can cause the virtual character to perform the performance that he or she created by using the system 1. Hereinafter, the virtual character operated by the first user in the metaverse space is referred to as the first avatar. Hereinafter, it is assumed that the first user causes the first avatar to perform a performance.

[0011] The second user terminal 3 may be operated by a user (hereinafter referred to as the second user) who watches the first avatar perform in the metaverse space. By using the system 1, the second user can watch the performance of the first avatar through a virtual character that serves as the second user's avatar. Hereinafter, the virtual character operated by the second user in the metaverse space is referred to as the second avatar.

[0012] A dedicated application (hereinafter also referred to as a dedicated app) for using the system 1 may be installed on the first user terminal 2 and the second user terminal 3. By launching the dedicated app, the first user and the second user can display the metaverse space on the first user terminal 2 and the second user terminal 3. The first user can operate a first avatar in the metaverse space by inputting an operation to the first user terminal 2. Furthermore, the second user can operate a second avatar in the metaverse space by inputting an operation to the second user terminal 3.

[0013] The server device 3 may provide a metaverse space to a user. The metaverse space may be a virtual three-dimensional space. The server device 3 may also control the creation and execution of performances by virtual characters in the metaverse space.

[0014] 2 is a block diagram showing the hardware configuration of a user terminal according to an embodiment of the present invention. The first user terminal 2 includes a control unit 21, a RAM 22, a storage unit 23, an input unit 24, a display unit 25, and a communication interface 26, all of which are connected via a bus.

[0015] The control unit 21 is composed of a CPU and a ROM. The control unit 21 executes programs stored in the storage unit 23 and controls the first user terminal 2. The RAM 22 is a work area for the control unit 21. The storage unit 23 is a memory area for saving programs and data. In other words, the storage unit 23 functions as a recording medium that stores programs. The control unit 21 performs arithmetic processing based on the programs and data read from the RAM 22 and data input via the input unit 24.

[0016] The display unit 25 has a display screen. The control unit 21 outputs a video signal for displaying an image on the display screen according to the result of the arithmetic processing. Here, the display screen of the display unit 25 may be a touch panel equipped with a touch sensor. In this case, the touch panel functions as the input unit 24.

[0017] The communication interface 26 can be connected to the communication network 5 wirelessly or via a wire, and can transmit and receive data to and from other computer devices via the communication network 5. Data received via the communication interface 26 is loaded into the RAM 22, and the control unit 21 performs arithmetic processing on the data.

[0018] Although not shown, the first user terminal 2 preferably includes an imaging unit such as a camera. The imaging unit may be connected to other functional units via a bus.

[0019] Regarding the hardware configuration of the second user terminal 3, the description of the hardware configuration of the first user terminal 2 can be adopted to the extent necessary.

[0020] 3 is a block diagram showing the hardware configuration of a server device according to an embodiment of the present invention. The server device 4 includes at least a control unit 41, a RAM 42, a storage unit 43, and a communication interface 44, which are connected to each other via an internal bus.

[0021] The control unit 41 is composed of a CPU and ROM, and executes programs stored in the storage unit 43 to control the server device 4. The control unit 41 also has an internal timer that measures time. The RAM 42 is the work area of ​​the control unit 41. The storage unit 43 is a memory area for saving programs and data. In other words, the storage unit 43 functions as a recording medium that stores programs. The control unit 41 reads out the programs and data from the RAM 42, and performs program execution processing based on information received from the first user terminal 2 and / or the second user terminal 3.

[0022] The program may also be stored on a recording medium such as a CD-ROM. In this case, the program stored on the recording medium may be installed in the first user terminal 2, the second user terminal 3, and / or the server device 4 to execute a predetermined function.

[0023] Alternatively, the program may be distributed from a computer device external to the system. In this case, the program distributed from the computer device external to the system may be installed in the first user terminal 2, the second user terminal 3, and / or the server device 4 to execute a predetermined function.

[0024] Hereinafter, an example will be described in which a first user creates a performance to be performed by a first avatar on a dedicated app, and the first avatar performs the performance, and a second user and the first user watch the performance.

[0025] A user who uses the dedicated app can start the dedicated app on the device that the user operates and log in to the system 1. When a user logs in to the system 1 for the first time, the user may perform user registration. In user registration, the system 1 may store information about the user, such as the user's name and email address. By performing user registration, the system 1 may assign an account to the user. In the system 1, information about the user may be stored in association with the account assigned to the user. When a user logs in to the system 1, the user may be required to input an email address, a password, etc. The password may be set by the user.

[0026] A user can operate his / her own avatar in the metaverse space by logging in to the system 1. The user may be able to change the appearance of his / her avatar, such as the gender, hairstyle, facial features, and clothing.

[0027] Within the metaverse space, virtual characters (user avatars) that can be operated by users, virtual characters controlled by the system 1, backgrounds (sky, sea, ground, etc.), buildings, vehicles, and other objects may be depicted. The objects depicted within the metaverse space are displayed on the display screen of the user terminal by performing perspective transformation of the metaverse space using a virtual camera. Within the metaverse space, one virtual camera may be assigned to one user (account).

[0028] A virtual camera is placed in the metaverse space, and the image projected onto the virtual screen becomes the image displayed on the display screen. The position of this virtual camera is the viewpoint, the direction of the virtual camera is the visual axis, and the area formed by the four lines connecting the viewpoint and the four corners of the virtual screen is the field of view. A user may be able to change the position of the virtual camera, i.e., the user's viewpoint, by operating the user terminal. For example, a user may place the virtual camera in a position that reflects the back of their avatar, or in a position that corresponds to the viewpoint (head) of their avatar.

[0029] The method of manipulating and moving an avatar in the Metaverse space is not particularly limited, and any method can be employed. For example, if the avatar is composed of a virtual skeleton and polygons associated with the virtual skeleton, the avatar may be moved by moving the virtual skeleton. The polygons that make up the avatar are drawn to move in accordance with the movement of the virtual skeleton. The extent to which the virtual skeleton is moved may be determined based on an operation input to the user terminal. The position of the virtual skeleton and the position of the polygons may be represented in a world coordinate system or a local coordinate system.

[0030] [Performance Creation Process] The first user may be able to start creating a performance by pressing a button or the like displayed on the display screen of the first user terminal 2 to request the start of performance creation. Note that a "performance" refers to an act performed by a virtual character. A "performance" may include dancing, singing, acting, and the like. Preferably, a "performance" includes the movement and actions of the virtual character.

[0031] FIG. 4 is a flowchart of a performance production process according to an embodiment of the present invention.

[0032] First, the first user terminal 2 accepts input of movement information related to the movement of a virtual character (step S101). The input movement information is transmitted from the first user terminal 2 to the server device 4 (step S102). The transmitted movement information is received by the server device 4 (step S103).

[0033] The movement information in step S101 is not particularly limited as long as it is information relating to the movement of a virtual character. The movement information may be information relating to a change in the position of the virtual character. The change in position may be associated with time. The position of the virtual character may be represented by position coordinates in a virtual three-dimensional space. For example, the movement information may include position coordinates in the virtual three-dimensional space associated with time.

[0034] Furthermore, the movement information may be obtained by operating the first user terminal 2. For example, the movement information may be obtained by inputting position coordinates in a virtual three-dimensional space into the first user terminal 2. It is preferable that the movement information be input without using motion capture technology.

[0035] 5 is a diagram illustrating an example of an execution screen according to an embodiment of the present invention. In step S101, a movement information input screen as shown in FIG. 5 may be displayed on the display screen of the first user terminal 2 as the execution screen 100.

[0036] The movement information input screen shown in FIG. 5 displays a top view 101 of the stage, grid lines 102, icons 103 (103a to 103c) indicating the positions to which the virtual character will move, information 104 (104a to 104c) relating to the time the virtual character will move, a trajectory 105 (105a and 105b) of the movement of the virtual character, a play button 106, a seek bar 107, a stage change button 108, and an OK button 109.

[0037] The stage top view 101 may be a top view of a stage on which a virtual character performs. The stage top refers to the surface of the stage on which a performer stands. The stage top may be a surface parallel to the ground. FIG. 5 shows a rectangular top view.

[0038] The first user may be able to change the stage on which the virtual character performs a performance by pressing a stage change button 108. The shape of the top view of the stage is not particularly limited and may be designed as appropriate. The shape of the top view of the stage may be, for example, rectangular, square, circular, elliptical, or star-shaped.

[0039] In FIG. 5 , grid lines 102 in the form of a roughly square grid are applied vertically and horizontally to a top view 101 of the stage. Note that in FIG. 5 , the reference numerals of the grid lines 102 are omitted except for some. By applying the grid lines 102 to the top view 101 of the stage, it becomes easy to grasp the position on the top view 101 of the stage. The shape of the grid lines 102 is not particularly limited and can be designed as appropriate. The shape of the grid lines 102 may be, for example, a square grid, a rectangular grid, a honeycomb shape of equilateral triangles, a honeycomb shape of regular hexagons, a horizontal ruled line shape, or a vertical ruled line shape. Alternatively, the top view 101 of the stage may not have grid lines 102 applied to it.

[0040] The first user may be able to input a position to which the virtual character will move by selecting, for example, an arbitrary point on the top view 101 of the stage. The position coordinates of the selected point may be input as movement information. The position coordinates may be expressed in a world coordinate system or a local coordinate system.

[0041] A circle icon 103 as shown in Fig. 5 may be displayed at a location selected by the first user. The shape of the icon is not particularly limited and may be designed as appropriate. In Fig. 5, the number 1 is assigned to icon 103a, the number 2 is assigned to icon 103b, and the number 3 is assigned to icon 103c. The numbers assigned to the icons 103 may represent the order in which the virtual character moves. The order in which the virtual character moves may be input by the first user.

[0042] In FIG. 5 , icon 103a indicating the first position to which the virtual character will move and icon 103b indicating the second position to which the virtual character will move are connected by a movement trajectory 105a. Furthermore, icon 103b indicating the second position to which the virtual character will move and icon 103c indicating the third position to which the virtual character will move are connected by a movement trajectory 105b. In FIG. 5 , the movement trajectory 105 of the virtual character is a straight line. That is, the virtual character moves in a straight line between the icons 103. The movement trajectory 105 of the virtual character is not particularly limited and can be designed as appropriate. The movement trajectory 105 of the virtual character may be, for example, a straight line, an arc, or a serpentine line. The movement trajectory 105 of the virtual character may be input by the first user.

[0043] The first user may be able to input a time for which the virtual character moves in association with a position to which the virtual character moves. For example, the first user may be able to input a time for which the virtual character stays at a selected position and a time for which the virtual character moves between the selected positions. For example, the first user may be able to input a time for which the virtual character stays at a position indicated by each icon 103 by tapping or otherwise selecting information 104 related to time displayed near each icon 103. By inputting the time for which the virtual character stays at a position indicated by each icon 103, the time for which the virtual character moves between each icon 103 is specified. Furthermore, by inputting the time for which the virtual character stays at a position indicated by each icon 103, the order in which the virtual characters move may be specified.

[0044] In FIG. 5 , time information 104a displays "00:00 → 00:02," time information 104b displays "00:04 → 00:06," and time information 104c displays "00:07 → 00:10." Furthermore, time information 104a to 104c are displayed near icons 103a to 103c, respectively. Therefore, it is possible to understand that the virtual character will stay at the position indicated by icon 103a for two seconds from the start of the performance, and will then move to the position indicated by icon 103b over two seconds. It is also possible to understand that the virtual character will stay at the position indicated by icon 103b for two seconds, and will then move to the position indicated by icon 103c over one second. Furthermore, it is also possible to understand that the virtual character will stay at the position indicated by icon 103c for three seconds.

[0045] The first user may be able to preview the virtual character moving on the stage in accordance with the input movement information by pressing play button 106. For example, when play button 106 is pressed, the virtual character may be displayed on top view 101 of the stage. Also, the virtual character may be displayed walking from icon 103a to icon 103b and then to icon 103c in accordance with the time indicated in information 104 about time.

[0046] While checking the preview, the first user may be able to grasp the playback status of the preview. For example, a circular slider on the seek bar 107 may be displayed to move from the left end to the right end of the seek bar 107 as the preview is played. Furthermore, for example, information regarding the playback time of the preview may be displayed near the seek bar 107. In FIG. 5 , the preview start time, "00:00," is displayed below the left end of the seek bar 107, and the preview end time, "00:10," is displayed below the right end of the seek bar 107. In other words, the total playback time of the preview is 10 seconds. While the preview is being played, the preview playback time (the time elapsed since the preview started) may be displayed below the circular slider on the seek bar 107.

[0047] If there is no problem with the way the virtual character moves, the first user can confirm the input movement information by pressing OK button 109. When the first user presses OK button 109, the movement information may be transmitted to server device 4 in step S102.

[0048] Although the above example shows a case where position coordinates on a top view of the stage, i.e., coordinates on the xy plane in virtual three-dimensional space, are input as movement information, it is also possible to input coordinates on the z plane in virtual three-dimensional space. By including position coordinates on the z plane in virtual three-dimensional space in the movement information, it is possible to incorporate, for example, a flying effect of a virtual character into a performance.

[0049] Although the above example illustrates a mode in which the first user inputs the position and time for which the virtual character moves, the mode for inputting the movement information is not particularly limited and can be designed as appropriate. For example, the first user may be able to select any movement information from movement information templates in which the position and / or time for which the virtual character moves are predetermined.

[0050] Returning to the description of Fig. 4 , after the movement information is transmitted to the server device 4, the first user terminal 2 accepts input of motion information relating to the motion of the virtual character (step S104). The input motion information is transmitted from the first user terminal 2 to the server device 4 (step S105). The transmitted motion information is received by the server device 4 (step S106).

[0051] The movement information in step S104 is not particularly limited as long as it is information about the movement of the virtual character. The information about the movement of the virtual character may be information about the movement of the virtual skeleton of the virtual character. The information about the movement of the virtual skeleton may include information about the position coordinates of the virtual skeleton. The information about the movement of the virtual character may be associated with time.

[0052] Furthermore, the movement information may be obtained by capturing an image of the movement of the first user. The movement of the first user may be the movement of the first user's entire body or a part of the first user's body. The part of the body may be the upper body, an arm, a head, a lower body, or a leg. The movement information may include information regarding the movement of the virtual character viewed from a horizontal direction.

[0053] 6 is a diagram illustrating an example of an execution screen according to an embodiment of the present invention. In step S104, an operation information input screen as shown in FIG. 6 may be displayed on the display screen of the first user terminal 2 as the execution screen 200.

[0054] The motion information input screen shown in FIG. 6 displays an image 201 of the real world, an image 202 of a virtual three-dimensional space, a first user 203, a virtual character 204, a shooting start / end button 205, and a shooting time 206.

[0055] The real-world image 201 may be an image of the real world captured by a camera of the first user terminal 2. The real-world image 201 is preferably an image of the first user captured from a horizontal direction.

[0056] The movements of first user 203 displayed in real-world image 201 may be reflected in real time on virtual character 204 displayed in virtual three-dimensional space image 202 using motion capture technology. For example, the movements of first user 203 can be reflected on virtual character 204 by extracting the silhouette of first user 203 displayed in real-world image 201 to create a skeletal model and acquiring data such as joint angles in the movements of first user 203.

[0057] By displaying the virtual character 204 reflecting the action of the first user 203 together with the first user 203, the first user can input action information while understanding how the virtual character will perform the action of the first user. The first user 203 and the virtual character 204 may be displayed side by side vertically or horizontally on the display screen of the first user terminal 2.

[0058] By pressing a start / end shooting button 205 displayed on the motion information input screen, shooting of the first user's motion in the real world may be started or ended. Data capturing the captured motion of the first user may be input as motion information in step S104. The data capturing the motion of the first user may be associated with time (shooting time).

[0059] While shooting is in progress, the time that has elapsed since the start of shooting may be displayed as shooting time 206. After shooting is finished, the total time of the shot moving images may be displayed as shooting time 206.

[0060] When the shooting start / end button 205 is pressed during shooting and shooting ends, operation information may be transmitted to the server device 4 in step S105.

[0061] Although the above example illustrates inputting motion information by capturing an image of the first user's motion, the manner in which the motion information is input is not particularly limited and can be designed as appropriate. For example, the motion information may be obtained by converting the first user's actual motion in the real world into data using any motion capture technology. Examples of motion capture technology include video motion capture, optical motion capture, inertial sensor motion capture, mechanical motion capture, and magnetic motion capture.

[0062] In addition to the movement information and the action information, the first user may be able to input performance information related to the performance to be performed by the virtual character. The performance information may include, for example, facial expression information related to the facial expression of the virtual character, audio information related to the audio of the virtual character, music information related to the music to be played during the performance, production information related to the production on stage during the performance, effect information related to the effects to be generated by the virtual character during the performance, etc.

[0063] The manner in which performance information is input is not particularly limited and can be designed as appropriate. Facial expression information may be input, for example, by capturing an image of the face of the first user in the real world with a camera and capturing the facial expression of the first user. Audio information may be input, for example, by acquiring the voice of the first user in the real world with a microphone, or by inputting electronic audio data reading out arbitrary text information. Music information may be input, for example, as part or all of music data stored in the first user terminal 2 or the server device 4. Performance information may be input, for example, by the first user selecting a predetermined performance. Effect information may be input, for example, by the first user selecting a predetermined effect. The selectable types of performances and types of effects may be stored in advance in the system 1.

[0064] Each piece of performance information may be managed as a track that constitutes a performance in the system 1. A performance can be executed by overlapping and processing multiple pieces of information input as tracks.

[0065] Returning to the explanation of FIG. 4 , the first user can input performance information including movement information and action information, and then have the virtual character perform a performance. The first user terminal 2 receives an input of a request to have the virtual character perform a performance (hereinafter also referred to as an execution request) (step S107). The input execution request is transmitted from the first user terminal 2 to the server device 4 (step S108). The transmitted execution request is received by the server device 4 (step S109). The server device 4 causes the virtual character to move in accordance with the movement information and perform an action in accordance with the action information (step S110).

[0066] Next, the server device 4 generates drawing information for performing perspective transformation on the virtual character's actions and movements and displaying the same on the display screen (step S111). The server device 4 transmits the generated drawing information to the first user terminal 2 (step S112). The first user terminal 2 receives the transmitted drawing information (step S113). The first user terminal 2 displays the virtual character's actions and movements based on the received drawing information (step S114).

[0067] The first user terminal 2 receives an input of a request to store movement information and action information (hereinafter also referred to as a storage request) (step S115). The input storage request is transmitted from the first user terminal 2 to the server device 4 (step S116). The transmitted storage request is received by the server device 4 (step S117). The server device 4 stores the movement information and action information (step S118), and the performance production process ends.

[0068] 7 is a diagram illustrating an example of an execution screen according to an embodiment of the present invention. In step S107, a performance confirmation screen as shown in FIG. 7 may be displayed on the display screen of the first user terminal 2 as the execution screen 300.

[0069] The performance confirmation screen shown in Figure 7 displays performance information 301, track input status 302 (302a to 302g), performance start time 303, performance end time 304, performance display field 305, play button 306, seek bar 307, playback time 308, performance time 309, and save button 310.

[0070] The performance information 301 may be information relating to a performance to be performed by a virtual character. In Fig. 7, the performance information 301 includes a track input status 302, a performance start time 303, and a performance end time 304.

[0071] The track input status 302 may indicate the status of input of each piece of performance information as a track. In Fig. 7, the track input status 302 displays a movement information input status 302a, an action information input status 302b, a facial expression information input status 302c, a voice information input status 302d, a music information input status 302e, a performance information input status 302f, and an effect information input status 302g.

[0072] 7, the track input status 302 is shown in the form of a bar graph. The solid lines in the bar graph indicate that performance information has been input, and the dashed lines in the bar graph indicate that performance information has not been input.

[0073] Furthermore, the position within the bar graph may correspond to the time of the performance. For example, the left end of the bar graph may indicate the input status at the start of the performance, the center of the bar graph may indicate the input status during the performance, and the right end of the bar graph may indicate the input status at the end of the performance. Note that the longer the time that has elapsed since the start of the performance, the further to the right of the bar graph the input status will be.

[0074] For example, the bar graph for the movement information input status 302a is displayed with a solid line, which indicates that movement information was input throughout the entire performance. Furthermore, the number 1 is displayed at the left end of the bar graph, the number 2 is displayed in the center, and the number 3 is displayed at the right end of the bar graph, which indicates that the first position was input for a while after the performance started, the second position was input midway through the performance, and the third position was input at the end of the performance. Note that the first, second, and third positions correspond to the first, second, and third positions shown in FIG. 5 .

[0075] 7, the bar graphs for the movement information input status 302b, the facial expression information input status 302c, and the music information input status 302e are shown with solid lines, which indicates that the movement information, facial expression information, and music information are being input throughout the performance time. Also, in FIG. 7, the bar graph for the audio information input status 302d is shown with dashed lines, which indicates that no audio information is being input throughout the performance time.

[0076] 7, the left and right ends of the bar graph showing the input status of performance information 302f are displayed with solid lines, and the center is displayed with a dashed line. This indicates that performance information has been input for the first few seconds of the performance and the last few seconds before the end, while no performance information has been input for the last few seconds of the performance. Furthermore, the word "FIRE" is displayed in the solid line at the left end of the bar graph, and the word "FLOWER" is displayed in the solid line at the right end of the bar graph, indicating that a fire effect will be displayed on the stage for the first few seconds of the performance, and that a cherry blossom effect will be displayed for the last few seconds before the performance ends.

[0077] 7, the left and right ends of the bar graph showing the effect information input status 302g are displayed with dashed lines, and the center is displayed with a solid line. This indicates that no effect information is input for the first few seconds of the performance and the last few seconds of the performance, but that effect information is input for the middle few seconds of the performance. Furthermore, the word "SPARKLING" is displayed in the solid line in the center of the bar graph, indicating that a glowing effect of the virtual character will be displayed for the middle few seconds of the performance.

[0078] The first user may be able to check the performance rendered in accordance with the input performance information on the performance confirmation screen. For example, when the first user presses the play button 306, an execution request may be transmitted in step S108. Then, in step S110, the performance is executed in the virtual three-dimensional space in accordance with the performance information 301. That is, in step S110, the virtual character is caused to move in accordance with the movement information and to perform an action in accordance with the action information.

[0079] In addition, when the action information does not include leg action of the virtual character at the timing when a change in the position of the virtual character occurs based on the movement information, the server device 4 may cause the virtual character to perform a walking action.

[0080] Here, the movement of the virtual character according to the movement information may be considered as movement, and the movement of the virtual character according to the action information may be considered as action. For example, if the action information includes information about a movement that changes the position of the virtual character, the position of the virtual character will change according to the action information, and this change in position may also be considered as action.

[0081] "Performing a performance in accordance with the performance information 301" may include causing a virtual character to move in accordance with the movement information, causing a virtual character to move in accordance with the action information, causing a virtual character to make an expression in accordance with the expression information, playing a voice of the virtual character in accordance with the audio information, playing music in accordance with the music information, rendering a performance on the stage in accordance with the performance information, rendering an effect of the virtual character in accordance with the effect information, etc. When performing a performance, it is preferable that each piece of performance information is processed so that the times included in each piece of performance information are synchronized.

[0082] When a performance is performed, in step S111, drawing information is generated for performing perspective transformation of the performance and displaying it on a display screen. At this time, the position of the virtual camera that performs perspective transformation of the performance is not particularly limited and can be designed as appropriate. For example, the virtual camera may be positioned at a position from which the entire stage can be viewed, or at a position from which a portion of the stage can be viewed. Furthermore, the virtual camera may be positioned at a position from which the stage can be viewed from the front, the back, or the side.

[0083] In step S114, the state in which the performance is being performed, i.e., the state in which the virtual character is performing actions and moving, is displayed based on the generated drawing information. The state in which the performance is being performed may be displayed, for example, in a performance display field 305. In Fig. 7, the performance display field 305 displays the entire stage on which the performance is being performed as seen from the front.

[0084] While the performance is being played, the first user may be able to grasp the playback status of the performance. For example, a circle slider on the seek bar 307 may be displayed to move from the left end to the right end of the seek bar 307 as the performance is played. Furthermore, for example, information regarding the playback time of the performance may be displayed near the seek bar 307. In FIG. 7 , the information "00:10 / 00:10" is displayed below the left end of the seek bar 307. Here, the information displayed before the slash indicates the playback time 308 (the time elapsed since playback of the performance began), and the information displayed after the slash indicates the performance time 309 (the total time of the performance).

[0085] If there are no problems with the way the performance is being executed, the first user can store the input performance information by pressing the save button 310. When the first user presses the save button 310, input of a storage request may be accepted in step S115, and the storage request may be transmitted to the server device 4 in step S116.

[0086] In step S118, the performance information may be stored in the storage unit 43 of the server device 4. The movement information, action information, facial expression information, voice information, music information, performance information, effect information, and the like included in the performance information may be stored in association with each other as information related to one performance. The performance information may be stored in association with a number or name for identifying the performance.

[0087] In the above example, the virtual character performing the performance is the first avatar, but the virtual character performing the performance may be a virtual character different from the first avatar. Furthermore, when the virtual character performing the performance is the first avatar, the appearance of the first avatar while performing the performance may be the same as the usual appearance of the first avatar, or may be different from the usual appearance of the first avatar. The appearance of the virtual character performing the performance may be set by an operation of the first user.

[0088] [Performance Execution Process] The performance created in this way can be presented in the Metaverse space. Below, a description will be given of an example in which a first user causes a virtual character to perform a performance in the Metaverse space, and the first user and a second user watch the performance.

[0089] FIG. 8 is a flowchart of a performance execution process according to an embodiment of the present invention.

[0090] First, the first user terminal 2 receives an input of a performance execution request (step S201). The input execution request is transmitted from the first user terminal 2 to the server device 4 (step S202). The transmitted execution request is received by the server device 4 (step S203). The server device 4 causes the virtual character to move in accordance with the movement information and to perform an action in accordance with the action information (step S204).

[0091] The server device 4 generates second drawing information for performing perspective transformation on the actions and movements of the virtual character and displaying the second drawing information on the display screen of the second user terminal 3 (step S205). The server device 4 transmits the generated second drawing information to the second user terminal 3 (step S206). The second user terminal 3 receives the transmitted second drawing information (step S207). The second user terminal 3 displays the virtual character's actions and movements based on the received second drawing information (step S208), and the performance execution process ends.

[0092] Furthermore, after step S204, separately from steps S205 to S208, the server device 4 generates first drawing information for performing perspective transformation on the actions and movements of the virtual character and displaying it on the display screen of the first user terminal 2 (step S209). The server device 4 transmits the generated first drawing information to the first user terminal 2 (step S210). The first user terminal 2 receives the transmitted first drawing information (step S211). The first user terminal 2 displays the virtual character's actions and movements based on the received first drawing information (step S212), and the performance execution process ends.

[0093] The first user may input a request to perform a performance in step S201 by pressing a button or the like displayed on the display screen of the first user terminal 2 to request the start of performance execution. Also, in step S201, information for specifying the performance to be performed may be input along with the execution request. For example, the first user may select a performance to be performed by the virtual character and input a request to perform the performance.

[0094] The execution request may be input by an operation of a user other than the first user. Alternatively, if the performance information includes the start time of the performance, the performance may be executed when the predetermined time arrives, even if the execution request is not input by a user.

[0095] The description of the performance execution in the performance creation process can be adopted to the extent necessary for the execution of the performance in step S204. In step S204, the virtual character may be caused to move in accordance with the movement information and to perform an action in accordance with the action information.

[0096] In step S205, the position of the virtual camera that performs perspective transformation on the performance may be a position set as the viewpoint of the second user. The second user may place the virtual camera in a position that captures the back of their own avatar, or in a position that corresponds to the viewpoint (head) of their own avatar. The position of the virtual camera assigned to the second user may be changeable by an operation by the second user.

[0097] In step S208, the second user terminal 3 displays, based on the second drawing information, the virtual character moving in accordance with the movement information and performing an action in accordance with the action information.

[0098] 9 is a diagram illustrating an example of an execution screen according to an embodiment of the present invention. In step S208, a performance appreciation screen as shown in FIG. 9 may be displayed on the display screen of the second user terminal 3 as the execution screen 400.

[0099] The performance viewing screen shown in FIG. 9 displays a first avatar 401, a stage 402, and a second avatar 403.

[0100] First avatar 401 may be an avatar of a first user performing a performance, stage 402 may be a stage selected by the first user as the stage on which the performance will be performed, and second avatar 403 may be an avatar of a second user watching the performance.

[0101] 9, the virtual camera is positioned so as to capture the back of second avatar 403. The second user can view the performance by the first avatar from the viewpoint from which the second user views the Metaverse space.

[0102] In step S209, the position of the virtual camera that performs perspective transformation on the performance may be a position set as the first user's viewpoint, or may be a position set for viewing the performance from outside. The position set as the first user's viewpoint may, for example, be a position that reflects the back of the first avatar, or a position corresponding to the viewpoint (head) of the first avatar. The position set for viewing the performance from outside may, for example, be a position from which the entire stage can be viewed, or a position from which a portion of the stage can be viewed. Furthermore, the position set for viewing the performance from outside may be a position from which the stage can be viewed from the front, the back, or the side. The first user may be able to change the position of the virtual camera while the performance is being performed.

[0103] In step S212, the first user terminal 2 displays the virtual character moving in accordance with the movement information and performing the action in accordance with the action information based on the first drawing information. The first user can watch the performance by the first avatar from a viewpoint from which the first user views the inside of the Metaverse space or from a viewpoint from which the performance can be viewed from outside.

[0104] Although the above example shows both the second user and the first user watching the performance, either the second user or the first user may watch the performance, or a user other than the second user or the first user may watch the performance.

[0105] When both the movement information and the action information to be executed by the virtual character are input by motion capture, a space as large as a stage is required for the user to actually perform the performance. Furthermore, multiple cameras are required to film the user's actual performance. On the other hand, as described above, when the movement information is input by position coordinates and the action information is input by motion capture, all that is required is a user terminal with a single camera and a space large enough for the camera to capture the action. This makes it easier for the user to create a performance.

[0106] In this way, a system having at least one computer device can be provided that includes a first input receiving means for receiving input of movement information related to the movement of a virtual character, a second input receiving means for receiving input of operation information related to the operation of the virtual character, and a virtual character operation means for causing the virtual character to move in a virtual three-dimensional space in accordance with the received movement information and to perform an operation in accordance with the received operation information, thereby providing a system for controlling a virtual character.

[0107] Furthermore, since the system is provided with a display means for controlling the display of the virtual character's actions and movements on the display screen, it is possible to confirm the virtual character's movements and actions.

[0108] Furthermore, in this way, the first input receiving means receives input of movement information obtained by operating the user terminal operated by the user, so that the movement information can be input by operating the user terminal.

[0109] Furthermore, by having the second input receiving means receive input of movement information obtained by capturing an image of a person's movement, it is possible to make the virtual character perform the movement that the person actually performed.

[0110] Furthermore, since the movement information includes position coordinates in the virtual three-dimensional space associated with time, and the action information includes information related to the action of the virtual character as viewed from the horizontal direction, the virtual character can be made to move based on the position coordinates, and the actions to be performed by the virtual character can be made easier to understand.

[0111] Furthermore, in this system, which includes a server device, a first user terminal connectable to the server device, and may also include a second user terminal connectable to the server device, the first user terminal includes first input receiving means for receiving input of movement information related to the movement of a virtual character, and second input receiving means for receiving input of action information related to the action of the virtual character, and the server device includes virtual character operating means for causing the virtual character to move in a virtual three-dimensional space in accordance with the movement information and to perform an action in accordance with the action information, and for performing perspective transformation on the action and movement of the virtual character to display the action and movement of the virtual character on a display screen. The system is equipped with a generating means for generating drawing information to be displayed, and a transmitting means for transmitting the generated drawing information to a first user terminal or a second user terminal, and the server device transmits the generated drawing information to the first user terminal or the second user terminal, and the first user terminal or the second user terminal displays on a display screen how the virtual character moves in a virtual three-dimensional space in accordance with the movement information and performs actions in accordance with the action information based on the drawing information, thereby allowing the performance of the virtual character created by the first user to be confirmed on the first user terminal or the second user terminal.

[0112] 1 System 2 First user terminal 3 Second user terminal 4 Server device 5 Communication network 21 Control unit 22 RAM 23 Storage unit 24 Input unit 25 Display unit 26 Communication interface 41 Control unit 42 RAM 43 Storage unit 44 Communication interface 100 Execution screen 101 Top view of stage 102 Grid lines 103 Icon 104 Information related to time 105 Movement trajectory 106 Play button 107 Seek bar 108 Stage change button 109 OK button 200 Execution screen 201 Image of real world 202 Image of virtual three-dimensional space 203 First user 204 Virtual character 205 Start / end shooting button 206 Shooting time 300 Execution screen 301 Performance information 302 Track input status 303 Performance start time 304 Performance end time 305 Performance display area 306 Play button 307 Seek bar 308 Playback time 309 Performance time 310 Save button 400 Execution screen 401 First avatar 402 Stage 403 Second avatar

Claims

1. A system comprising at least one computer device, the system comprising: a first input receiving means for receiving an input of movement information regarding the movement of a virtual character; a second input receiving means for receiving an input of action information regarding the action of the virtual character; and a virtual character action means for causing the virtual character to execute a movement according to the received movement information and an action according to the received action information in a virtual three-dimensional space.

2. The system according to claim 1, further comprising a display means for controlling to display on a display screen a state in which the virtual character executes an action and a movement.

3. The system according to claim 1 or 2, wherein the first input receiving means receives an input of movement information obtained by an operation on a user terminal operated by a user.

4. The system according to claim 1 or 2, wherein the second input receiving means receives an input of action information obtained by imaging a human action.

5. The system according to claim 1 or 2, wherein the movement information includes position coordinates in a virtual three-dimensional space associated with time, and the action information includes information regarding the action of the virtual character viewed from the horizontal direction.

6. A method executed in a system comprising at least one computer device, the method comprising: a first input receiving step for receiving an input of movement information regarding the movement of a virtual character; a second input receiving step for receiving an input of action information regarding the action of the virtual character; and a virtual character action step for causing the virtual character to execute a movement according to the received movement information and an action according to the received action information in a virtual three-dimensional space.

7. A system may include a server device and a first user terminal connectable to the server device, and may further include a second user terminal connectable to the server device. The first user terminal includes: a first input receiving means for receiving an input of movement information regarding the movement of a virtual character; and a second input receiving means for receiving an input of action information regarding the action of the virtual character. The server device includes: a virtual character action means for causing the virtual character to execute a movement according to the movement information and an action according to the action information in a virtual three-dimensional space; a generating means for generating drawing information for displaying the action and movement of the virtual character on a display screen by perspective transformation; and a transmitting means for transmitting the generated drawing information to the first user terminal or the second user terminal. By transmitting the generated drawing information to the first user terminal or the second user terminal, the server device enables the first user terminal or the second user terminal to display on the display screen a state in which the virtual character executes a movement according to the movement information and an action according to the action information in the virtual three-dimensional space based on the drawing information.

8. A method executed in a system that includes a server device and a first user terminal connectable to the server device, and may further include a second user terminal connectable to the server device. The method includes: a first input receiving step in which the first user terminal receives an input of movement information regarding the movement of a virtual character; a second input receiving step in which the first user terminal receives an input of action information regarding the action of the virtual character; a virtual character action step in which the server device causes the virtual character to execute a movement according to the movement information and an action according to the action information in a virtual three-dimensional space; a generating step in which the server device generates drawing information for displaying the action and movement of the virtual character on a display screen by perspective transformation; a transmitting step in which the server device transmits the generated drawing information to the first user terminal or the second user terminal; and by transmitting the generated drawing information to the first user terminal or the second user terminal, the server device enables the first user terminal or the second user terminal to display on the display screen a state in which the virtual character executes a movement according to the movement information and an action according to the action information in the virtual three-dimensional space based on the drawing information.

Images