Display program, display method, and display system

Abstract

A display program controls the display of activities in a virtual space using characters that can be operated by a user, wherein the display program causes a computer to execute the following steps: a step of displaying a first screen including content information of the characters and content based on the activities; and a step of changing the display to a second screen in response to an operation related to the action of the characters, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, the character of the first screen and the character of the second screen differ in at least one of their form, state and function, and at least one of the character of the first screen and the character of the second screen causes an animation of a simulated ear to move in response to the user's operation.

Description

Technical Field

[0001] One aspect of the present disclosure relates to a display program, a display method, and a display system.

[0002] This application claims the priority based on Japanese application No. 2022-124758 filed on August 4, 2022, and uses all the contents described in the said Japanese application. Background Art

[0003] Japanese Patent Publication No. 2015-38725 (Patent Document 1) discloses a speech animation generation device for generating a facial image. The speech animation generation device uses a pre-prepared acoustic model, a language model, and a correspondence table between a plurality of phonemes and a plurality of visemes to gradually perform speech recognition on a speech signal to generate a viseme sequence. The speech animation generation device further smoothes the generated viseme sequence, generates a mixing weight for each of the plurality of visemes, mixes a plurality of facial models corresponding to the plurality of visemes according to the mixing weights for each of the plurality of visemes, and thereby generates a facial image.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-38725 Summary of the invention

[0007] A display program of one embodiment of the present disclosure controls the display of activities in a virtual space using characters that can be operated by a user, wherein the display program causes a computer to execute the following steps: a step of displaying a first screen including content information (contents) of a character and content based on the activity; and a step of changing the display to a second screen in response to an operation related to an action of the character, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, the character of the first screen and the character of the second screen differ in at least one of their form, state, and function, and at least one of the character of the first screen and the character of the second screen causes an animation of a simulated ear to move in response to the user's operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a diagram showing an example of a schematic configuration of a display system according to one embodiment.

[0009] Figure 2 This is a diagram showing an example of functional blocks of a terminal and a server.

[0010] Figure 3 It is a perspective view of the operating device.

[0011] Figure 4Yes means Figure 3 Figure 2 shows an example of functional blocks of an operating device.

[0012] Figure 5 This is a diagram showing an example of the movement of the ear animation.

[0013] Figure 6 This is a diagram showing an example of an activity selection screen.

[0014] Figure 7 This is a diagram showing an example of a screen displayed in a virtual space.

[0015] Figure 8 This is a diagram showing an example of a screen displayed in a virtual space.

[0016] Fig. 9 This is a diagram showing an example of a screen displayed in a virtual space.

[0017] Fig.10 This is a diagram showing an example of a screen displayed in a virtual space.

[0018] Fig.11 This is a diagram showing an example of a screen displayed in a virtual space.

[0019] Fig.12 This is a diagram showing an example of a screen displayed in a virtual space.

[0020] Fig.13 This is a diagram showing an example of a screen displayed in a virtual space.

[0021] Fig.14 This is a diagram showing an example of a screen displayed in a virtual space.

[0022] Fig.15 The diagram shows an example of the movement of the ear animation and an example of a horizontal bar chart.

[0023] Fig.16 This is a diagram showing an example of a screen displayed in a virtual space.

[0024] Fig.17 This is a diagram showing an example of a screen displayed in a virtual space.

[0025] Fig.18 This is a diagram showing an example of a screen displayed in a virtual space.

[0026] Fig.19 This is a diagram showing an example of a screen displayed in a virtual space.

[0027] Fig. 20 This is a diagram showing an example of a screen displayed in a virtual space.

[0028] Fig.21 This is a diagram showing an example of a screen displayed in a virtual space.

[0029] Fig. 22 This is a diagram for explaining the settings of the character's placement.

[0030] Fig.23 This is a diagram showing an example of a screen displayed in a virtual space.

[0031] Fig.24 This is a diagram showing an example of a screen displayed in a virtual space.

[0032] Fig.25 This is a diagram showing an example of a screen displayed in a virtual space.

[0033] Fig.26 This is a diagram showing an example of a screen displayed in a virtual space.

[0034] Fig. 27 This is a diagram showing an example of a screen displayed in a virtual space.

[0035] Fig.28 This is a diagram showing an example of a screen displayed in a virtual space.

[0036] Fig.29 This is a diagram showing an example of a screen displayed in a virtual space.

[0037] Fig.30 This is a diagram showing an example of a screen displayed in a virtual space.

[0038] Fig.31 This is a diagram showing an example of a screen displayed in a virtual space.

[0039] Fig.32 This is a diagram showing an example of a screen displayed in a virtual space.

[0040] Fig.33 This is a diagram showing an example of a screen displayed in a virtual space.

[0041] Fig.34 This is a diagram showing an example of a screen displayed in a virtual space.

[0042] Fig.35 This is a diagram showing an example of a screen displayed in a virtual space.

[0043] Fig.36This is a diagram showing an example of a screen displayed in a virtual space.

[0044] Fig.37 This is a diagram showing an example of deformation of a character's body.

[0045] Fig.38 This is a diagram showing an example of a state in which a display unit is displayed on a character.

[0046] Fig.39 This is a diagram showing an example of a screen displayed in a virtual space.

[0047] Fig.40 This is a diagram showing an example of deformation of a character's body.

[0048] Fig.41 This is a diagram showing an example of deformation of a character's body.

[0049] Fig.42 This is a diagram showing an example of deformation of a character's body.

[0050] Fig.43 This is a diagram showing an example of a state in which a display unit is displayed on a character.

[0051] Fig.44 This is a diagram showing an example of character transformation.

[0052] Fig.45 This is a diagram showing an example of deformation of a character's body.

[0053] Fig.46 This is a diagram showing an example of deformation of a character's body.

[0054] Fig.47 This is a diagram showing an example of deformation of a character's body.

[0055] Fig.48 This is a diagram showing an example of a screen displayed in a virtual space.

[0056] Fig.49 This is a diagram showing an example of a screen displayed in a virtual space.

[0057] Fig.50 This is a diagram showing an example of a screen displayed in a virtual space.

[0058] Fig.51 This is a diagram showing an example of a screen displayed in a virtual space.

[0059] Fig.52 This is a diagram showing an example of a screen displayed in a virtual space.

[0060] Fig.53 This is a diagram showing an example of a screen displayed in a virtual space.

[0061] Fig.54 This is a diagram showing an example of a screen displayed in a virtual space.

[0062] Fig.55 This is a diagram showing an example of a screen displayed in a virtual space.

[0063] Fig.56 This is a diagram showing an example of a screen displayed in a virtual space.

[0064] Fig.57 This is a diagram showing an example of a screen displayed in a virtual space.

[0065] Fig.58 This is a diagram showing an example of a screen displayed in a virtual space.

[0066] Fig.59 This is a diagram showing an example of a screen displayed in a virtual space.

[0067] Fig.60 This is a diagram showing an example of a screen displayed in a virtual space.

[0068] Fig.61 This is a diagram showing the display form of the character and the display unit.

[0069] Fig.62 This is a diagram showing a time chart of video recording. DETAILED DESCRIPTION

[0070] [Problems to be Solved by the Present Disclosure]

[0071] In recent years, services using characters such as avatars in virtual spaces such as the metaverse have been provided. There is a demand for improved user convenience in the activities of such characters in the virtual space.

[0072] An object of the present disclosure is to provide a display program, a display method, and a display system that can improve the convenience of a user who uses a character when the character acts in a virtual space.

[0073] [Effects of the present disclosure]

[0074] According to the present disclosure, when a character moves in a virtual space, it is possible to improve the convenience of a user who uses the character.

[0075] [Description of Embodiments of the Present Disclosure]

[0076] First, the contents of the embodiments of the present disclosure will be described by listing them. At least a part of the embodiments described below may be arbitrarily combined.

[0077] (1) A display program according to one aspect of the present disclosure controls the display of activities in a virtual space using characters that can be operated by a user, wherein the display program causes a computer to execute the following steps: a step of displaying a first screen including content information of the characters and content based on the activities; and a step of changing the display to a second screen in response to an operation related to an action of the characters, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, the character of the first screen and the character of the second screen differ in at least one of their form, state, and function, and at least one of the character of the first screen and the character of the second screen causes an animation of a simulated ear to move in response to the operation of the user.

[0078] In a display program of one aspect of the present disclosure, when an operation related to the action of a character is performed in a first screen including content information of a character and content based on an activity, the display is changed to a second screen including a character corresponding to the character in the first screen. Therefore, in the display program, a screen change corresponding to the action of the character that performs an action through the operation intended by the user can be performed. That is, in the display program, the screen changes in response to the operation intended by the user. Therefore, in the display program, when a character performs an activity in a virtual space, the convenience of the user who uses the character can be improved.

[0079] (2) In the display procedure of (1) above, a plurality of display areas may be displayed in a virtual space, and a character may be displayed in at least a portion of the plurality of display areas. Each of the plurality of display areas for displaying the character may be divided corresponding to the attributes of the character, and the shape of the character may be changed in each display area. In this configuration, for example, the size of the leading character who explains the content information, etc., may be displayed larger in one display area. Therefore, the explanation, etc. may be easily understood.

[0080] (3) In the display program of (1) or (2) above, a plurality of display areas may be displayed in the virtual space, and the characters may be displayed in at least a portion of the plurality of display areas, wherein the plurality of display areas for displaying the characters include: a display area in which the output of the voice of the characters in the virtual space is restricted; and a display area in which the voice of the characters in the space can be output. In this configuration, the output of the voice is automatically switched on and off in the display area, thereby eliminating the need for switching the output on and off.

[0081] (4) In the display program of (2) or (3) above, the characters displayed in the display area may be displayed in a manner that allows them to be distinguished based on the attributes of the users. In this configuration, the attributes of the users participating in the event can be confirmed (understood) at a glance.

[0082] (5) In any of the display procedures of (2) to (4) above, the voice of a character displayed in one display area may be made inaudible to the voice of a character displayed in another display area in a plurality of display areas. In this configuration, the characters can have a conversation in each display area.

[0083] (6) In any of the display programs in (2) to (5) above, when an operation of dragging and dropping a character from one display area to another display area is accepted in the first screen, the display of the character in the one display area and the other display area is changed in the second screen. In this configuration, the display of the character is changed by performing an operation such as dragging and dropping the character, which is accompanied by the movement of the character. Therefore, it is expected that the feeling of participation in the movement time and the movement action of the character can be easily generated, and the effect of easily bringing in emotions such as switching to a new scene can be expected.

[0084] (7) In any of the display programs (1) to (6) above, when a predetermined operation is received from the user, the front or back directions of the plurality of characters may be changed at once for display. In this configuration, the front or back directions of the plurality of characters can be easily changed.

[0085] (8) In any of the display programs in (1) to (7) above, the display unit may be configured to display a part of the character on the second screen by at least one of a predetermined operation and voice input by the user on the first screen. In this configuration, the display unit can be displayed on the character. Therefore, the display unit of the character can display materials used in the character's presentation, a translation of the character's conversation, and the like.

[0086] (9) In the display program of (8) above, at least one of a conversation text converted into text of the character's speech, a translation of the conversation text, and a translation of the display content of the display unit may be displayed on the display unit. In this configuration, communication between different languages ​​can be achieved.

[0087] (10) In the display program of (8) or (9) above, at least one of a required picture, text, music, moving image, still image, and website may be displayed on the display unit by using AI based on at least one of voice and text input of the character. In this configuration, the explanation given by the character is easier to understand.

[0088] (11) In the display program of (1) above, a display unit for displaying content information may be displayed in a virtual space. In this configuration, the content information is displayed on the display unit, so that the content information is easy to recognize.

[0089] (12) In the display program of (11) above, the input content input by the user may be displayed on the display unit. In this configuration, opinions on the content information and the like can be displayed on the display unit.

[0090] (13) In the display program of (11) or (12) above, when a drag-and-drop operation of a character is received with respect to the display unit, a predetermined function may be executed on the display unit. In this configuration, by performing an operation such as drag-and-dropping a character accompanied by the movement of the character, it is expected that the effect of easily generating a feeling of participation in the movement time and movement action of the character, and easily bringing in emotions to the new scene switching, etc., can be expected.

[0091] (14) In the display program of (13) above, after the character is dragged and dropped relative to the display unit, when an operation of dragging and dropping the character displayed on the display unit to an area outside the display unit is accepted, the execution of the prescribed function is terminated. In this configuration, the prescribed function can be terminated by a simple operation of dragging and dropping the character.

[0092] (15) In any of the display programs of (11) to (14) above, when a predetermined operation on the display unit is received from the user, an operation screen for operating the display content on the display unit is displayed. In this configuration, the operation screen can be easily displayed by the predetermined operation performed by the user.

[0093] (16) In any of the display programs of (11) to (15) above, the size of the character may be changed, and when the size of one of the multiple characters displayed in the virtual space is changed, the sizes of the other characters are also changed in the same manner. In this configuration, since the sizes of all the characters are changed in the same manner, a unified activity can be realized in the virtual space.

[0094] (17) In the display program of (1) above, a content information display unit for displaying content information, a text display unit for displaying a conversation text in which the voice of a character is converted into text, and at least two of the characters may be displayed in a virtual space. In this configuration, for example, when the activity is counseling, the counseling can be promoted while confirming the content information (counseling content, etc.) that is the purpose of the counseling and the content of the conversation.

[0095] (18) In the display procedure of (17) above, when an operation on the conversation text displayed on the text display unit is accepted from the user, the conversation text may be displayed on the content information display unit. In this configuration, for example, when the user clicks or drags and drops the conversation text, the conversation text is displayed on the content information display unit. Thus, new matters, important matters, etc. generated during the conversation can be displayed on the content information display unit. Therefore, the consultation can be made more effective.

[0096] (19) In the display program of (17) or (18) above, the configuration and size of the characters may be changed, and when the configuration or size of one of the two characters displayed in the virtual space is changed, the configuration or size of the other character is also changed in accordance with the configuration or size of the one character. In this configuration, the configuration or size of the two characters is changed in accordance with the configuration or size of the one character, so that activities with a sense of unity can be realized in the virtual space.

[0097] (20) In the display program of (11) above, one or more moving bodies may be displayed, and the one or more moving bodies are provided for one or more characters and are configured to be movable in the virtual space. In this configuration, the characters can move on the moving bodies in the virtual space. Thus, for example, the characters can engage in activities such as sightseeing and museum viewing while riding on the moving bodies.

[0098] (21) In the display program of (20) above, when a plurality of characters are arranged on each of a plurality of moving bodies, the voices of the plurality of characters arranged on one moving body and the voices of the plurality of characters arranged on other moving bodies may be made inaudible to each other. In this configuration, a conversation can be conducted for each group of characters riding on the moving bodies.

[0099] (22) In the display program of (11) above, explanations about the content information displayed on the display unit may be displayed. In this configuration, for example, when the activity is art gallery viewing, explanations about paintings that are content information may be displayed.

[0100] (23) In the display program of (11) above, the animation of the character in the action in the virtual space may be output as an editable video. In this configuration, the animation can be edited after the action is created as a video.

[0101] (24) In any of the display procedures of (1) to (23) above, an image or video of the user's face may be displayed on a part of the character. In this configuration, when it is necessary to display the user's face, the face can be displayed temporarily, for example.

[0102] (25) In any of the display programs of (1) to (24) above, one or more characters may be displayed in a virtual space, and at least one of the one or more characters may be operated by an AI. In this configuration, it is also applicable to a case where the AI ​​is a user.

[0103] (26) In the display program of (25) above, the character operated by AI may cause an animation of simulated ears to move through the operation of AI, and a display unit corresponding to the character operated by AI may be displayed, and at least one of text generated by AI, a still image associated with the text, a dynamic image associated with the text, an article associated with the text, and website information associated with the text may be displayed on the display unit. In this configuration, AI causes the ear animation to move according to the state and condition of AI and the state and condition of the other party of communication. In this way, by displaying the ear animation as non-verbal information, more emotional communication can be achieved compared to communication with text only. In addition, since the text-related information of AI is displayed on the display unit, it is easier to understand the explanation given by the character (AI).

[0104] (27) In the display program of (25) or (26) above, the character may output a voice corresponding to the text generated by the AI. In this configuration, the state of the AI ​​can be recognized by the voice.

[0105] (28) In any of the display programs in (1) to (27) above, the character may be rotated according to the position of a specified object so that the character can be imagined to be looking at the specified object. In this configuration, the character is rotated in accordance with the transition of the screen, the change of at least one of the character's form, state, and function, so that the character can be imagined to be looking at (paying attention to) the specified object, thereby imagining that the character is concerned about the specified object. The character's rotational movement becomes a non-verbal communication effect similar to the change of line of sight or posture, showing active concern for the conversation. Therefore, rich communication such as improved responsiveness to the conversation and improved sense of presence can be achieved.

[0106] (29) A display method according to one aspect of the present disclosure controls the display of activities in a virtual space using characters that can be operated by a user, wherein the display method includes the following steps: a step of displaying a first screen including content information of the character and content based on the activities; and a step of changing the display to a second screen in response to an operation related to an action of the character, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, the character of the first screen and the character of the second screen differ in at least one of form, state and function, and at least one of the character of the first screen and the character of the second screen causes an animation of a simulated ear to move in response to the operation of the user.

[0107] In a display method of one aspect of the present disclosure, when an operation related to the action of a character is performed in a first screen including content information of a character and content based on an activity, the display is changed to a second screen including a character corresponding to the character in the first screen. Therefore, in the display method, a screen change corresponding to the action of the character that performs an action through an operation intended by a user can be performed. That is, in the display method, the screen change is made in response to the operation intended by the user. Therefore, in the display method, when a character performs an activity in a virtual space, the convenience of the user who uses the character can be improved.

[0108] (30) A display system according to one aspect of the present disclosure controls the display of activities in a virtual space using characters that can be operated by a user, wherein the display system is configured to: display a first screen including content information of the characters and content based on the activities; and in response to an operation related to an action of the characters, change the display to a second screen, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, the character of the first screen and the character of the second screen differ in at least one of form, state and function, and at least one of the character of the first screen and the character of the second screen causes an animation of a simulated ear to move in response to the operation of the user.

[0109] In a display system of one aspect of the present disclosure, when an operation of a character's action is performed in a first screen including content information of a character and content based on an activity, the display is changed to a second screen including a character corresponding to the character in the first screen. Therefore, in the display system, a screen change corresponding to the action of the character that performs the action through the operation intended by the user can be performed. That is, in the display system, the screen is changed according to the operation intended by the user. Therefore, in the display system, when a character performs an activity in a virtual space, the convenience of the user who uses the character is improved.

[0110] [Details of the embodiments of the present disclosure]

[0111] Hereinafter, specific examples of the embodiments of the present disclosure will be described with reference to the accompanying drawings. The present invention is not limited to these examples, but is shown by the claims, and is intended to include all changes within the meaning and scope equivalent to the claims. In the description of the drawings, the same elements are marked with the same reference numerals, and repeated descriptions are omitted.

[0112] Figure 1 1 is a diagram showing an example of a schematic configuration of a display system (hereinafter, sometimes referred to as a "system"). The system 100 is a virtual space R (see Figure 7) in a virtual space R using a character C (see Figure 5 Mode A), and the character C makes the ear animation E according to the operation of the users U1, U2, and U3 (refer to Figure 5 Mode A) of the operation.

[0113] The character C represents the users U1, U2, and U3. The character C is a dynamic character. The character C is generated by image materials that are independent of the actual images (photographing data, etc.) of the users U1, U2, and U3. The image materials are materials for generating animated characters. The character C can also be drawn by two-dimensional or three-dimensional computer graphics (CG: Computer Graphic).

[0114] As an example, character C is composed of body B (see Figure 5 Mode A) and ear animation E. For character C, body B and ear animation E are displayed as a whole. Being displayed as a whole means that when observing character C, body B and ear animation E are visually recognized as a whole. In other words, being displayed as a whole means that it is possible to recognize at a glance that body B and ear animation E are related (corresponding). Therefore, being displayed as a whole can be a state in which body B and ear animation E are in contact or a state in which body B and ear animation E are separated.

[0115] Body B may include at least one of an image representing the user and text representing the user. A shadow may be displayed on character C. Body B may also be spherical and include text ("A", "B", "C") that can identify the user. Character C may be configured to include a speech animation (not shown), or may be configured to include animations that simulate arms, legs, etc.

[0116] Ear animation E is an animation that expresses non-verbal information. Ear animation E is an animation that expresses the user's state (action) and emotion. Ear animation E is an animation that expresses the user's state in a form different from the user's actual action part. Ear animation E may or may not include dynamically changing facial expressions. Ear animation E is displayed as protruding from the display area of ​​body B to the outside of the display area. Ear animation E is an animation that simulates the ears of animals (for example, rabbits, etc.), and moves according to changes in the user's state. Ear animation E moves according to the user's actions. Ear animation E performs actions such as stretching, bending, and swinging. Ear animation E is freely deformable, and visual deformation that recalls the states of users U1, U2, U3 is performed in accordance with the states of users U1, U2, U3.

[0117] In the system 100 , the status, emotion, reaction, etc. of the users U1 , U2 , and U3 can be confirmed through the ear animation E of the character C.

[0118] <System Configuration>

[0119] System 100 includes terminal 10, terminal 20 and terminal 30. Figure 1 In the example shown, terminal 10 is a laptop personal computer (PC: Personal Computer) used by user U1. Terminal 20 is a portable telephone (smartphone, etc.) used by user U2. Terminal 30 is a desktop PC used by user U3. Terminal 10, terminal 20 and terminal 30 can be connected to network N. In this example, terminal 10 is connected to network N via access point AP, terminal 20 is connected to network N via base station BS, and terminal 30 is connected to network N via a wire (not shown). In addition to the above-mentioned PCs and portable telephones, tablet terminals, wearable terminals, head mounted display (HMD: Head Mounted Display) terminals, and other various terminals can also be used for terminal 10, terminal 20 and terminal 30. It should be noted that the number of terminals included in system 100 is not limited to. Figure 1 Example.

[0120] The system 100 includes a server 40. The server 40 is connected to a network N. Therefore, the terminal 10, the terminal 20, the terminal 30 and the server 40 can communicate with each other via the network N. It should be noted that the number of servers included in the system 100 is not limited to Figure 1 Example.

[0121] The configuration of the network N is not limited. For example, the network N may be configured to include the Internet (public network), a communication carrier network, a carrier network, and the like.

[0122] Figure 2 1 is a diagram showing an example of functional blocks of the terminal 10, the terminal 20, the terminal 30, and the server 40. The functional blocks of the terminal 20 and the terminal 30 are the same as the functional blocks of the terminal 10, so the terminal 10 and the server 40 will be described below. The terminal 10 will be described first, and then the server 40 will be described.

[0123] <Terminal configuration>

[0124] The terminal 10 includes an input unit 11. The input unit 11 is a part for inputting voice data and image data of the user U1. The input unit 11 may also be used to input image data of the user U1. The image includes the voice data and image data of the user U1.

[0125] The terminal 10 includes a control unit 12. The control unit 12 is a part that generates control data of the user U1. The control data of the user U1 is generated based on the action data output from the operating device 50. In the present embodiment, for example, data for causing the ear animation E of the character C to perform six actions of "nodding", "agreeing", "appreciating (clapping)", "questioning", "raising hands" and "surprised" can be generated as control data. The control unit 12 can also generate control data about other actions (for example, actions indicating "yes (YES)" and "no (NO)").

[0126] The control data and the action data are associated with each other. In the present embodiment, the action data and the control data of "nod", "agree", "applaud (clap)", "question", "raise hand" and "surprise" are associated with each other. The action data is data sent from the operating device 50. When the action data and the control data are associated with each other, the control unit 12 generates the control data.

[0127] The control data may also be generated based on the voice data and image data of the user U1 input to the input unit 11. The image data is composed of a plurality of continuous unit frames. The unit frame generally includes the image information of the user U1. For example, image data of 60 frames / second includes 60 unit frames in one second. The details of the generation of the control data by the control unit 12 will be described later.

[0128] The control unit 12 is also a part that controls the motion of the character C based on the control data. The control unit 12 displays the character C in the virtual space R and controls the motion of the ear animation E of the character C based on the generated control data and the received control data.

[0129] The control unit 12 generates image data of the virtual space R. The image data of the virtual space R includes image data of the character C controlled based on the control data. In addition, various objects appearing in the virtual space R can be included in the image data of the virtual space R. The method for generating the virtual space R by the control unit 12 is not particularly limited, and various known methods can be used. The control unit 12 can also generate the image data of the virtual space R based on a captured image obtained by a camera, for example. Alternatively, the control unit 12 can also generate the image data of the virtual space R based on three-dimensional CAD (Computer Aided Design) data.

[0130] The terminal 10 includes a storage unit 13. The storage unit 13 is a part that stores information necessary for the control (processing) of the terminal 10. Figure 21 shows a terminal program 13P as information stored in the storage unit 13. The terminal program 13P is a program for operating the terminal 10, and operates the computer in a manner such as to execute the control performed by the control unit 12. Since the system 100 is a display system, the terminal program 13P can also be called a display program.

[0131] The terminal 10 includes a communication unit 14. The communication unit 14 communicates with the network N (see Figure 1 ) is a part that communicates with an external device of the terminal 10. Examples of the external device of the terminal 10 are the terminal 20, the terminal 30, the server 40, and the operating device 50. The communication unit 14 sends the voice data and the control data generated by the control unit 12 to the server 40, or receives the voice data and the control data from the server 40. The communication unit 14 is a part that communicates with the operating device 50. The communication unit 14 receives the action data from the operating device 50.

[0132] The terminal 10 includes an output unit 15 . The output unit 15 outputs an image of the virtual space R. The image of the virtual space R is an image based on image data of the virtual space R generated by the control unit 12 .

[0133] <Server Configuration>

[0134] The server 40 includes a control unit 42. The control unit 42 controls the operation of the server 40. The server 40 includes a storage unit 43. The storage unit 43 is a part that stores information necessary for controlling the server 40. Figure 2 4 shows a server program 43P and received data 43S as information stored in the storage unit 43. The server program 43P is a program for causing the server 40 to operate, and causes the computer to operate in a manner such as executing the control performed by the control unit 42. The received data 43S is data sent from the terminal 10, the terminal 20, and the terminal 30 to the server 40, and may include control data generated by the terminal 10, the terminal 20, and the terminal 30.

[0135] The server 40 includes a communication unit 44. The communication unit 44 communicates with the server 40 via a network N (see Figure 1 ) communicates with an external device of the server 40. Examples of the external device of the server 40 are the terminal 10, the terminal 20, and the terminal 30. The communication unit 44 receives control data of the user U1, the control data of the user U2, and the control data of the user U3 from the terminal 10, the terminal 20, and the terminal 30, respectively, or sends control data generated by the terminal 10, the terminal 20, and the terminal 30 to the server 40.

[0136] <Hardware Configuration>

[0137] An example of hardware configuration related to each functional part of the terminal 10 and the server 40 is described. The communication unit 44 of the server 40 can be configured using a network card or a wireless communication device so that it can access the network N. The control unit 42 of the server 40 can be configured using a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a clock, and a built-in memory. The control unit 42 can also be configured as a hardware (SoC: System On a Chip) obtained by integrating a processor, a clock, a built-in memory, a storage unit 43, and a communication unit 44. The control unit 42 operates based on the server program 43P, thereby causing the server computer to operate as the server 40. The storage unit 43 of the server 40 can be configured using a non-volatile storage medium such as a flash memory, a hard disk, and an SSD (Solid State Disk).

[0138] The input unit 11 of the terminal 10 can be configured using a microphone, a camera, and the like. The voice of the user U1 is acquired through the microphone. That is, the microphone is a type of voice sensor that acquires voice. The image of the user U1 is acquired through the camera. That is, the camera is a type of image sensor that acquires images. Moreover, the input unit 11 can also be configured using operating devices such as a keyboard, a mouse, and a touch panel. The control unit 12 of the terminal 10 can be configured in the same manner as the control unit 42 of the server 40. The control unit 12 operates based on the terminal program 13P, thereby causing a general-purpose computer to operate as the terminal 10. The storage unit 13 of the terminal 10 can be configured in the same manner as the storage unit 43 of the server 40. The communication unit 14 of the terminal 10 can be configured in the same manner as the communication unit 44 of the server 40. The output unit 15 of the terminal 10 can be configured using a display device such as a liquid crystal panel, an organic EL (Electro Luminescent) panel, etc. (it can also be a touch panel). Moreover, the output unit 15 can also be configured using a speaker.

[0139] <Configuration of the operating device>

[0140] Figure 3 It is a perspective view of the operating device 50 . Figure 3 1 is a diagram showing a state where the operating device 50 is operated by the finger F of the user U1, U2, U3. The operating device 50 is a device operated by the user U1, U2, U3. The ear animation E moves by the operation of the operating device 50 by the user U1, U2, U3. The operating device 50 is a touch panel. As the touch panel type, an electrostatic capacitance type, a pressure-sensitive touch type, an ultrasonic type, etc. can be used.

[0141] The operating device 50 has a main body 51 as a hardware configuration. The main body 51 has an operating surface 51A. The operating device 50 is operated by a finger F of a hand H of the user U1, U2, U3. The operating device 50 can be provided as a single unit or as part of a remote controller for operating a television display or the like. That is, the operating device 50 can also be part of a device.

[0142] Figure 4 is a diagram showing an example of a functional block of the operating device 50. Figure 4 As shown, the operating device 50 includes a communication unit 52. The communication unit 52 is a part that communicates with an external device through wireless communication or wired communication. Examples of wireless communication are LAN (Local Area Network), Bluetooth (registered trademark), Wifi, etc. Examples of external devices are the terminal 10, the terminal 20, and the terminal 30. The communication unit 52 sends the action data output from the control unit 54 to the terminal 10, the terminal 20, and the terminal 30.

[0143] The operation device 50 includes an input unit 53. The input unit 53 is a part that inputs the operation of the user U1, U2, U3 on the operation surface 51A. The input unit 53 detects the operation (gesture) of the operation surface 51A performed by the user U1, U2, U3. Examples of the operation include click, tap, slide, scroll, etc. The input unit 53 outputs the operation signal to the control unit 54.

[0144] The operating device 50 includes a control unit 54. The control unit 54 generates motion data based on the operation signal of the input unit 53. The control unit 54 outputs the motion data to the communication unit 52.

[0145] The control unit 54 may be configured using a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a clock, and a built-in memory. The control unit 54 may also be configured as a single piece of hardware (SoC: System On a Chip) in which a processor, a clock, a built-in memory, and the communication unit 52 are integrated.

[0146] The operating device 50 includes a battery 55 . The battery 55 supplies power to each unit of the operating device 50 .

[0147] Next, the operation of the operating device 50 performed by the users U1, U2, and U3 in causing the ear animation E of the character C to move in a manner such as "nodding," "agreeing," "applauding (clapping)," "questioning," "raising hands," or "surprised" is described. That is, the relationship between the movement of the ear animation E of the character C and the movement of the users U1, U2, and U3 detected by the operating device 50 is described. The movement of the ear animation E is similar to (associated with) the movement of the finger F of the users U1, U2, and U3 used to move the ear animation E. For example, in the case of moving the ear animation E of "nodding," a movement similar to the nodding movement is performed on the operating surface 51A of the operating device 50 with the finger F. That is, the users U1, U2, and U3 can move the ear animation E by performing a movement similar to the ear animation E that they want to move at the operating device 50.

[0148] Figure 5 : is a diagram showing an example of the movement of the ear animation E. Figure 5 Mode A shows the general situation of ear animation E.

[0149] Figure 5 Mode B of FIG. 4 shows the “nodding” action of the ear animation E. Figure 5 As shown in the pattern B of , for the "nod" action, in the ear animation E of the character C, the upper 1 / 2 of the ear is bent forward. In order to make the ear animation E perform the "nod" action, the user U1, U2, U3 touches the operation surface 51A of the main body 51 of the operation device 50 with two fingers F (for example, the index finger and the middle finger). Alternatively, the user U1, U2, U3 slides the two fingers F while the two fingers F are touching the operation surface 51A. The sliding amount of the fingers F is, for example, 1 cm to 2 cm.

[0150] When the above-mentioned action is performed, the operation device 50 detects the action through the input unit 53 and transmits the action as action data to the terminals 10, 20, and 30. The control unit 12 of the terminals 10, 20, and 30 generates control data for making the ear animation E move in a "nodding" manner based on the action data.

[0151] Figure 5The "agree" action of the ear animation E is shown in mode C. For the "agree" action, in the ear animation E of the character C, the ear is bent forward as a whole. In order to make the ear animation E perform the "agree" action, the users U1, U2, and U3 slide two fingers F (for example, the index finger and the middle finger) while touching the operation surface 51A. The sliding amount of the fingers F is, for example, longer than 2 cm, for example, 3 cm to 4 cm. That is, in order to make the ear animation E perform the "agree" action, the users U1, U2, and U3 slide the fingers F longer than the sliding amount of the fingers F when making the ear animation E perform the "nod" action.

[0152] When the above-mentioned action is performed, the operation device 50 detects the action through the input unit 53 and transmits the action as action data to the terminals 10, 20, 30. The control unit 12 of the terminals 10, 20, 30 generates control data for causing the ear animation E to move in an "agree" manner based on the action data.

[0153] Figure 5 The "clapping (appreciation)" action of the ear animation E is shown in mode D. For the "clapping (appreciation)" action, the ears of the character C are sometimes brought close together and sometimes separated in the ear animation E. In order to make the ear animation E perform the "clapping (appreciation)" action, the users U1, U2, and U3 make two fingers F (for example, the thumb and index finger, or the index finger and middle finger) touch the operation surface 51A and make the two fingers F sometimes approach (contact) and sometimes separate.

[0154] When the above-mentioned action is performed, the operating device 50 detects the action through the input unit 53 and sends the action to the terminal 10, 20, 30 in the form of action data. The control unit 12 of the terminal 10, 20, 30 generates control data for making the ear animation E move in a "clapping (appreciation)" manner based on the action data. The control unit 12 of the terminal 10, 20, 30 may also generate control data in the following manner: clapping multiple times with respect to the action data of one action of bringing the two fingers F closer and farther apart.

[0155] Figure 5The "questioning" action of the ear animation E is shown in mode E. For the "questioning" action, in the ear animation E of the character C, the ears are entangled. In order to make the ear animation E perform the "questioning" action, the users U1, U2, and U3 slide two fingers F (for example, the index finger and the middle finger) in a manner of drawing a circle (semicircle) while touching the operation surface 51A. It should be noted that in order to make the ear animation E perform the "questioning" action, the users U1, U2, and U3 may slide two fingers F (for example, the index finger and the middle finger) in a manner of drawing ∞ (infinity) while touching the operation surface 51A.

[0156] When the above-mentioned action is performed, the operation device 50 detects the action through the input unit 53 and transmits the action as action data to the terminals 10, 20, and 30. The control unit 12 of the terminals 10, 20, and 30 generates control data for making the ear animation E move in a "questioning" manner based on the action data.

[0157] Figure 5 The "hand raising" action of the ear animation E is shown in the mode F of FIG. For the "hand raising" action, the ear of the character C is stretched in the ear animation E. In order to make the ear animation E perform the "hand raising" action, the users U1, U2, and U3 slide only one finger F (middle finger) upward while touching the operation surface 51A with two fingers F (for example, the index finger and the middle finger).

[0158] When the above action is performed, the operation device 50 detects the action through the input unit 53 and sends the action as action data to the terminals 10, 20, and 30. The control unit 12 of the terminals 10, 20, and 30 generates control data for making the ear animation E move in a "hand-raising" manner based on the action data.

[0159] Figure 5 The "surprise" action of the ear animation E is shown in the mode G of FIG. For the "surprise" action, the top of the ear becomes larger in the ear animation E of the character C. At this time, the body B of the character C bends backward. In order to make the ear animation E perform the "surprise" action, the users U1, U2, and U3 slide two fingers F (for example, the index finger and the middle finger) upward while separating them while touching the operation surface 51A.

[0160] When the above action is performed, the operation device 50 detects the action through the input unit 53 and transmits the action as action data to the terminals 10, 20, 30. The control unit 12 of the terminals 10, 20, 30 generates control data for making the ear animation E move in a "surprised" manner based on the action data.

[0161] <How to operate the ear animation>

[0162] Next, the method of operating the ear animation E of the character C is described. When operating the ear animation E, the users U1, U2, and U3 perform, for example, Figure 5 Mode B Figure 5 Mode C, Figure 5 Mode D, Figure 5 Mode E, Figure 5 Mode F or Figure 5 The operation device 50 generates the operation data in the operation device 50. The operation device 50 transmits the operation data to the terminals 10, 20, and 30.

[0163] When the terminal 10, 20, 30 receives the motion data, the terminal 10, 20, 30 generates control data corresponding to the motion data. As a result, in the virtual space R of the terminal 10, 20, 30, the ear animation E of the character C performs any one of the actions of "nodding", "agreeing", "appreciating (clapping)", "questioning", "raising hands" or "surprised". In this way, the action of the ear animation E of the character C is operated by the operation of the operating device 50 performed by the users U1, U2, U3.

[0164] <Activities in Virtual Space>

[0165] Next, the activities of the character C in the virtual space R displayed based on the image data generated by the control unit 12 are described. The activities of the character C are performed in the virtual space R. In the virtual space R, the characters C (users U1, U2, and U3) can have conversations or communicate with each other. The character C and content information based on the content of the activities of the character C are displayed in the virtual space R. In the virtual space R, the screen changes based on the actions of the character C. That is, in the activities in the virtual space, the computer is caused to execute the following steps: a step of displaying a first screen including the character C and the content information based on the content of the activities; and a step of changing the display to a second screen different from the first screen according to the actions of the character C.

[0166] Examples of the content of the character C in the virtual space R include: lectures, presentations, training, meetings, reports, consultations, negotiations, entertainment, sightseeing, and video production. In the system 100, an application (software) may be configured for each content, or a plurality of contents may be included in one application. When a plurality of contents are included in one application, a screen for selecting an activity may be displayed on the output unit 15 of the terminal 10, 20, or 30. For example, Figure 6As shown, a selection button is displayed on the selection screen G1 for each of a plurality of activities (activity 1, activity 2, activity 3, activity 4, activity 5, and activity 6). Activity 1, activity 2, activity 3, activity 4, activity 5, and activity 6 correspond to the activities described later.

[0167] (Activity 1)

[0168] Activity 1 is a lecture, training session, or presentation. The following uses a lecture as an example. For a lecture, roles C of professors (teachers, lecturers) and students (apprentices) can participate. Figure 7 As shown, on a screen G2 showing an image of a virtual space R for a lecture, a host window (display area) W1, a presenter window (display area) W2, a participant window (display area) W3, and a material window (display area) W4 are displayed.

[0169] The host window W1 displays the character C who is the host of the lecture. The host window W1 is displayed, for example, at the upper left side of the screen G2. In the lecture, the professor can be equivalent to the host (leader). In the host window W1, the character C is displayed facing the front. That is, the character C in the host window W1 is displayed face to face with the users U1, U2, and U3 of the terminals 10, 20, and 30. The camera images (images) of the users U1, U2, and U3 who are the hosts can also be displayed in the host window W1. The users U1, U2, and U3 can switch between displaying the character C and displaying the camera images.

[0170] The character C who becomes the lecturer is displayed in the lecturer window W2. The lecturer window W2 is displayed, for example, in the upper right part of the screen G2. In a lecture, a professor or a student can be equivalent to a lecturer (leader). That is, in a lecture, a professor can be both a host and a lecturer. In the lecturer window W2, the character C is displayed facing the front. The camera images (images) of the users U1, U2, and U3 who become lecturers can also be displayed in the lecturer window W2. The users U1, U2, and U3 can switch between displaying the character C and displaying the camera images.

[0171] The character C who is a participant of the lecture is displayed in the participant window W3. The participant window W3 is displayed, for example, in the lower part of the screen G2 (lower part of the host window W1 and the lecturer window W2). In the lecture, students can be equivalent to participants. The seat (position) of the character C in the participant window W3 can be a reserved seat or a non-reserved seat. The seat of the character C can be allocated in the order of entering the participant window W3 based on a predetermined pattern.

[0172] In the participant window W3, the character C is displayed facing the back. It should be noted that in the terminals 10, 20, 30 of the host and the lecturer, the character C in the participant window W3 can also be displayed facing the front. The voice of the character C displayed in the participant window W3 is turned off. That is, the output of the voice of the character C displayed in the participant window W3 in the virtual space R is restricted. Thereby, for example, the progress of the teaching can be prevented from being hindered by the voice of the students. The voice of the character C can be the voice of the users U1, U2, U3, or it can be the voice after the voice of the users U1, U2, U3 is transformed (voice after voice change). That is, it can also have a voice transformation function. When a specified operation is accepted from the users U1, U2, U3, the orientation of the front or back of multiple characters C can be changed and displayed together.

[0173] When the number of displayed characters C is large (when the number of participating students is large), a scroll bar SB is displayed in the participant window W3. By operating the scroll bar SB, all participating student characters C can be confirmed in the participant window W3.

[0174] Identification information may be displayed in the character C displayed in the participant window W3. The identification information may be, for example, a student number, a name, an initial, etc. The character C in the participant window W3 is displayed smaller than the character C in the host window W1 and the lecturer window W2. In other words, the character C in the host window W1 and the lecturer window W2 is displayed larger than the character C in the participant window W3.

[0175] The document window W4 displays the document, etc. The document, etc. is content information based on the content of the activity of the character C. That is, the document, etc. displayed in the document window W4 during the lecture indicates the content (inclusive information) of the lecture in the listening activity of the lecture of the character C. The document window W4 is displayed, for example, in the center of the upper part of the screen G2 (between the host window W1 and the lecturer window W2).

[0176] The host and the lecturer can operate the document window W4. The host and the lecturer can operate the display content displayed in the document window W4 in a predetermined operation screen. The operation screen is displayed on the output unit 15 of the host or the lecturer, for example, when the host or the lecturer taps (clicks) the document window W4. In the operation screen, the document (text data, image data, presentation data, etc.) can be displayed in the document window W4, or handwriting can be input to the displayed document or board.

[0177] The arrangement and size of the host window W1, the lecturer window W2, the participant window W3, and the document window W4 can be arbitrarily set by the administrator. In the case of a lecture, the administrator is, for example, a lecturer.

[0178] like Figure 8 As shown, in the participant window W3, the character C can also be displayed so as to be distinguishable by the attributes of the participants (users U1, U2, U3). In this way, the attributes of the users U1, U2, U3 participating in the event can be confirmed (understood) at a glance. Figure 8 In the example shown, the character C is distinguished into five attributes. Specifically, the attributes can be distinguished by the line type of the hatching, for example. The line type is solid line (thin line), solid line (thick line), dotted line, single-dash line, double-dash line. In the case of lectures, the attributes can be, for example, departments, subjects, etc. Specifically, for example, the Department of Literature, Department of Engineering, Department of Science, etc. The attributes are set based on the pre-set information of users U1, U2, and U3. It should be noted that in Figure 8 In the example shown, the attributes of the users U1 , U2 , and U3 of the character C are distinguished by hatching line types, but the attributes of the users U1 , U2 , and U3 of the character C may be shown by color types.

[0179] like Fig. 9 As shown in FIG. 1 , the display size of the character C displayed in the participant window W3 can be changed. Fig. 9 In the example shown, the size of character C is displayed as Figure 8 The example shown is larger. In the participant window W3, the size of the character C displayed on the front row side (upper side in the figure) can also be increased, and the size of the character C displayed on the back row side (lower side in the figure) can be reduced. By changing the size of the character C, the number of characters C displayed in the participant window W3 can be set without scrolling. When the size of the character C is enlarged, the number of characters C displayed in the participant window W3 is reduced, and when the size of the character C is reduced, the number of characters C displayed in the participant window W3 is increased. Such settings can be set arbitrarily by the administrator. The administrator can set it in the specified setting screen.

[0180] like Fig.10 As shown, by dragging and dropping the character C displayed in the lecturer window W2 to the data window W4, the character C displayed in the lecturer window W2 can be displayed in (moved to) the data window W4. The host and the lecturer can drag and drop the character C. The lecturer's identification information is displayed in the character C dragged and dropped from the lecturer window W2 to the data window W4. The identification information can be, for example, the student number, name, initials, etc. Fig.10 In the example shown, "AAA" is displayed on the body B of the character C.

[0181] In addition, the character C displayed in the host window W1 can be displayed in the data window W4 by dragging and dropping the character C displayed in the host window W1 to the data window W4. For example, the host character C can be displayed in the data window W4 before the lecture starts, and when the lecture starts, the host character C can be moved to the host window W1.

[0182] like Fig.11 As shown, when the ear animations E of multiple characters C raise their hands in response to questions from the host and / or lecturer, the hand raiser window (display area) W5 is displayed instead of the data window W4. In the hand raiser window W5, the characters C whose ear animations E raise their hands are displayed in the order in which the actions were performed. In the hand raiser window W5, for example, the characters C who raise their hands quickly are displayed in sequence from left to right. It should be noted that, in the case where the character C moves from the participant window W3 to the hand raiser window W5, a prescribed mark (identification information, etc.) can also be displayed at the place where the seat becomes empty. Thereby, it is possible to prevent other characters C from sitting (being located) in the part of the participant window W3 that becomes an empty seat due to the character C moving to the hand raiser window W5.

[0183] Furthermore, when the ear animation E of the character C performs a hand-raising motion, a speaker window (display area) W6 is displayed. The speaker window W6 is displayed below the moderator window W1 (between the moderator window W1 and the participant window W3).

[0184] like Fig.12 As shown, the character C displayed in the hand-raiser window W5 can be moved to the speaker window W6. The movement of the character C can be performed by clicking or dragging and dropping the character C. The host and the speaker can click or drag and drop the character C. In the case where the character C is displayed in the speaker window W6, the information of the user U1, U2, U3 of the character C can also be displayed on the body B of the character C. When the character C is displayed in the speaker window W6, the voice of the character C is automatically turned on. As a result, the speech content of the user U1, U2, U3 of the character C is transmitted to other users U1, U2, U3 participating in the lecture.

[0185] When the ear animations E of multiple characters C raise their hands in response to questions from the host and / or speaker, only the speaker window W6 is displayed. The character C that has raised its hand can be moved to the speaker window W6. The character C can be moved by clicking or dragging and dropping the character C. The host and speaker can click or drag and drop the character C.

[0186] When the speech of the character C in the speaker window W6 is finished, the character C is moved to the participant window W3. The movement of the character C can be performed by clicking or dragging the character C. The host and the speaker can click or drag the character C. The character C returns to the predetermined position of the participant window W3. As a result, the voice of the character C is automatically turned off.

[0187] like Fig.13 As shown, the character C displayed in the participant window W3 can be moved to the breakout rooms (display areas) BR1, BR2, BR3, and BR4. A plurality of breakout rooms BR1, BR2, BR3, and BR4 can be set according to the number of characters C displayed in the participant window W3 (the number of students participating in the lecture).

[0188] The host or lecturer can arbitrarily assign the role C to the breakout rooms BR1, BR2, BR3, and BR4. The assignment of the role C to the breakout rooms BR1, BR2, BR3, and BR4 can also be performed automatically based on the attributes of the role C. The movement to the breakout rooms BR1, BR2, BR3, and BR4 is performed according to the operation of the host or lecturer. That is, when the host or lecturer performs an operation (input) to instruct the movement to the breakout room BR, the role C displayed in the participant window W3 moves to each breakout room BR1, BR2, BR3, and BR4.

[0189] The document window W7 is displayed on the screen G3 displaying the images of the group discussion rooms BR1, BR2, BR3, and BR4. Documents, etc. are displayed in the document window W7. The output unit 15 of the terminal 10, 20, or 30 of the student who is attending the lecture displays Fig.13 The screen shown. That is, only the group discussion room BR1, BR2, BR3, BR4 to which the student is assigned is displayed on the student's terminal 10, 20, 30. In each group discussion room BR1, BR2, BR3, BR4, the voice of the character C displayed in other group discussion rooms BR1, BR2, BR3, BR4 cannot be heard by each other. That is, in each group discussion room BR1, BR2, BR3, BR4, the character C can have a conversation in each room.

[0190] like Fig.14 As shown in FIG. 1 , the host or lecturer can confirm multiple breakout rooms BR1, BR2, BR3, and BR4. Fig.14 In the example shown, four breakout rooms BR1, BR2, BR3, and BR4 are shown. By clicking one of the breakout rooms BR1, BR2, BR3, and BR4, the host or lecturer can Fig.13 As shown, breakout rooms BR1, BR2, BR3, and BR4 are displayed.

[0191] In a lecture, in response to inquiries from the host or speaker about understanding, students can Fig.15 Mode A, Fig.15 Mode B and Fig.15 The ear animation E shown in mode C reacts. Fig.15 Mode A of FIG. 1 shows the "fully understood" action of ear animation E. Fig.15 Mode B of shows the "barely understand" action of ear animation E. Fig.15 Mode C of FIG. 1 shows the “don’t understand” action of the ear animation E. The “don’t understand” action of the ear animation E is similar to Figure 5 In response to the inquiry about the understanding degree from the host or lecturer, the users U1, U2, and U3 operate the operating device 50 to make the ear animation E of the character C move. It should be noted that Fig.15 Mode A, Fig.15 Mode B and Fig.15 The movement of the ear animation E shown in the pattern C can be controlled by a predetermined operation on the operating device 50, or by an operation of a button displayed on the screen by the participant.

[0192] The reaction of the participant, that is, the movement of the ear animation E of the character C can be displayed visually. Fig.15 As shown, the statistics of the movement of the ear animation E of the character C are displayed as a bar graph, for example. This allows the host or lecturer to understand the degree of understanding of the students in the lecture.

[0193] Next, an example of a method for participating in the above-mentioned lecture is described. First, the students (users U1, U2, U3) as participants open the school's homepage on the terminal 10, 20, 30. Then, the students enter their ID and password on the homepage to log in. After logging in, the lecture schedule is displayed on a specified screen. When it is time for the lecture, the application is started, and the characters C of the users U1, U2, and U3 are displayed in the participant window W3. Thus, the students complete their participation in the lecture.

[0194] The activities of the character C in the virtual space R (such as attending a lecture) can be recorded. The recording can be performed by the administrator or by all participants. It can also be set that when the administrator records the video, the recorded video is made public to the participants. Users U1, U2, and U3 can review the lecture through the recorded video.

[0195] (Activity 2)

[0196] Next, activity 2 is described. Activity 2 is a meeting, a presentation, etc. First, a meeting is used as an example for description. The meeting may be a brainstorming session.

[0197] like Fig.16 As shown, a document window (display unit) W9 is displayed on a screen G4 showing an image of a virtual space R of a conference. The document window W9 is displayed in the center of the screen G4. The character C is displayed across the document window W9. Fig.16 In the example shown, six characters C are displayed at positions opposite to each other across the data window W9. For example, all characters C are displayed in the same size. The size of the character C can be set arbitrarily by the manager and the participants. In the case of a meeting, the manager is, for example, the host. It is also possible that when the manager or the participant changes the size of one character C, the sizes of all characters C are changed accordingly. Of course, it is also possible to change only the size of a specific character C.

[0198] The position of character C can be set arbitrarily by the users U1, U2, and U3 attending the meeting, or can be set in advance by the organizer. If the position of character C is not set (in the case of free seats), the position of character C can be assigned according to the order of attending the meeting. The position of character C can be changed appropriately.

[0199] The document window W9 displays the document, etc. The document, etc. is content information based on the content of the activity of the character C. That is, the document, etc. displayed in the document window W9 during the meeting indicates the content (information included) of the meeting in the activity of the character C participating in the meeting. The conversation of the character C may also be displayed in the document window W9 as a text. In addition, the translation of the conversation of the character C may also be displayed in the document window W9.

[0200] The participant can operate the data window W9. The participant can operate the display content displayed in the data window W9 in a prescribed operation screen. The operation screen is displayed on the output unit 15 of the participant, for example, by the participant tapping (clicking) the data window W9. In the operation screen, data (text data, image data, demonstration data, etc.) can be displayed in the data window W9, or input into the displayed data or board by handwriting. It can also be set that: in the case of an explanation using a pointer, the character C automatically rotates to face the position indicated by the pointer (prescribed object). That is, the character C can also be rotated according to the position of the prescribed object to imagine that the character C is looking at the prescribed object. In addition, it can also be set that: in the case of talking to a specific character C, by clicking on the character C, the character C of the dialogue object and the character C (prescribed object) of the dialogue object are facing each other.

[0201] The document window W9 can be created by the users U1, U2, and U3 set in advance such as the administrator and all participants. For example, the document window W9 can be created by clicking the document window in the task bar (not shown). The document window W9 can also be ended by an end button (not shown).

[0202] The configuration and size of the document window W9 can be set arbitrarily by the administrator. In the case of a meeting, the administrator is, for example, the host. In accordance with the change in the size of the document window W9, the size of the character C is also changed. For example, when the size of the document window W9 is enlarged, the size of the character C is reduced. The character C is displayed in the outer area (blank) of the document window W9 in the screen G4. When the size of the document window W9 is enlarged to the maximum, the size of the character C can be reduced to about 10 mm. The document window W9 can also be displayed full screen in the screen G4. In this case, the character C is displayed overlapping the document window W9. In addition, each user U1, U2, and U3 can arbitrarily change the display viewpoint. For example, the viewpoint can be rotated in 3D.

[0203] like Fig.17 As shown, when a participant fills in text in the data window W9, it can be displayed to know which participant filled in the text. Specifically, it can be set to display a pointer corresponding to the character C in the data window W9 so that it can be known at a glance that the character C filled in the data window W9 corresponds to the pointer P. Specifically, for example, a hatch can be displayed on the pointer P and the character C, or the pointer P and the character C can be displayed in the same color.

[0204] In addition, if Fig.18 As shown, the body B of the participant's character C may be transformed into a data window CW. Thus, the character C filled in the data window CW can be recognized at a glance. The switching of the display from the data window W9 to the data window CW can be performed by dragging and dropping the character C to the data window W9. That is, when the operation of dragging and dropping the character C to the data window W9 is accepted, the predetermined function of switching the display is performed in the data window W9. The data of the users U1, U2, U3 of the dragged character C, etc., or the content input by the users U1, U2, U3 is displayed in the data window CW.

[0205] The data window CW may also be displayed to show which participant it is (who is the operator). Specifically, for example, in order to know the participant at a glance, the ear animation E of the participant's character C or a similar design thereof may be set outside or inside the window, or a frame color or frame pattern may be set. At this time, it may also be preset that not only an ear animation E of the same color, same pattern, and a design with a similar association is added to the outside of the window, but also the color, pattern, etc. of the frame of the corresponding window is set to a thickness to be consistent with the color and pattern of the character C and to establish a similar relationship. The frame may not be a complete quadrilateral, but may be designed with slightly rounded corners. In addition, a pointer P may be displayed in the data window CW (see Fig.17). Similarly, the color, frame, periphery, and background of the pointer P can be set in advance to be the same as the color of the ear animation E of the character C, or the same system color.

[0206] The switching of the display from the document window CW to the document window W9 can be performed by dragging and dropping (dragging in) the character C displayed in the document window CW to an area outside the document window CW. That is, after the character C is dragged and dropped to the document window W9, when the operation of dragging and dropping the character C displayed in the document window CW to an area outside the document window CW is accepted, the execution of the specified function of switching the display from the document window W9 to the document window CW is terminated. In addition, when another character C is dragged and dropped (draggled in) to the document window CW, the display can be changed to the document window CW of another character C. In addition, the display can be switched from the document window CW to the document window W9 by clicking the exit button (not shown).

[0207] It should be noted that the deformation of the body B of the character C can also have multiple modes. For example, the body B can be deformed in a way that the display area becomes larger in stages. Users U1, U2, and U3 can change the deformation mode of the body B according to the situation, etc. In addition, there is no limitation on the data displayed in the data window, and it can also be applied to the pen input window, image window, etc. as information display. In particular, for pen input, when the character C to be replaced is dragged and dropped into the pen input window during the replacement of the pen input operator, if it is pre-set to continue the state of the previous pen input window for additional writing, the pen input replacement can be smoothly performed.

[0208] like Fig.19 As shown, a data window W10 can be displayed in a screen G5 showing an image of a virtual space R of a conference, and a data window CW after the body B of a participant's character C is transformed can be displayed. As described above, the data window CW can be displayed by dragging and dropping the character C to a data window (not shown). The switching of the display from the data window CW to the data window (not shown) can be performed by dragging and dropping (drag-out) the character C displayed in the data window CW to an area outside the data window CW.

[0209] The data window W10 is displayed, for example, in the upper center of the screen G5. A tab can also be set in the data window W10. A tab can be added by clicking the "+" displayed in the upper right corner of the data window W10. Fig.19 In the example shown, three ("1", "2", "3") tabs are displayed. By clicking on the tabs, the display of the data in the data window W10 can be switched. The data window CW composed of the body B of the character C is displayed in the lower center of the screen G5 (below the data window W10).

[0210] In the data window W10, tabs may be associated with the characters C. For example, tab "1" is associated with the character C whose body B displays "1", tab "2" is associated with the character C whose body B displays "2", and tab "3" is associated with the character C whose body B displays "3". Users U1, U2, and U3 of the characters C can display any information in their tabs' data windows W10.

[0211] In addition, in the data window CW, the text of the conversation of the character C can also be displayed. In the case of different languages, the text can be displayed in the language specified by each participant in conjunction with the automatic interpretation function. In this way, non-verbal information using the ear animation E and the translated text between different languages ​​are displayed. Therefore, the automatic interpretation conversation with the visual support of the status, feelings, and words (translated text) of the users U1, U2, and U3 can be realized.

[0212] The conversation text can be dragged and dropped to the data window W10 at will. The dragged and dropped conversation text is displayed in the window of the tab corresponding to the character C. It should be noted that the textualization of the conversation of the character C can also be set not to display the specified words. The specified words are, for example, "um, um", "ah", "I see", "yes, yes", etc. It can also be set that: when it is set not to display the specified words, the ear animation E of the character C is moved by the specified words. In addition, the ear animation E of the character C constituting the data window CW can also be extended according to the volume of the character C's voice.

[0213] In addition, the following function may be provided: a real image window (not shown) is displayed in the virtual space R, and when the character C is dragged and dropped into the real image window, the character C becomes the real image (video) of the users U1, U2, and U3. It may also be provided that when the real image is closed, the display of the character C is restored. It should be noted that, preferably, in the case of switching to the real image, the image to be displayed is confirmed (approved) in advance. That is, an approval step for the screen transition may be provided.

[0214] The above series of operations and scene transitions can also be achieved by clicking on the function switch in the character C. By setting the "movement process" of the character C itself realized by dragging and the "transformation process" of the character C realized by clicking on the switch of the character C, a "state transition" can be performed, thereby realizing the scene transition realized by sensory and intuitive operations with a sense of time. The sensory and intuitive mentioned here have the following characteristics: the operation can be switched to the next scene only by the target movement of the character C realized by the user U1, U2, and U3 themselves and the operation of the character C itself, rather than by the operation of the screen operation panel from the title bar at the top of the screen, which is greatly different from the displayed character operation. For example, this is the same as the difference between moving to the recycling bin of waste on the intuitive screen and clicking on the non-intuitive title bar to operate the delete key. In the past, file operations were used as the object, but in this embodiment, intuitive operations of the character C appearing in remote conferences and metaverses are provided. Moreover, it is also characterized in that the transformation is performed while maintaining the ear animation E expressed by non-verbal information.

[0215] Next, we will use a report meeting as an example to explain. Fig. 20 As shown, a participant window W11 and a data window W12 are displayed in a screen G6 showing an image of a virtual space R of a lecture. A character C as a participant of the lecture is displayed in the participant window W11. The participant window W11 is displayed, for example, at the lower portion of the screen G6. In the participant window W11, the character C is displayed facing backward. The voice of the character C displayed in the participant window W11 is muted. When a large number of characters C are displayed (when a large number of participants are present), a scroll bar SB is displayed in the participant window W11. By operating the scroll bar SB, all participating characters C can be confirmed in the participant window W11. Identification information may also be displayed in the character C displayed in the participant window W11. The identification information may be, for example, a position, a name, an initial, and the like.

[0216] The data etc. are displayed in the data window W12. The data etc. are content information based on the content of the activity of the character C. That is, the data etc. displayed in the data window W12 during the report meeting show the content (included information) of the report meeting when the character C participates in the report meeting. The data window W12 is displayed at the top (above the participant window W11) of the screen G6, for example. The character C of the reporter participating in the report meeting is displayed on the right side of the data window W12 in the screen G6. It should be noted that the reporter's character C can also be displayed on the data window W12 (overlapping with the data window W12).

[0217] The reporter can operate the document window W12. The reporter can operate the display content displayed in the document window W12 in a prescribed operation screen. The operation screen is displayed on the reporter's output unit 15, for example, by the reporter tapping (clicking) the document window W12. It should be noted that it can also be set as follows: the participant is given the operation authority of the document window W12 by dragging and dropping the participant's character C to the document window W12. In this case, when the character C moves from the participant window W11 to the document window W12, a prescribed mark (identification information, etc.) can also be displayed at the place where the empty seat becomes. In this way, it can be avoided that other characters C sit (are located) in the part of the participant window W11 that becomes an empty seat due to the movement of the character C to the document window W12.

[0218] The arrangement and size of the participant window W11 and the document window W12 can be arbitrarily set by the manager. In the case of a lecture, the manager is, for example, the host.

[0219] In the participant window W11, the character C can also be displayed so as to be distinguishable by the attributes of the participants (users U1, U2, U3). For example, the attributes can be distinguished by the line type or color of the hatching. The display size of the character C displayed in the participant window W11 can be changed. In the participant window W11, for example, the size of the character C displayed on the front row side (the upper side in the figure) can be increased, and the size of the character C displayed on the back row side (the lower side in the figure) can be reduced.

[0220] Next, an example of a method for participating in the above-mentioned conference or seminar is described. The organizer downloads the application. When the application is opened, Figure 6 As shown, the selection screen G1 is displayed. When the organizer selects "Event 2" in the selection screen G1, a selection screen for "Meeting" or "Lecture Session" is displayed. The organizer selects "Meeting" or "Lecture Session", further selects participants, and a notification is sent to the participants.

[0221] The notification may include a description of the meeting, ID, and password. In addition, a URL or calendar may be displayed in the notification. The meeting can be attended by clicking on the URL or calendar. It should be noted that the notification may also include a description of supporting the download of an application for users who are attending the meeting for the first time. When the time for the meeting comes, the participant attends the meeting by inputting an ID, password, clicking on a URL, etc.

[0222] The activities (meetings, lectures, etc.) of the character C in the virtual space R can be recorded. The recording can be done by the administrator or by all participants. It can also be set that when the administrator records the video, the recorded video is made public to the participants. Users U1, U2, and U3 can review the meeting, etc. through the recorded video.

[0223] (Activity 3)

[0224] Next, activity 3 is described. Activity 3 is a one-on-one activity of role C, which is consultation, call center support, explanation conversation, prior consultation, telemedicine, private lessons, etc. In this embodiment, activity 3 is described as an example of consultation. For consultation, the role C of the consultant and the consultant (client) can participate. Fig.21 As shown, a screen G7 showing an image of a virtual space R for consultation displays: a display board (content information display unit) B1, a chat board (text display unit) B2, a character C, and a table T.

[0225] The consultation content, etc. are displayed on the display board B1. The consultation content, etc. are content information based on the content of the activity of the character C. That is, the consultation content, etc. displayed on the display board B1 during the consultation represents the content (included information) of the consultation during the consultation activity of the character C. The display board B1 is displayed at the top of the screen G7, for example. A tab can also be set on the display board B1. A tab can be added by clicking the "+" displayed on the upper right of the display board B1. Fig.21 In the example shown, three ("1", "2", and "3") tabs are displayed. The display of the materials and the like in the display panel B1 can be switched by clicking the tabs.

[0226] The consultant can operate the display panel B1. The consultant can operate the display content displayed on the display panel B1 on a predetermined operation screen. The operation screen is displayed on the consultant's output unit 15, for example, when the consultant taps (clicks) the display panel B1. In the operation screen, data can be displayed on the display panel B1 or input by handwriting.

[0227] In the chat board B2, a conversation text showing the textualized content of the conversation of the character C is displayed in the form of a chat. The conversation text is obtained by converting the speech of the users U1, U2, and U3 into text through speech recognition processing. A predetermined number of conversation texts are displayed in a time series starting from the latest conversation text among a plurality of conversation texts produced based on the speech of the users U1, U2, and U3 in the chat board B2. Fig.21 In the example shown, three conversation texts are displayed. The chat board B2 is displayed, for example, in the center of the screen G7 (below the display board B1). In the chat board B2, a symbol of the character C may be displayed near the conversation texts.

[0228] It should be noted that in the chat board B2, not only the text converted into text by voice recognition processing can be displayed, but also the text input by the users U1, U2, and U3 from the keyboard, etc. can be displayed. In the chat board B2, the display of the input text is effective in situations where you want to avoid voice speaking. For example, when leaving or withdrawing from a meeting, by inputting the meaning and displaying it on the chat board B2, it is possible to reduce interference with the current conversation and convey the status of leaving or withdrawing to the participants. Of course, it can also be used for a wide range of other purposes. For example, it can also be applied to scenes for conveying cautious comments, etc. It can also be pre-set that the chat board B2 can respond to each type separately through the sound on and off function, the selection of sound content, the automatic generation of standard types, etc., according to the conditions of providing silent scenes, providing scenes with sound, and providing effect sounds.

[0229] The character C is displayed at the bottom of the screen G7 (below the chat board B2), for example. For example, the counselor's character C is displayed on the left, and the consultant's character C is displayed on the right. The characters C are arranged to face each other with the table T interposed therebetween at an angle.

[0230] like Fig. 22 As shown, the configuration of character C can be set arbitrarily. Specifically, the distance L between character C and character C, the face-to-face angle θ of character C, and the rotation angle of character C can be set. The face-to-face angle θ of character C can be changed by adjusting the angle of table T. It can also be set that: when the configuration of one character C is changed, the configuration of other characters C is also changed in conjunction (together). In addition, the size of character C can also be set arbitrarily. The size of character C can be set separately for two characters C. In addition, it can also be set that: when the size of one character C is changed, the size of another character C is also changed in conjunction. In addition, each user U1, U2, and U3 can arbitrarily change the display viewpoint. For example, the viewpoint can be rotated in 3D.

[0231] like Fig.23 As shown, when an operation from users U1, U2, and U3 on the conversation text displayed on the chat board B2 is accepted, the conversation text is displayed on the display board B1. For example, when users U1, U2, and U3 click on the conversation text displayed on the chat board B2, the clicked conversation text is displayed on the display board B1. It should be noted that users U1, U2, and U3 can also drag and drop the conversation text displayed on the chat board B2 to the display board B1.

[0232] like Fig.24 As shown, a scroll bar SB may be provided on the display panel B1. Thus, a large amount of information can be displayed on the display panel B1. Fig.25 As shown, the display panel B1 can also be displayed on the table T. Fig.26As shown, the conversation text may also be displayed in the body B of the character C. It should be noted that the consultant can arbitrarily set the background of the character C. The background may be selected and set from a plurality of images that are effective for consultation and stored in advance, or may be created by the consultant.

[0233] The activities of the character C in the virtual space R (participation in consultation) can be recorded. The recording can be done by the consultant or the consultant. When the consultant records the video, the recorded video can be made public to the consultant. Users U1, U2, and U3 can review the consultation through the recorded video.

[0234] In addition, a screen G7 showing an image of a virtual space R can also be produced based on the recorded (recorded) image (voice) of a real face-to-face consultation. In this case, the characters C of the consultant and the consultant are displayed in the screen G7. The ear animation E of the character C can also move according to the voice. The voice of the character C can be the voice of the user U1, U2, U3, or it can be the voice after the voice of the user U1, U2, U3 is transformed (voice after voice transformation). In addition, in the screen G7, based on the content of the recorded conversation between the consultant and the consultant, a conversation text representing the textualized conversation content of the character C is displayed in the chat board B2 in the form of a chat. The users U1, U2, U3 who watch the produced image can display the conversation text on the display board B1 by operating the conversation text displayed on the chat board B2.

[0235] The image of virtual space R is effective for retrospective observation after consultation, and it is easy to get new experience. In particular, the textual display of the conversation (discourse) in the chat board B2 not only acts on the auditory stimulation in the real conversation, but also acts on the visual stimulation, and it can be expected to bring experience more easily. In addition, since textualization can also be performed offline, there is no need to worry about delays that become problems in real-time display. In addition, by using character C, the consultant does not show his face (no camera is used), so it is also effective for privacy protection. Furthermore, if a voice changer is used to output voice, stronger privacy protection can be achieved. In the past, from the perspective of privacy protection, audio and video recording in consultation has not been widely used, but in this method, it is easy to review. In addition, it is also easy for the mentor (supervisor) who guides the consultant to work.

[0236] It should be noted that, although the action operation of the character C is not performed in real dialogue, the expression of the ear animation E caused by the voice can be automatically expressed. In addition, when silence continues for a certain period of time, the expression can also be emphasized by changing the color of the character C. In addition, for example, by combining with the measurement values ​​of the heart rate monitor installed on the wristband, the acceleration sensor of the body swing, etc., it is possible to link and analyze with quantitative and objective data.

[0237] (Activity 4)

[0238] Next, activity 4 is described. Activity 4 is, for example, entertainment. Examples of entertainment include music performance and games. First, music performance is described.

[0239] For music performance, the performer role C can participate. Fig. 27 As shown in FIG. 1 , the display panel B3 is displayed on the screen G8 showing the image of the virtual space R in which the music is played. Fig. 27 An example of a band performance is shown in FIG.

[0240] The display board B3 displays music scores and the like. The music scores and the like are content information based on the content of the activity of the character C. That is, the music scores and the like displayed on the display board B3 during the music performance represent the content (inclusive information) of the music performance during the activity of the character C participating in the music performance. The display board B3 is displayed in the center of the screen G8, for example. The configuration and size of the display board B3 can be set arbitrarily. In the display board B3, the music scores can also be changed (slid) and displayed according to the performance.

[0241] Character C is displayed around the display panel B3. Fig. 27 In the example shown, the body B of the character C becomes a guitar, drums, double bass, saxophone, and keyboard. That is, the user of the character C with the body B of the guitar plays the guitar. Similarly, the user of the character C with the body B of the drums plays the drums.

[0242] In accordance with the change in the size of the display board B3, the size of the character C is also changed. For example, when the size of the display board B3 is enlarged, the size of the character C is reduced. The character C is displayed in the outer area (blank) of the display board B3 in the screen G8. When the size of the display board B3 is enlarged to the maximum, the size of the character C can be reduced to about 10 mm. The display board B3 can also be displayed in full screen in the screen G8. In this case, the character C is displayed overlappingly on the display board B3.

[0243] The ear animation E of the character C can be animated according to the actions of the users U1, U2, and U3. The actions of the users U1, U2, and U3 can be detected by sensors, etc. The control unit 12 generates control data for the ear animation E based on the detected actions of the users U1, U2, and U3.

[0244] like Fig.28 As shown in FIG. 1 , the display panel B3 is displayed on the screen G9 showing the image of the virtual space R in which the music is played. Fig.28 An example of an orchestral performance is shown in .

[0245] The character C is displayed so as to surround the display panel B3. Fig.28In the example shown, the body B of character C becomes a violin, a clarinet, a double bass, and a flute. Fig.28 In the example shown, the character C as the conductor is displayed. The conducting motions of the users U1, U2, and U3 as the conductor can be detected by sensors or the like. The control unit 12 generates control data for the ear animation E based on the detected conducting motions of the users U1, U2, and U3. Thus, the ear animation E of the conductor character C conducts. Thus, the users U1, U2, and U3 of other characters C can perform while watching the conduct.

[0246] Next, the game is described. For the game, the player character C and the viewer character C of the game can participate. Fig.29 As shown in FIG. 1 , the display panel B4 is displayed on the screen G10 showing the image of the virtual space R of the game. Fig.29 An example of Go is shown in FIG.

[0247] A game board in the form of a Go board is displayed on the display board B4. The game board is content information based on the content of the activity of the character C. That is, the game board displayed on the display board B4 during the game indicates the content (information included) of the game in the activity of the character C participating in the game. The display board B4 is displayed in the center of the screen G10, for example.

[0248] The character C for the player is displayed across the display panel B4. The character C for the viewer can be displayed at the top of the screen G10. Fig.30 As shown, the display panel B4 can be changed in configuration and size. Fig.30 In the example shown, the display panel B4 is displayed in an enlarged manner. In accordance with the enlargement of the display panel B4, the size of the character C can be reduced.

[0249] The activities (music performance, game, etc.) of the character C in the virtual space R can be recorded. The recording can be done by the administrator or by all participants. It can also be set that when the administrator records the video, the recorded video is made public to the participants. Users U1, U2, and U3 can review the music performance, game, etc. through the recorded video.

[0250] (Activity 5)

[0251] Next, activity 5 is described. Activity 5 is sightseeing, appreciation (sports appreciation, movie appreciation, concert appreciation), etc. First, sightseeing is used as an example. For sightseeing, tourist role C can participate. Fig.31 As shown, a display panel B5 and a moving stage (moving object) S are displayed on a screen G11 showing an image of a virtual space R for sightseeing.

[0252] The display panel B5 displays images of tourist attractions, etc. The images of tourist attractions, etc. are content information based on the content of the activities of the character C. That is, the images of tourist attractions, etc. displayed on the display panel B5 during sightseeing show the content of sightseeing (inclusive information) in the sightseeing activities of the character C. The display panel B5 is displayed at the top of the screen G11, for example.

[0253] The organizer of the tour can operate the display panel B5. The organizer can operate the display content displayed on the display panel B5 in a predetermined operation screen. The operation screen is displayed on the output unit 15 of the organizer, for example, by the organizer tapping (clicking) the display panel B5. In the operation screen, an image can be displayed on the display panel B5, or handwriting can be input to the displayed image.

[0254] The arrangement and size of the display panel B5 can be arbitrarily set by the manager. In the case of sightseeing, the manager is, for example, the organizer. It is also possible to display information of participants, guide information, etc. on the manager's screen G11.

[0255] The character C is arranged on the mobile station S. The mobile station S is set to be movable. The mobile station S is operated by users U1, U2, and U3. That is, the users U1, U2, and U3 can move the mobile station S to any position. In addition, each user U1, U2, and U3 can arbitrarily change the display viewpoint. For example, the viewpoint can be rotated in 3D.

[0256] like Fig.32 As shown in FIG. 1 , a display panel B5 and a plurality of moving stations S are displayed on a screen G12 showing an image of a virtual space R for sightseeing. Fig.32 In the example shown, in addition to the tourist character C, the guide character CG can participate. Fig.32 In the example shown, the guide can provide explanations about the images displayed on the display panel B5. At this time, the guide's voice is transmitted to all the characters C in the virtual space R. In addition, the voices of one or more characters C configured on one mobile station S can be made inaudible to the voices of one or more characters C configured on other mobile stations S. The users U1, U2, and U3 who are guides control the on and off of the voices according to the situation.

[0257] like Fig.33 As shown, the display panel B5 can be displayed in full screen in the screen G12. In this case, the character C and the moving platform S are displayed overlappingly on the display panel B5. That is, the character C and the moving platform S are displayed on the image displayed on the display panel B5. In this case, sightseeing with a sense of presence can be realized.

[0258] Next, we will take the appreciation of a painting as an example. For the appreciation, the appreciator role C can participate. Fig.34As shown in FIG. 1 , the display panels B6, B7 and the moving table S are displayed on the screen G13 showing the image of the virtual space R to be appreciated. Fig.35 As shown, for example, when the display panel B6 is clicked, the display panel B8 is displayed. The display panel B8 displays an explanation of the painting displayed on the display panel B6.

[0259] (Activity 6)

[0260] Activity 6 is, for example, video production. For video production, in an activity in a virtual space, the ear animation E of the character C is output as an editable video. Fig.36 As shown, in the screen G14 showing the image of the created virtual space R, for example, a display board B9 and two characters C are shown. Various content information can be displayed on the display board B9. The character C uses a pointer to explain the content displayed on the display board B9. The pointer can be distinguished by color for each character C. In this way, the character C who uses the pointer to explain, etc. can be identified. In addition, the direction of the character C can also be automatically rotated corresponding to the display and position of the pointer. In this way, the sense of presence of the explanation is increased by rotating the character C in correspondence (linkage) with the movement of the pointer.

[0261] In the created video, the movement of the ear animation E of the character C, the position and size of the character C, the position and size of the display panel B9, and the content displayed on the display panel B9 can be edited. The created video can be uploaded to a video sharing service or the like.

[0262] [Effect]

[0263] As described above, in the system 100 of the present embodiment, when an operation related to the action of the character C is performed in the first screen including the character C and the content information based on the content of the activity, the display is changed to the second screen including the character C corresponding to the character C in the first screen. For example, for activity 1, when the ear animations E of multiple characters C perform a hand-raising action in response to the operation of the users U1, U2, and U3, the hand-raiser window W5 is displayed instead of the data window W4. In the hand-raiser window W5, the characters C whose ear animations E perform the hand-raising action are displayed in the order in which the actions were performed. In this way, in the system 100, the screen transition corresponding to the action of the character C that performs the action through the operation intended by the users U1, U2, and U3 can be performed. Therefore, in the system 100, when the character C performs an activity in the virtual space R, the convenience of the users U1, U2, and U3 who use the character C can be improved.

[0264] As for the operation related to the action of the character C, in addition to moving the character C of the user U1, U2, U3, or displaying the display unit on the character C, it can also include generating a display unit on other characters C (such as AI's character C) by inputting text (keyboard input, voice input, etc.) on the display unit of the character C. In addition, as for the ear animation E to move in response to the operation of the user U1, U2, U3, in addition to causing the ear animation E of the character C to move through the operation of the user U1, U2, U3, it can also include causing the ear animation E of other characters C to move (react, link) in response to the operation of the user U1, U2, U3. By inducing screen transitions and the action of the ear animation E of other characters C in linkage with the text input, voice operation, etc. of the character C of the user U1, U2, U3, dynamic communication with a richer sense of response and presence can be provided compared to communication with only text display.

[0265] The character C included in the first screen (hereinafter also referred to as the "first character C") corresponds to the character C included in the second screen (hereinafter also referred to as the "second character C"). The first character C and the second character C may have the same form (appearance, shape) or different forms. For example, the first character C included in the first screen and the second character C included in the second screen may have the same form of body B before and after the screen transition (for example, Figure 5 For example, the first character C included in the first screen may be Figure 5 In the state shown in mode A, the second character C included in the second screen can be as follows Fig.18 As shown, the body B is deformed to become the data window CW. In addition, for example, the first character C included in the first screen and the second character C2 included in the second screen may have different colors, patterns, and functions. In short, as long as the character C corresponding to the character C in the first screen is included in the second screen, it will be fine.

[0266] It should be noted that the character C refers to an ear animation E simulating an ear that becomes a constituent element of the avatar of the users U1, U2, and U3. The character C has a non-verbal information method for expressing the status and emotions of the users U1, U2, and U3 through the ear animation E. For the character C, it does not matter whether there is facial information of the users U1, U2, and U3.

[0267] In the system 100 of this embodiment, for activity 1, the host window W1, the lecturer window W2, and the participant window W3 are displayed, the host window W1 and the lecturer window W2 display the character C that becomes the leader of the activity, and the participant window W3 displays the character C that becomes the participant of the activity. The size of the character C displayed in the host window W1 and the lecturer window W2 may be displayed larger than the character C displayed in the participant window W3. In this configuration, since the size of the leader character C that explains the lecture materials is displayed larger, the explanation is easier to understand.

[0268] In the system 100 of the present embodiment, for activity 1, when the ear animation E of the participant character C performs a motion indicating raising a hand, a hand raiser window W5 and a speaker window W6 are displayed, in which the character C is moved from the participant window W3 and displayed, and the output of the voice of the character C in the virtual space R is restricted, and in the speaker window W6, at least one character C among the characters C displayed in the hand raiser window W5 is moved from the hand raiser window W5 and displayed, and the voice of the character C can be output in the virtual space R. In response to the character C moving from the hand raiser window W5 to the speaker window W6, the voice of the character C in the speaker window W6 can be output.

[0269] In the above configuration, when the ear animation E of the character C performs a hand-raising action, the character C is displayed in the hand-raiser window W5, so the leader can easily recognize the character C who raised his hand. In addition, the output of the voice is automatically switched on and off in accordance with the movement of the character C from the hand-raiser window W5 to the speaker window W6, so the time for performing the on and off operation can be saved. Furthermore, since only the voice of the character C displayed in the speaker window W6 is output, it is possible to prevent the occurrence of a situation where the voices of multiple characters C displayed in the hand-raiser window W5 cannot be heard clearly.

[0270] In the system 100 of the present embodiment, for activity 1, dozens to hundreds of students can participate in the lecture. In this case, if the images (real images) of all participants are displayed in the virtual space R, or the avatars with facial expressions are displayed, a considerable amount of transmission capacity is required. As a result, the movement will be slowed down or the image will be disordered. In this regard, in the system 100, a character C that does not include dynamically changing facial expressions is displayed. Therefore, even for lectures attended by many people, the transmission capacity can be reduced. Therefore, smooth activities can be carried out. In addition, when the images of students are displayed, it is difficult to confirm the status and reaction of the students. In this regard, in the system 100, the status and reaction of the students are represented by the movement of the ear animation E of the character C that is easily visually recognizable. Therefore, in the system 100, the status and reaction of the students can be easily confirmed.

[0271] In the system 100 of the present embodiment, for activities 2 to 5, when the size of one character C among the multiple characters C displayed in the virtual space R is changed, the sizes of the other characters C may also be changed in the same manner. In this configuration, since the sizes of all the characters C are changed in the same manner, an activity with a sense of unity can be realized in the virtual space R.

[0272] In the system 100 of the present embodiment, for activity 2, when the operation of dragging and dropping the character C relative to the data window W9 is accepted, the prescribed function is executed in the data window W9. Specifically, the data window W9 is changed to the data window CW of the character C that can be operated by the users U1, U2, and U3. In this configuration, by dragging and dropping the character C, which is accompanied by the movement of the character C, unlike a simple button click, in the feeling of moving time and the movement action of the character C, it can be expected that the effect of easily generating a feeling of participation and easily bringing in emotions such as switching to a new scene can be expected. This operation can be considered to be similar to the atmosphere of discarding a document by dragging it to the recycle bin in the screen. It can also be considered that: in order to process quickly, it is preferred to delete the click action, but when the event is carefully generated, having the action time can generate a feeling of preparing to move to the scene more in accordance with the natural feeling.

[0273] It should be noted that if auditory stimulation or sound such as sound effects or short BGM is used according to the content of scene switching and the change of ear animation E, the transmission and recognition of scene generation and switching can be expected to be improved. By matching the sound with the content of the change in advance, the change can be easily recognized by the sound, making the communication richer.

[0274] In the system 100 of this embodiment, for Activity 2, after the character C is dragged and dropped relative to the data window W9, when an operation of dragging and dropping the character C displayed in the data window CW to an area outside the data window CW is accepted, the execution of the prescribed function is terminated. Specifically, the display is changed from the data window CW to the data window W9. In this configuration, by performing an operation such as dragging and dropping the character accompanied by the movement of the character C, an exit process and an exit feeling can be generated.

[0275] In the system 100 of this embodiment, in the virtual space R of activity 3, the display board B1 and the chat board B2 are displayed, the display board B1 displays content information, and the chat board B2 displays the conversation text after the voice of the character C is converted into text. A predetermined number of conversation texts are displayed on the chat board B2, among the multiple conversation texts produced based on the voice of the character C, starting from the latest conversation text in time series. In this configuration, for consultation, the content information (consultation content, etc.) that is the purpose of consultation and the content of the conversation can be confirmed while the consultation is promoted. Therefore, effective consultation can be achieved.

[0276] In addition, character C is used in the consultation in the virtual space R. Thus, even if the consultant does not want to show his face, the consultant's non-verbal information such as nodding, agreeing, and questioning can be expressed through the ear animation E. In addition, since character C does not have dynamically changing eyes, the consultant and the consultant can watch the screen without paying attention to each other's sight. In the screen, the conversation text based on the voice of character C is displayed on the chat board B2, so the speech of character C can be followed.

[0277] In addition, the conversation text displayed on the chat board B2 is not displayed as the conversation progresses. In consultation, it is important to extract and deepen the words that become keywords in the conversation. The extraction of words is preferably performed in a manner that does not cause obstacles to the conversation process as much as possible. Therefore, in the system 100, when an operation from users U1, U2, and U3 on the conversation text displayed on the chat board B2 is accepted, the conversation text is displayed on the display board B1. Thereby, new matters, important matters, etc. generated in the conversation can be displayed on the display board B1. Therefore, the words can be extracted without causing obstacles to the conversation process.

[0278] In the system 100 of this embodiment, for activity 4, the character C is displayed around the display board B3. In the system 100, the actions of the users U1, U2, and U3 may be detected, and the ear animation E may be moved based on the actions of the users U1, U2, and U3. In this configuration, when the character C plays a musical instrument, the rhythm of the actual musical instrument played by the users U1, U2, and U3 can be reflected in the ear animation E. In this way, an activity with a sense of unity and presence can be realized in the music performance.

[0279] In the system 100 of this embodiment, for activity 5, one or more moving stations S are displayed, and one or more characters are arranged on the one or more moving stations S, and are set to be movable in the virtual space R. In this configuration, the character C can be moved in the virtual space R by riding the moving station S. Thus, for example, it is possible to carry out activities such as sightseeing and art museum appreciation while riding the moving station S. By moving on the moving station S, a sense of presence can be obtained. In addition, by multiple characters C moving on the moving station S, a sense of unity can be obtained.

[0280] In the system 100 of the present embodiment, for activity 5, when a plurality of characters C are respectively arranged on a plurality of mobile stations S, the voices of the plurality of characters C arranged on one mobile station S may be prevented from being heard by the voices of the plurality of characters C arranged on other mobile stations S. In this configuration, a conversation can be conducted for each group of characters C riding on the mobile stations S. Therefore, the users U1, U2, and U3 of other groups can have a conversation without paying attention to each other.

[0281] In the system 100 of this embodiment, for the activity 6, the activity in the virtual space R is output as a video that can edit the animation of the character C. In this configuration, the animation can be edited after the activity is created as a video. Therefore, in the post-editing of the video, the performance of the ear animation E of the character C can be added or changed.

[0282] In the system 100 of the present embodiment, the action of the character C in the virtual space R is recorded and output. In this configuration, the action of the character in the virtual space R can be reviewed later.

[0283] In the system 100 of the present embodiment, the character C is rotated according to the position of a specified object so that the character C can be imagined to be looking at the specified object. In this configuration, the character C is rotated in response to at least one of the changes in the transition of the screen, the shape, state, and function of the character C, so that the character C can be imagined to be looking at the specified object (paying attention to the direction of the specified object). Thus, the character C can be imagined to be looking at and paying attention to the specified object, just like the object of a person's attention, the movement of the line of sight relative to the changing object, and the change of posture. The rotation of the character C becomes a non-verbal communication effect similar to the change of line of sight and posture, showing active concern for the conversation. Therefore, rich communication such as improved responsiveness to the conversation and improved sense of presence can be achieved.

[0284] It should be noted that the rotation of the character C in this embodiment can be expressed by the rotation of the ear animation E of the character C or by the rotation of the body B of the character C. The rotation direction is not necessarily limited to the center of the gaze area. The gaze in this embodiment, for example, when the predetermined object is a person, is described as an example, it may not be the gaze of the eyes of the other party as the object, but the gaze in the direction of the predetermined object such as the chest of the other party, the general direction of the other party, etc. As long as the atmosphere of concern is conveyed by the rotation in the direction of the gaze area, the rotation angle may also be small.

[0285] As mentioned above, although embodiment of this disclosure was described, this disclosure is not necessarily limited to the said embodiment, Various changes can be made within the range which does not deviate from the summary.

[0286] In the above-mentioned embodiment, an example is given in which the ear animation E is made to move in any one of the following forms: "nod", "agree", "applaud (clap)", "question", "raise hand" or "surprise". However, the action of the ear animation E corresponding to the user's action is not limited to this. The action of the ear animation E may also be other actions, and may also include other actions. The bending angle and speed of the ear animation E can be set (changed) appropriately. In this way, subtle emotions can be expressed, and the intensity of the expression of the emotion can also be adjusted (enhanced, weakened). For the ear animation E, when the intensity of the action is set to be strong, for example, the non-verbal information expression of people who are not good at expressing emotions can be expressed through the content, intensity, and speed of the action of the ear animation E. In addition, for communication between cultures with different non-verbal information expressions, the setting of the action form can be changed corresponding to the culture of users U1, U2, and U3.

[0287] In the above embodiment, the operation device 50 is used to detect the movement of the hand H or finger F of the user U1, U2, U3. However, the movement of the hand H or finger F of the user U1, U2, U3 may be detected by an image sensor (camera, laser, etc.).

[0288] In addition, the actions of the ear animation E are basically performed based on the operations of users U1, U2, and U3, but can also be combined with the expressions of users U1, U2, and U3 (face recognition performed by a camera), the voices of users U1, U2, and U3, and expression control performed by AI (Artificial Intelligence).

[0289] The operation of character C is not limited to being performed by humans, but can also be performed by AI (Chat GPT, Siri (registered trademark), Google Assistant (registered trademark), Alexa (registered trademark), Cortana (registered trademark)). That is, the user who operates character C can also be AI. User refers to the object that operates character C. The words spoken by AI are displayed as text, and the actions corresponding to the state and emotions of AI at that time are expressed through ear animation E. For example, when AI understands "OK", the ear animation E of AI is made to perform the "OK mark" action, and when AI cannot understand "NG", the ear animation E of AI is made to perform the "NG mark" action. When AI finds a good suggestion, the ear animation E can also be made to perform the "clapping" action, or the ear animation E can be changed into the action of two surprise marks. The emotions of the caller can also be read by AI and reacted through the actions of ear animation E. AI does not have original inherent facial information and posture information, so expressing AI's non-verbal information through ear animation E can communicate creatively and richly.

[0290] The animation simulating the ear is not limited to the ear itself, and may be an animation similar to the ear. The animation simulating the ear may also be a design reminiscent of the ear. The animation simulating the ear may also be an animation that uses the ear as a basic shape and is deformed from the ear.

[0291] In the above-mentioned embodiment, the operating device only needs to be able to detect acceleration and angular velocity. The operating device may be, for example, a portable phone (smartphone, etc.) having a display (display unit) that displays the ear animation E. In the case of a portable phone, the motion data is generated by operating the portable phone in the same manner as the operating device 50.

[0292] In the above embodiment, the form in which the control data is generated by the terminal 10, the terminal 20, and the terminal 30 is described as an example. However, the control data may also be generated by the server 40. In this configuration, the terminal 10, the terminal 20, and the terminal 30 send the motion data to the server 40, and the server 40 generates the control data based on the motion data.

[0293] In the above embodiment, the form in which the body B and the ear animation E are displayed on the terminals 10, 20, and 30 is used as an example for explanation. However, when the terminal is a wearable terminal or a head-mounted display (HMD) terminal, the body B and the ear animation E can be displayed on a lens or the like, or can be displayed as a hologram. That is, the character C is displayed in the cyberspace. The cyberspace can include VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0294] In the above embodiment, Fig.18 In the example shown, the display switching from the data window W9 to the data window CW is performed by dragging and dropping the character C to the data window W9. However, the display switching to the data window, that is, the method of displaying the display unit on the body B can also be performed by other methods.

[0295] For example, Fig.37 As shown in mode A, by clicking (touching) the body B of the character C, a "+" mark M1 is displayed on the body B. Fig.37 As shown in the mode B of FIG. 1 , when the “+” mark M1 displayed on the body B is dragged outward, the body B is enlarged and the display portion D is formed (displayed) on the body B. Fig.37As shown in the mode C of the embodiment, when the "+" mark M1 is further dragged outward, the body B is deformed into a roughly rectangular shape, and a display unit D of a predetermined size is displayed. By changing (enlarging or reducing) the size of the body B, the size of the display unit D can be appropriately set (adjusted). The display unit D displays data or functions as a whiteboard. Specifically, the display unit D has a function of selecting and displaying data, images, text, camera images, user self-images, pen input whiteboards, etc.

[0296] like Fig.38 As shown, it is also possible to Fig.37 In the state shown in mode C, the "+" mark M1 is further dragged outward to display multiple (two) display parts D1 and D2. Fig.38 In the example shown, input contents such as documents input by users U1, U2, and U3 may be displayed on the display unit D1, and a translation of the contents displayed on the display unit D1 may be displayed on the display unit D2.

[0297] The arrangement and size of the display units D1 and D2 can be set appropriately. In a form in which the body B functions as the display unit D (D1 and D2) (a form in which the body B becomes the display unit, a form in which the display unit is displayed on the body B), the display unit D1 and D2 can be set appropriately. Fig.18 As shown, the entire body B becomes the display unit, and the ear animation E is displayed on the upper part of the body B. Fig.37 As shown in the mode C, a part of the body B before deformation (a semicircular part) and ear animation E are displayed on the upper part of the body B after deformation. Fig.37 In the example shown in the pattern C, a part of the body B and the ear animation E are in the shape of a hat, and are arranged in a manner such that a hat is worn on the upper part of the body B. The shape of the part of the body B is not limited to a semicircular shape, and may also be a rectangular shape, etc. Fig.37 As in the example shown in mode C, the ear animation E and a part of the body B before deformation are displayed, and a part of the body B remains undeformed, thereby making it easy to identify the relationship between the body B before and after deformation.

[0298] like Fig.39As shown, two characters C displayed on the display parts D1 and D2 can also be displayed in a dialogue. In the dialogue between the characters C (users U1, U2, and U3), the dialogue text can be displayed on the display parts D1 and D2. It is also possible to obtain the voice data of users U1, U2, and U3 in the display parts D1 and D2, link the data with AI, and automatically or manually display the corresponding pictures, texts, dynamic images, camera images, dynamic image self-images captured by the camera, still image self-images, or texts corresponding to the voice, etc. in accordance with the dialogue. For example, in the dialogue between the doctor and the patient, when explaining cataract surgery, the eyeball structure is automatically displayed on the display part D2 of the doctor character C based on the content of the dialogue. As a result, it is easier for the patient to understand the replacement instructions of the artificial lens and the lens. In this way, based on keywords and key sentences, by making use of AI, the best pictures, articles, music, dynamic images, still images, websites, etc. are automatically displayed and output to the display parts D1 and D2 of the explainer character C, thereby dynamically and efficiently forming a dialogue.

[0299] In addition, as another method of making the display unit appear on the body B, Fig.40 As shown in the mode A of FIG. 1 , by clicking (touching) the upper side of the body B of the character C (the side of the ear animation E), a "+" mark M2, for example, is displayed on the body B. Fig.40 As shown in the mode B of FIG. 1 , when the “+” mark M3 displayed on the body B is dragged, the body B is enlarged and the display portion D is formed (displayed) on the body B. Fig.41 As shown in mode A, when the "+" mark M3 displayed on the display portion D is further dragged, as shown in Fig.41 As shown in mode B, the display portion D is enlarged.

[0300] like Fig.42 As shown in mode A, by clicking (touching) the upper side of the body B of the character C (the side of the ear animation E) again, a "+" mark M2 is displayed on the body B. By clicking the "+" mark M2 displayed on the body B, a new display part D2 is formed. Fig.42 As shown in mode B, when the “+” marks M3 and M4 displayed on the display parts D1 and D2 are further dragged toward the outside, the display part D is enlarged.

[0301] like Fig.43As shown, the function button FB can also be displayed in the display unit D1 and D2. One or more function buttons FB can be displayed. By clicking the function button FB, the functions in the display units D1 and D2 can be switched. As the function button FB, for example, a data display button, a handwriting input button, a translation button, an AI search button, etc. can be included. In each display unit D1 and D2, the function of the selected function button FB is exerted. For example, when the data button is selected as the function button FB in the display unit D1, the screen of a personal computer, etc. is displayed in the display unit D1, and the data can be selected and displayed. For example, when the handwriting input button is selected as the function button FB in the display unit D1, the tool for handwriting input (pen, pen color / thickness selection, eraser, etc.) is displayed in the display unit D1. For example, when the translation button is selected in the display unit D2, the translation is displayed in the display unit D2. For example, when the AI ​​search button is selected in the display unit D2, the image, text, dynamic image, still image or voice corresponding to the conversation is automatically displayed and output in the display unit D2.

[0302] As described above, by the operation of enlarging (reducing) the body B by the users U1, U2, and U3, functional change elements such as displaying the display unit D (D1, D2) are added. Therefore, visual dialogue can be easily provided, and its changes are interesting, and the dialogue becomes enjoyable. Since the operation of the body B is performed by the users U1, U2, and U3, it can be easily understood visually, including by the surrounding participants. In addition, with regard to the generation of the display unit D (D1, D2) achieved by the transformation of the body B of the character C, the user interface is significantly improved by freely and easily configuring it by dragging the "+" mark displayed on the body B for multiple displays or size changes. Therefore, it is highly useful as an audio-visual support tool for dialogue.

[0303] In addition, as for character C, in various places (environments, scenes) such as specific places, exhibition halls, classrooms, libraries, cafeterias, etc. in the virtual space, body B can also automatically transform (deform) according to the place, attributes, etc. The transformation of body B includes changes in form (shape), color, design (pattern, etc.). In this way, by transforming body B according to the place, etc., the current atmosphere can be further felt. Users U1, U2, and U3 can set whether body B can transform. In addition, it can also be preset that when character C transforms, multiple options can be displayed for selection.

[0304] Furthermore, the character C may also be integrated with the avatar in the virtual space. Fig.44 The role C shown in the pattern A can be Fig.44For example, the character C is displayed integrally with the avatar AB as shown in mode B. The avatar AB may or may not include facial expressions. In this case, a part of the character C, for example, the upper part of the body B including the ear animation E, is attached to the upper part of the avatar AB. By attaching not only the ear animation E but also a part of the body B (for example, a semicircular part, identification information), the character C can be transformed smoothly, and even when the transformation is performed, the continuity and visual recognizability of the identification can be maintained by continuing a part of the design that gave the original identification, and the discomfort related to the identification caused by the transformation can be reduced.

[0305] The part attached to the avatar AB is not limited to the ear animation E and a part of the body B, but may be only the ear animation E. In addition, the part attached to the avatar AB may also be a base body with ear animation E, a supporting body, etc., such as a hat, a headset with ear animation E, etc. The shape of the hat can be various shapes (top hat, baseball cap, crown, helmet, etc.). The avatar AB can be distinguished by the shape of the hat. Text information, name, nickname, attribute, mark or NFT that can be specially specified can also be displayed on the hat. As a result, the distinguishability of the avatar AB is further improved.

[0306] As described above, by adding ear animation E (including a part of body B and a hat) to avatar AB, the function of ear animation E can be maintained even when avatar AB is used. In this way, by integrating ear animation E and avatar AB, non-verbal communication can be enhanced in avatar AB, and identification information can be expressed and corresponding brands suitable for the scene can be matched. In addition, by integrating design and function with NFT (Non-Fungible Token), services can be further expanded.

[0307] It should be noted that character C is not limited to human-shaped avatars AB, but can also be applied to "objects" such as cars and displays without facial expressions. By applying ear animation E to "objects", the "objects" can be given the feeling of living things. Character C may not be a "object", but a character with ear animation E attached to designs such as NFTs, marks such as brands, images, and text information. In this way, if objects including ear animation E have certain functions and effects as specific groups, attributes, and identification displays, they can be widely used as character C.

[0308] In addition, as another method of displaying the display unit on the body B (a method of changing the form), other animations may be displayed when the display unit is displayed on the body B from the normal state of the character C (a state where the display unit is not displayed). Fig.45As shown, for example, an animation AM of smoke, cloud, tornado (cyclone), etc. may be displayed between the normal state of the character C (hereinafter, also referred to as the "first state") and the state displayed on the body B by the display unit (hereinafter, also referred to as the "second state") The animation AM can be a variety of animations as long as the appearance is different from the first state and the second state.

[0309] Thus, by displaying the animation AM between the first form and the second form, the impression of an immediate (sudden) change from the first form to the second form can be alleviated. In this case, for example, the discomfort of the users U1, U2, and U3 in the change from the first form to the second form can be reduced.

[0310] It is preferably pre-set that the first form and the second form can identify the users U1, U2, and U3 before and after the form changes. Fig.46 As shown, in the first form and the second form, the identification information (name, initials, specific design (NFT, pattern, etc.), brand mark) displayed on the body B can be made the same. In addition, in the second form, the shape of the body B can be the same as in the first form. That is, in the second form, the display part D is displayed while maintaining the shape of the body B that is the same as in the first form. In addition, the same color can be used in the first form and the second form.

[0311] like Fig.47 As shown in the mode A of FIG. 1 , in the second form, a facial image (video) may also be displayed on the body B. The display of the facial image is realized, for example, by acquiring a facial part from an image including the faces of the users U1, U2, and U3 obtained by a camera or other imaging device by performing facial recognition using AI or the like, and displaying the facial part on the body B. In addition, as Fig.47 As shown in mode B, in the second form, identification information such as a name can also be displayed in the frame portion formed by the body B.

[0312] As described above, in the case of changing from the first form to the second form, it is preferred to maintain the ease of identification of users U1, U2, and U3 in the first form and the second form. Here, the viewing angle of a person is generally about 120°. In the case of high performance required for reading information in the screen, there is a clear field of view (central field of view) within a viewing angle of 10°. Therefore, when observing the screen, it is difficult to read the information on the entire screen, and you have to be careful everywhere. For example, when the viewing angle of the central field of view is set to 10° at a distance of 50cm to 60cm from the screen, it is equivalent to an area with a diameter of about 8cm to 10cm. In this way, when reading information in the screen, the viewing angle is preferably within 10°. Even in the case where it is difficult to set the viewing angle to 10°, it is preferred to reduce the viewing angle as much as possible.

[0313] In order to make the viewing angle within 10° or as small as possible, it is preferred that the character C and the display unit are not simply combined, but the identification information (identification information) is maintained in the first form and the second form, and the identification of the character C as a whole is maintained, and the display unit D is displayed integrally in the character C. Therefore, if Fig.46 or Fig.47 As in the example shown, it is preferred to form a frame around the display portion D with the same color and design as the character C, or to display a name, initials, a specific design (NFT, pattern, etc.), or a brand mark as identification information on a part of the body B, such as the upper part, as part of the frame design.

[0314] For example, when a character C and a translation are displayed on the screen, it is difficult to read the changes in the ear animation E of the character C and the text of the translation at the same time in a wide viewing angle area. Fig.48 As shown in FIG. 1 , when the character C and the translation are displayed but separately from the display window W13, the viewing angle becomes larger, so it may be difficult to read the changes in the ear animation E of the character C and the text of the translation at the same time. Fig.49 As shown, for example, when the display window W14 is displayed at the bottom of the character C who is the speaker, the viewing angle is Fig.48 The examples shown are small, but the individual fields of view are larger than 10°. Fig.50 As shown, in the case where the character C and the display window W15 displayed integrally with the avatar AB are displayed, the viewing angle becomes larger.

[0315] like Fig.51 As shown in the example, when the body B of the character C displays the display unit D, the ear animation E and the text information displayed on the display unit D are displayed at a closer position than when the character C and the display window are displayed separately, so the viewing angle becomes smaller. Therefore, visual recognition is improved and information processing becomes easier. As a result, the ear animation E (non-verbal information) and the text information can be understood at the same time. Moreover, in the structure in which the body B of the character C displays the display unit D and is integrated, it can be expected that the sense of unity between the ear animation E and the text information will be improved, and the sense of intimacy and dynamism will be improved compared to the structure in which the character C and the display window are displayed separately.

[0316] In the future, as AI-based simultaneous interpretation technology spreads, it is predicted that communication in different languages ​​will become commonplace in daily life around the world. Character C that integrally displays ear animation E and display unit D that displays the interpreted text can be expected to be used as an application for realizing communication in different languages. In particular, from the perspective of GX (Green Transformation), it is very effective to reduce transmission capacity (energy reduction) by expressing non-verbal expressions through ear animation E instead of always displaying facial information as real dynamic images. It should be noted that in cases where facial information recognition is required, such as Fig.47 As shown, a still image of the face or the like can be displayed on the body B of the character C.

[0317] In addition, as another method of making the display unit appear on the body B, Fig.52 As shown, character C can also be clicked (touched, selected) in screen G14 displaying a plurality of characters C. Screen G14 is, for example, a screen displayed in a conference activity. Screen G14 displays a participant window W16 and an operation window W17 displaying character C. Character C participating in the conference is displayed in participant window W16. Information (for example, "A" to "H") that can identify the participant is displayed on character C's body B. Multiple action modes of the ear animation E of character C are displayed in operation window W17. Users U1, U2, and U3 click on one of the action modes displayed in operation window W17. As a result, the ear animation E of character C of users U1, U2, and U3 moves in a specified mode.

[0318] In screen G14, when users U1, U2, and U3 click on their own character C, Fig.53 As shown, a function selection window W18 is displayed near the character C. The function selection window W18 pops up near the character C like a dialog box. In the function selection window W18, a plurality of functions to be performed in the character C are displayed. For example, the "Presentation", "Self-image" and "Design" functions are displayed in the function selection window W18. When the "Presentation" function is selected, the display portion D ( Fig.54 ). When the "self-image" function is selected, the facial images (videos) of the users U1, U2, and U3 are displayed on the body B. When the display of the facial images is stopped, the "self-image" function is selected again. When the "design" function is selected, a screen for setting (changing) the design of the character C is displayed. In the design setting screen, the text ("A" etc.) displayed on the character C, the color of the character C, etc. can be set.

[0319] When "Demo" is selected in the function selection window W18, Fig.54As shown, a display portion D is formed (displayed) on the body B of the character C. Identification information is displayed on the body B of the character C. The character C ("A") having the display portion D is displayed in an enlarged manner in the participant window W16. The other characters C ("B" to "H") are displayed in a reduced manner. The other characters C ("B" to "H") are arranged side by side in the upper part of the participant window W16. The display portion D of the body B of the character C displays the presentation materials, etc.

[0320] A "+" mark M5 and an "×" mark M6 are displayed on the body B. When the "+" mark M5 is dragged, the display portion D can be enlarged or reduced. When the "×" mark M6 is clicked, the display portion D of the body B ends. In this case, the display portion D is restored to Fig.52 The display of the participant window W16 is shown.

[0321] In the example shown above, the display unit D is displayed on the body B of the character C by displaying the function selection window W18 and clicking "Demo" in the function selection window W18. However, the method of displaying the display unit D on the body B of the character C may also be other methods. For example, the display unit D may also be displayed based on the voice of the users U1, U2, and U3. Specifically, when the users U1, U2, and U3 start to speak, the display unit D may be automatically displayed on the body B of the character C. In this case, when the display unit D is displayed in the case of all the echoing words such as "Hmm", "I see", etc., the screen display is frequently switched, and the users U1, U2, and U3 may sometimes feel uncomfortable. Therefore, for example, it may also be set that: when the users U1, U2, and U3 speak continuously for more than 5 seconds, the display unit D is displayed on the body B. In addition, as another method, it may also be set that: when the space bar of the keyboard is pressed, the display unit D is displayed on the body B of the character C. As described above, the display on the body B by the display unit D may be performed by manual operation at an operation level that does not hinder the conversation or by automation according to the situation.

[0322] In the above embodiment, the operation of the character C by the users U1, U2, and U3 is described as an example. However, as described above, the operation of the character C may also be performed by AI. One or more characters C are displayed in the virtual space R, and at least one of the one or more characters C is operated by AI. A display window (display unit) W21 corresponding to the character C operated by AI may also be displayed (see Fig.56 ). At least one of the text generated by AI, a still image associated with the text, a dynamic image associated with the text, an article associated with the text, and website information associated with the text is displayed in the display window W21. In addition, the character C can also output a voice corresponding to the text generated by AI.

[0323] Through the operation of the character C by the AI, the character C performs the following actions, for example. When the AI ​​understands the input text and / or voice of the communication partner, the AI ​​operates the character C to make the ear animation E of the character C nod ( Figure 5 Mode B). While AI is processing (thinking), AI operates character C to repeatedly perform the ear-entwining action of character C's ear animation E ( Figure 5 When AI finishes processing (getting a result, etc.), AI operates character C to make character C's ear animation E perform an action that imagines an exclamation mark (!) (refer to Fig.59 ). When AI cannot cope with it, AI controls character C to repeatedly perform character C's ear animation E's ear deeply bent action ( Figure 5 Mode C). When the other party has understood and comprehended the communication, the AI ​​controls character C to perform the ear clapping action of character C's ear animation E.

[0324] An example of the form of AI operating character C is described below. Fig.55 As shown, the screen G15 displays a participant window W19 and an operation window W20 showing the character C. The participant window W19 displays the character C who is attending the meeting. Information (e.g., "A" to "E", "AI") that can identify the participant is displayed on the body B of the character C. Multiple action patterns of the ear animation E of the character C are displayed in the operation window W20.

[0325] In screen G15, when a user (including users U1, U2, U3, etc.) clicks on character C of "AI", Fig.56 As shown, the screen changes to screen G16. In screen G16, a display window (display unit) W21 is displayed near the AI ​​character C, and a text window W22 is displayed. The display window W21 is a display unit corresponding to the AI ​​character C, and is displayed in a form that allows identification of the AI ​​character C and the display window W21. Fig.56 In the example shown, the display window W21 is displayed below the AI ​​character C. Text can be input in the text window W22. As a method of inputting text, keyboard input, handwriting input, voice input, etc. can be listed.

[0326] When any character C (in Fig.56 In the example shown, when the character "D" is dragged and dropped to the text window W22, Fig.57As shown, the screen changes to screen G17. If there are multiple characters C, drag and drop the multiple characters C to the text window W22. The following describes the case where there is only character C "D". In screen G17, AI character C and display window W21, questioner character C and input window W23, and operation window W20 are displayed. Input window W23 is displayed corresponding to questioner character C. Fig.56 In the example shown, the input window W23 is displayed below the questioner character C. In the screen G17, the characters C other than the questioner are displayed collectively in a predetermined area. The characters C other than the questioner may be displayed smaller than the AI ​​character C and the questioner character C.

[0327] In screen G17, text such as "What is it?" is displayed in display window W21. The user enters text (for example, "Please tell me 00") in input window W23. As a method of inputting text, keyboard input, handwriting input, voice input, etc. can be listed. In the process of inputting text into input window W23, the ear animation E of the character C of the questioner performs an ear-entwining action. In addition, in the process of inputting text into input window W23, the ear animation E of the character C of the AI ​​performs a nodding action. It should be noted that the ear animation E of the character C of the AI ​​at this time can also be a left-right swinging action.

[0328] When the text input to the input window W23 is completed, the user clicks the character C of AI. Fig.58 As shown, the screen changes to screen G18. While AI is processing the text (question) input to input window W23, the ear animation E of AI character C repeatedly performs a winding action. When AI's processing is completed, as shown in FIG. Fig.59 As shown, the ear animation E of the AI ​​character C performs an action of imagining an exclamation mark (!). In addition, the display window W21 displays the answer to the text (question) input into the input window W23. In the display window W21, in addition to the text generated by the AI, still images related to the text, dynamic images related to the text, articles related to the text, website information related to the text, etc. can also be displayed. In the display window W21, the text and the translation corresponding to the text, or the translation of the text can also be displayed. In addition, the voice corresponding to the text displayed in the display window W21 can also be output.

[0329] If the processing of the text (question) input to the input window W23 cannot be completed, the ear animation E of the AI ​​character C repeatedly performs the action of bending the ear deeply. Fig.60 As shown, the screen changes to screen G19. A text window W22 is displayed in screen G19.

[0330] In the above embodiment, the case where there is one AI character C is described as an example, but there may be two or more AI characters C. When there are two or more AI characters C, the users U1, U2, and U3 may click on the AI ​​character C for which they require an answer.

[0331] In the above-mentioned form, the form in which the display window (display unit) W21 is displayed below the AI ​​character C is described as an example. However, the form of the display unit corresponding to the AI ​​character C is not limited to this. Fig.61 The character C and the display unit can be displayed separately as shown in the mode A, or as shown in the mode Fig.61 The display unit and the ear animation are integrated as shown in the mode B. Fig.61 The display unit and the character can be integrated as shown in mode C (a part of the character's body can be used as the display unit), or Fig.61 As shown in mode D, the display unit and the character after the body is deformed are integrated.

[0332] In the above form, the conversation between AI character C and questioner character C is shown in the form of a chat. However, the communication between AI character C and questioner character C can also be Fig.39 In addition, when there is a character C of the AI ​​and a plurality of character Cs of users U1, U2, and U3, after inputting into one's own input window W23, the character C of the other party (the character C of the AI ​​or the character C of users U1, U2, and U3) is clicked. When the character C of the other party is the character C of users U1, U2, and U3, the content input into the input window W23 is sent to the AI.

[0333] On the basis of the above-mentioned embodiment, it is also possible to have functions such as automatic recording, playback, and time position information of a certain time before and after the action of the ear animation E is used as a signal. Regarding automatic recording, since the ear animation E only acts (reacts) after hearing the speech of other users U1, U2, and U3, the recording start position (time) using the action of the ear animation E as a signal is preferably before the action of the ear animation E, for example, 5 seconds before. The recording end position (time) can be set to a certain time after the action of the ear animation E (for example, 15 seconds). The recording end position can be determined by the operation of the users U1, U2, and U3, or the recording can be extended manually. In addition, the recording can also be performed during lectures according to the timing (position and time assigned with labels) set by the users U1, U2, and U3.

[0334] like Fig.62As shown, when reproducing the video, the action pattern of the ear animation E is clicked in the time chart. For example, when the action pattern of clapping is clicked, the dynamic image from 5 seconds before to 15 seconds after the action of the ear animation E is reproduced. In this way, the video can be viewed efficiently.

[0335] As described above, since the video is recorded starting from the movement of the ear animation E, no special operation is required to record the video. Therefore, it is easy to review the lectures, etc. In addition, the lecturer can also confirm the reaction based on the movement of the ear animation E, which helps to review the demonstration. Moreover, for example, it is also possible to realize individual records focusing on the reaction of a specific user instead of the statistics of the entire lecture. Therefore, detailed reaction analysis can be performed. The aggregation of the analysis results can also be performed by AI.

[0336] [Note]

[0337] As understood from the various examples described above, the present disclosure includes the following aspects.

[0338] (Note 1)

[0339] A display program controls the display of activities in a virtual space using a character that can be operated by a user, wherein the character causes an animation of a simulated ear to move in response to the user's operation, and the display program causes a computer to execute the following steps: a step of displaying a first screen including content information of the character and content based on the activity; and a step of changing the display to a second screen different from the first screen in response to the character's action.

[0340] (Note 2)

[0341] The display program according to Supplementary Note 1, wherein a plurality of display areas are displayed in the virtual space, and the content information is displayed in at least one of the plurality of display areas.

[0342] (Note 3)

[0343] According to the display procedure described in Note 2, the multiple display areas include: a first display area, in which the character who is the leader of the activity is displayed; and a second display area, in which the character who is the participant of the activity is displayed, and the size of the character displayed in the first display area is displayed larger than the size of the character displayed in the second display area.

[0344] (Note 4)

[0345] A display procedure according to Note 3, wherein the plurality of display areas include: a third display area, in which, when the animation of the character of the participant performs an action indicating raising a hand, the character moves from the second display area and is displayed, and the output of the character's voice in the virtual space is restricted; and a fourth display area, in which at least one of the characters displayed in the third display area moves from the third display area and is displayed, and is capable of outputting the character's voice in the virtual space, and the voice of the character in the fourth display area is enabled to be output in response to the character moving from the third display area to the fourth display area.

[0346] (Note 5)

[0347] The display program according to Supplement 2 or Supplement 3, wherein each of the plurality of display areas can change at least one of a size and a position in the virtual space.

[0348] (Note 6)

[0349] The display program according to Supplementary Note 3 or Supplementary Note 4, wherein the character displayed in the second display area is displayed in a manner that can be distinguished based on an attribute of the user.

[0350] (Note 7)

[0351] The display program according to Supplement 3 or Supplement 4, wherein, for the character displayed in the second display area, output of the character's voice in the virtual space is restricted.

[0352] (Note 8)

[0353] The display program according to Supplementary Note 2, wherein input contents inputted from the user are displayed in the display area displaying the contents information.

[0354] (Note 9)

[0355] The display program according to Supplementary Note 1, wherein a display unit that displays the content information is displayed in the virtual space.

[0356] (Note 10)

[0357] The display program according to Supplementary Note 9, wherein input contents inputted by the user are displayed on the display unit.

[0358] (Note 11)

[0359] The display program according to Supplementary Note 9 or Supplementary Note 10, wherein, when a drag-and-drop operation of the character with respect to the display unit is accepted, a predetermined function is executed on the display unit.

[0360] (Note 12)

[0361] The display program according to Supplementary Note 11, wherein the user operating the display unit is displayed in a discernible manner.

[0362] (Note 13)

[0363] The display program according to Supplementary Note 11, wherein, after the character is dragged and dropped with respect to the display unit, when an operation of dragging and dropping the character displayed on the display unit to an area outside the display unit is accepted, execution of the predetermined function is terminated.

[0364] (Note 14)

[0365] The display program according to Supplementary Note 9 or Supplementary Note 10, wherein, when a predetermined operation on the display unit is accepted from the user, an operation screen for operating display content on the display unit is displayed.

[0366] (Note 15)

[0367] The display program according to Supplementary Note 8 or Supplementary Note 10, wherein the character of the user who makes the input is displayed in a distinguishable manner from the other characters.

[0368] (Note 16)

[0369] The display program according to Supplementary Note 9, wherein positions of the character and the display unit in the virtual space can be changed.

[0370] (Note 17)

[0371] According to the display program according to Supplement 9 or Supplement 10, the size of the character can be changed, and when the size of one of the plurality of characters displayed in the virtual space is changed, the sizes of the other characters are also changed in the same manner.

[0372] (Note 18)

[0373] The display program according to Supplement 1, wherein in the virtual space, a content information display unit that displays the content information, a text display unit that displays a conversation text obtained by texting the voice of the character, and at least two of the characters are displayed.

[0374] (Note 19)

[0375] The display program according to Supplementary Note 18, wherein a predetermined number of the conversation texts, counted in time series from the latest conversation text, among the plurality of the conversation texts created based on the voice of the character are displayed in the text display unit.

[0376] (Note 20)

[0377] The display program according to Supplementary Note 18 or Supplementary Note 19, wherein when an operation on the conversation text displayed on the text display unit is accepted from the user, the conversation text is displayed on the content information display unit.

[0378] (Note 21)

[0379] According to the display program described in Note 18 or Note 19, the configuration and size of the character can be changed, and when the configuration or size of one of the two characters displayed in the virtual space is changed, the configuration or size of the other character is also changed accordingly.

[0380] (Note 22)

[0381] The display program according to Supplementary Note 9 or Supplementary Note 10, wherein the character is displayed around the display unit, an action of the user is detected, and the animation is operated based on the action of the user.

[0382] (Note 23)

[0383] The display program according to Supplementary Note 9, wherein one or more moving bodies are displayed, the one or more moving bodies being configured for one or more of the characters and being set to be movable in the virtual space.

[0384] (Note 24)

[0385] The display program according to Supplementary Note 23, wherein, when a plurality of the characters are arranged on each of a plurality of the moving bodies, the voices of the plurality of the characters arranged on one of the moving bodies are made inaudible from the voices of the plurality of the characters arranged on other moving bodies.

[0386] (Note 25)

[0387] The display program according to Supplementary Note 9, wherein a commentary on the content information displayed on the display unit is displayed.

[0388] (Note 26)

[0389] The display program according to Supplementary Note 9, wherein, in the activity in the virtual space, the animation of the character is output as an editable image.

[0390] (Note 27)

[0391] The display program according to Supplement 1 or Supplement 2, wherein the movement of the character in the virtual space is recorded and output.

[0392] (Note 28)

[0393] A display method controls the display of activities in a virtual space using a character that can be operated by a user, wherein the character causes an animation of a simulated ear to move in response to the user's operation, wherein the display method includes the following steps: a step of displaying a first screen including content information of the character and content based on the activity; and a step of changing the display to a second screen different from the first screen in response to the character's action.

[0394] (Note 29)

[0395] A display system controls the display of activities in a virtual space using a character that can be operated by a user, and the character causes an animation of a simulated ear to move in response to the user's operation, wherein the display system is configured to: display a first screen containing content information of the character and content based on the activity; and change the display to a second screen different from the first screen in response to the character's action.

[0396] Description of Reference Numerals

[0397] 10: terminal, 11: input unit, 12: control unit, 13: storage unit, 13P: terminal program, 14: communication unit, 15: output unit, 20: terminal, 30: terminal, 40: server, 42: control unit, 43: storage unit, 43P: server program, 43S: receiving data, 44: communication unit, 50: operation device, 51A: operation surface, 52: communication unit, 53: input unit, 54: control unit, 55: battery, 100: display system, AB: avatar, AM: animation, AP: access point, B: body, B1: display board (content information display unit) ), B2: Chat board (text display part), B3: Display board, B4: Display board, B5: Display board, B6: Display board, B7: Display board, B8: Display board, B9: Display board, BR1: Group discussion room, BR2: Group discussion room, BR3: Group discussion room, BR4: Group discussion room, BS: Base station, C: Character, CG: Character, CW: Data window, D, D1, D2: Display part, E: Ear animation, F: Finger, G1: Selection screen, G2: Screen, G3: Screen, G4: Screen, G5: Screen, G6: Screen, G7: Screen, G8 : Screen, G9: Screen, G10: Screen, G11: Screen, G12: Screen, G13: Screen, G14: Screen, G15: Screen, G16: Screen, G17: Screen, G18: Screen, G19: Screen, H: Hand, L: Distance, M1: Mark, M2: Mark, M3: Mark, M4: Mark, M5: Mark, N: Network, P: Pointer, R: Virtual Space, S: Mobile Station, SB: Scroll Bar, T: Table, U1: User, U2: User, U3: User, W1: Host Window (Display Area), W2: Lecturer Window (Display Area) domain), W3: participant window (display area), W4: data window (display area), W5: hand raiser window (display area), W6: speaker window (display area), W7: data window, W9: data window, W10: data window, W11: participant window, W12: data window, W13: display window, W14: display window, W16: participant window, W17: operation window, W18: function selection window, W19: participant window, W20: operation window, W21: display window, W22: text window, W23: input window.

Claims

1. A computer program product comprising a display program for controlling the display of activities in a virtual space using a character operable by a user, wherein: The display program enables the computer to execute the following steps: a step of causing a first screen including content information of the character and content based on the activity to be displayed; and In response to an operation related to the action of the character, the step of changing the display to a second screen, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, The character in the first screen is different from the character in the second screen in at least one of appearance, state and function, At least one of the character on the first screen and the character on the second screen moves an animation simulating an ear in response to an operation by the user, The display unit displays a part of the character on the second screen by at least one of a predetermined operation and a voice input performed by the user on the first screen.

2. The computer program product according to claim 1, wherein: causing a plurality of display areas to be displayed in the virtual space, displaying the character in at least a portion of the plurality of display areas, Each of the plurality of display areas for displaying the character is divided corresponding to the attribute of the character, The form of the character is changed for each of the display areas.

3. The computer program product according to claim 1 or 2, wherein: causing a plurality of display areas to be displayed in the virtual space, displaying the character in at least a portion of the plurality of display areas, The plurality of display areas for displaying the character include: the display area in which output of the voice of the character in the virtual space is restricted; and The display area is capable of outputting the voice of the character in the space.

4. The computer program product according to claim 3, wherein: The role displayed in the display area is displayed in a distinguishable manner based on the attribute of the user.

5. The computer program product according to claim 2, wherein: In the plurality of display areas, the voice of the character displayed in one of the display areas is prevented from being heard from the voice of the character displayed in the other display areas.

6. The computer program product of claim 2, wherein: When an operation of dragging and dropping the character from one of the display areas to the other of the display areas is accepted on the first screen, display of the character in the one of the display areas and the other of the display areas is changed on the second screen.

7. The computer program product according to claim 1 or 2, wherein: When a predetermined operation is accepted from the user, the front or back directions of the plurality of characters are collectively changed and displayed.

8. The computer program product according to claim 1 or 2, wherein: At least one of a conversation text obtained by converting the voice of the character into text, a translation of the conversation text, and a translation of the display content of the display unit is displayed on the display unit.

9. The computer program product according to claim 8, wherein: At least one of necessary pictures, texts, music, moving images, still images, and websites is displayed on the display unit by using AI based on at least one of the voice and text input of the character.

10. The computer program product of claim 1, wherein: A content information display unit that displays the content information is displayed in the virtual space.

11. The computer program product according to claim 10, wherein: The input content input by the user is displayed on the content information display unit.

12. The computer program product according to claim 10 or 11, wherein: When an operation of dragging and dropping the character with respect to the content information display unit is accepted, a predetermined function is executed on the content information display unit.

13. The computer program product of claim 12, wherein: After the character is dragged and dropped with respect to the content information display unit, when an operation of dragging and dropping the character displayed on the content information display unit to an area outside the content information display unit is accepted, execution of the predetermined function is terminated.

14. The computer program product according to claim 10 or 11, wherein: When a predetermined operation on the content information display unit is accepted from the user, an operation screen for operating the displayed content on the content information display unit is displayed.

15. The computer program product according to claim 10 or 11, wherein: Ability to change the size of said character, When the size of one of the plurality of characters displayed in the virtual space is changed, the sizes of the other characters are also changed in the same manner.

16. The computer program product of claim 1, wherein: In the virtual space, a content information display unit that displays the content information, a text display unit that displays a conversation text obtained by converting the voice of the character into text, and at least two of the characters are displayed.

17. The computer program product of claim 16, wherein: When an operation on the conversation text displayed on the text display unit is accepted from the user, the conversation text is displayed on the content information display unit.

18. The computer program product of claim 15, wherein: Ability to change the configuration and size of said character, When the position or size of one of the two characters displayed in the virtual space is changed, the position or size of the other character is also changed in accordance with the position or size of the one character.

19. The computer program product of claim 10, wherein: One or more moving bodies are displayed, the one or more moving bodies being configured for one or more of the characters and being set to be able to move in the virtual space.

20. The computer program product of claim 19, wherein: When a plurality of the characters are arranged on each of the plurality of the moving objects, the voices of the plurality of the characters arranged on one of the moving objects are prevented from being heard from the voices of the plurality of the characters arranged on other moving objects.

21. The computer program product of claim 10, wherein: Displaying explanations about the content information displayed on the content information display unit.

22. The computer program product of claim 10, wherein: During the activity in the virtual space, the animation of the character is output as an editable image.

23. The computer program product according to claim 1 or 2, wherein: An image or video of the user's face is displayed on a portion of the character.

24. The computer program product of claim 1, wherein: displaying one or more of the characters in the virtual space, At least one of the one or more characters is operated by AI.

25. The computer program product of claim 24, wherein: The character operated by the AI ​​causes the animation of the simulated ear to move through the operation of the AI. a display unit that displays the character operated by the AI, At least one of text generated by the AI, a still image associated with the text, a moving image associated with the text, an article associated with the text, and website information associated with the text is displayed on the display unit corresponding to the character operated by the AI.

26. A computer program product according to claim 24 or 25, wherein: The character outputs speech corresponding to the text generated by the AI.

27. The computer program product of claim 1, wherein: The character is rotated according to the position of a predetermined object so that it is imagined that the character is looking at the predetermined object.

28. A display method for controlling the display of an activity in a virtual space using a character that can be operated by a user, wherein: The display method comprises the following steps: a step of causing a first screen including content information of the character and content based on the activity to be displayed; as well as In response to an operation related to the action of the character, the step of causing the display to change to a second screen, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, The character in the first screen is different from the character in the second screen in at least one of appearance, state and function, At least one of the character on the first screen and the character on the second screen moves an animation simulating an ear in response to an operation by the user, The display unit displays a part of the character on the second screen by at least one of a predetermined operation and a voice input performed by the user on the first screen.

29. A display system for controlling the display of activities in a virtual space using a character that can be operated by a user, wherein: The display system is configured as follows: causing a first screen including content information of the character and content based on the activity to be displayed; and In response to an operation related to the action of the character, causing the display to change to a second screen, wherein the second screen is different from the first screen and includes a character corresponding to the character of the first screen, The character in the first screen is different from the character in the second screen in at least one of appearance, state and function, At least one of the character on the first screen and the character on the second screen moves an animation simulating an ear in response to an operation by the user, The display unit displays a part of the character on the second screen by at least one of a predetermined operation and a voice input performed by the user on the first screen.

Images