Information processing system, program, and information processing method

Abstract

To provide an information processing system and so on which can improve sensation as if a user has dialog with a person through a character.SOLUTION: According to an aspect of the present invention, there is provided an information processing system. The information processing system has a processor. The processor, in the display step, executes display processing for displaying a character operated in a virtual space. In the dialogue step, the processor executes dialogue processing for allowing the character being displayed in the display processing to have dialogue with other characters. The dialogue processing includes first dialogue processing for allowing the character to have dialogue based on the input by a person and second dialogue processing for allowing the character to have dialogue even without input by a person.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a program, and an information processing method. [Background technology]

[0002] Patent Document 1 discloses a technique for performing a conversation process when a PC (Player Character) and an NPC (Non Player Character) come into contact with each other in a game space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-141435 A Summary of the Invention [Problem to be solved by the invention]

[0004] Since the characters (avatars) operated by users may visit the metaverse space at any time and in large numbers, a mechanism for the NPCs controlled by the system to respond to these user characters is conceivable. In order to increase the sense of realism in the metaverse space, it is desirable for even NPCs to behave as human-like as possible.

[0005] In view of the above circumstances, the present invention provides an information processing system and the like that can improve the feeling of interacting with a person through a character. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an information processing system. The information processing system includes a processor, and in a display step, the processor executes a display process for displaying a character operating in a virtual space. In a dialogue step, the processor executes a dialogue process for the character displayed in the display process to have a dialogue with another character. The dialogue process includes a first dialogue process for the character to have a dialogue based on a human input, and a second dialogue process for the character to have a dialogue even without a human input.

[0007] According to this embodiment, it is possible to improve the feeling of interacting with a person through a character. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a metaverse-related processing system 1. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a server device 10. [Diagram 3] FIG. 2 is a diagram showing a hardware configuration of a first user terminal 20. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of a control unit of each device. [Diagram 5] FIG. 11 is an activity diagram showing an example of a character control process. [Figure 6] FIG. 13 is a diagram showing an example of a mode switching operation for a first character. [Figure 7] FIG. 11 is a diagram illustrating an example of a mode database. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In addition, in this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, in this embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be performed on the circuit in the broad sense.

[0012] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. In other words, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the metaverse-related processing system according to this embodiment will be described.

[0014] Fig. 1 is a diagram showing the overall configuration of the metaverse-related processing system 1. Fig. 1 shows an overview of each device included in the metaverse-related processing system 1, users who use each device, and information processed in each device. Each overview will be explained as necessary with reference to other figures.

[0015] The metaverse-related processing system 1 is an information processing system that executes processing related to the metaverse. The metaverse is, for example, a virtual space in which multiple users can participate via terminals connected to a communication line. Most metaverses are three-dimensional virtual spaces, but they may be two-dimensional virtual spaces. Multiple users can participate in the metaverse at the same time. Each user participates in the metaverse using a character (also called an avatar) associated with the user, and performs various activities.

[0016] Users participating in the metaverse include providers who provide various services to other users participating in the metaverse. These providers provide, for example, electronic commerce services, video distribution services, music distribution services, information distribution services, and SNS (Social Networking Service) services. Hereinafter, providers of services in the metaverse are referred to as first users (also referred to as operators), and users of these services are referred to as second users (also referred to as end users).

[0017] The metaverse-related processing system 1 includes a communication line 2, a server device 10, a first user terminal 20, and a second user terminal 30. The communication line 2 includes the Internet and the like, and mediates data exchange between devices connected to the communication line. Each device is connected to the communication line 2 by wire or wirelessly. The server device 10 is an information processing device that executes various processes for providing users with an experience in the metaverse. For example, the server device 10 performs a process for displaying characters that exist in the metaverse.

[0018] The first user terminal 20 is a terminal used by the first user described above, and the second user terminal 30 is a terminal used by the second user described above. The first user terminal 20 and the second user terminal 30 are, for example, smartphones, tablet terminals, or notebook computers. In the example of FIG. 1, the server device 10 performs processing to display a first character (character Z in the example of FIG. 1) associated with the first user Z and second characters (characters A, B, and C in the example of FIG. 1) associated with the second users A, B, and C, respectively, on the terminals of the users, and to cause the characters to interact with each other.

[0019] 2 is a diagram showing a hardware configuration of server device 10. Server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. Bus 14 electrically connects each unit included in server device 10.

[0020] (Control unit 11) The control unit 11 is, for example, a central processing unit (CPU) not shown. The control unit 11 is a computer that realizes various functions related to the metaverse-related processing system 1 by reading out a predetermined program stored in the storage unit 12. That is, information processing by software stored in the storage unit 12 can be specifically realized by the control unit 11, which is an example of hardware, and executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have multiple control units 11 for each function. Also, a combination of these may be used.

[0021] (Storage unit 12) The storage unit 12 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs related to the metaverse-related processing system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program's calculations. The storage unit 12 stores various programs, variables, etc. related to the metaverse-related processing system 1 executed by the control unit 11.

[0022] (Communications Department 13) The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. The communication unit 13 is also configured to be able to receive various electrical signals from the external components to the server device 10. More preferably, the communication unit 13 has a network communication function, and may be implemented so that various information can be communicated between the server device 10 and external devices via the communication line 2.

[0023] 3 is a diagram showing a hardware configuration of the first user terminal 20. The first user terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, and a bus 26. The bus 26 electrically connects the various units included in the first user terminal 20. The control unit 21, the storage unit 22, and the communication unit 23 are similar hardware to the control unit 11, the storage unit 12, and the communication unit 13 shown in FIG. 2, although the specifications, model, and the like may be different.

[0024] (Input section 24) The input unit 24 has one or more input interfaces of keys, buttons, a touch screen, a mouse, etc., and accepts input by the user. The input unit 24 also has an interface for inputting voice, such as a microphone, and accepts voice input by the user.

[0025] (Output section 25) The output unit 25 has an output device such as a display, and displays visual information generated in a manner that is visible to the user, such as a screen, an image, an icon, text, etc., on the display surface. The output unit 25 also has an output device for sounds (including human voices and sounds other than voices), such as a speaker, and outputs those sounds.

[0026] The second user terminal 30 has the same hardware configuration as the first user terminal 20. Note that the control unit of the second user terminal 30 will be described with the reference numeral 31 to distinguish it from the control unit 21 of the first user terminal 20.

[0027] 2. Functional configuration In this section, the functional configuration of the present embodiment will be described. As described above, information processing by software stored in the storage unit of each device is specifically realized by a control unit, which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0028] 4 is a diagram showing an example of the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes a server display unit 111, a user authentication unit 112, a manual operation processing unit 113, an automatic operation processing unit 114, a manual dialogue processing unit 115, an automatic dialogue processing unit 116, a dialogue learning processing unit 117, an operation learning processing unit 118, a mode management unit 119, and a sound processing unit 120. The control unit 21 of the first user terminal 20 includes a user display unit 211, an operation acceptance unit 212, and a sound processing unit 213. The control unit 31 of the second user terminal 30 includes a user display unit 311, an operation acceptance unit 312, and a sound processing unit 313.

[0029] The server display unit 111 executes processing for displaying a system screen related to the metaverse-related processing system 1 on the first user terminal 20. The server display unit 111 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, and displays a web page showing the system screen on the first user terminal 20. The server display unit 111 may also perform processing such as generating and transmitting display data for an application for using the metaverse-related processing system 1.

[0030] The user authentication unit 112 authenticates users who have logged in via the first user terminal 20 and the second user terminal 30, based on authentication information such as a registered user ID (Identification) and password. This authentication enables the server device 10 to process data exchanged with the first user terminal 20 and the second user terminal 30 in association with the user related to the data.

[0031] The manual action processing unit 113 and the automatic action processing unit 114 execute processing for controlling the actions of characters existing in the metaverse. The actions of a character include, for example, walking, running, turning around, crouching, jumping, and making gestures. The manual action processing unit 113 controls the actions of a character associated with a user based on the user's operation. The automatic action processing unit 114 automatically controls the action of a character without the user's operation.

[0032] The manual dialogue processor 115 and the automatic dialogue processor 116 execute processing for controlling dialogue between characters existing in the metaverse. A character's dialogue means communicating with other characters through words, and includes both voice dialogue and text dialogue. The manual dialogue processor 115 controls the dialogue of a character associated with a user based on the user's operation. The automatic dialogue processor 116 automatically controls the dialogue of a character without the user's operation.

[0033] Note that mouth movements, gestures, hand movements, and other actions performed during dialogue may be controlled by the manual action processing unit 113 and the automatic action processing unit 114, but since the control of these actions can also be considered a part of dialogue control, in the following explanation, it is assumed that the manual dialogue processing unit 115 and the automatic dialogue processing unit 116 control these actions associated with dialogue. A character whose action is controlled by the manual action processing unit 113 and the automatic action processing unit 114 and whose dialogue action is controlled by the manual dialogue processing unit 115 and the automatic dialogue processing unit 116 is displayed by the server display unit 111 on each terminal (the first user terminal 20 and the second user terminal 30) as a character existing in the metaverse.

[0034] The automatic control of operations by the automatic operation processing unit 114 and the automatic control of dialogue by the automatic dialogue processing unit 116 are performed by, for example, an AI (Artificial Intelligence) engine. An AI engine is a system having a learning model that is adjusted (tuned) using AI technology so that a predetermined output is obtained for a predetermined input. The "input" here includes input by a person via a terminal (input of characters by keyboard operation, input of operation keys, input of voice, etc.) and input of information by a computer. Furthermore, the "output" includes all kinds of output, such as output of character strings indicating phrases or sentences, output of voice, and output of images.

[0035] The automatic action processor 114 uses an AI engine for action, and the automatic dialogue processor 116 uses an AI engine for dialogue. For example, when an AI engine for action receives information required to determine an action (such as information indicating the positions of surrounding objects and the movements of surrounding characters), it outputs information required for the action (such as the character's movement pattern and amount of movement). When an AI engine for dialogue receives information required to determine the content of the dialogue (such as the content spoken to by another character), it outputs information required for carrying out the dialogue (such as voice or text indicating the content of the statement).

[0036] The dialogue learning processing unit 117 executes processing for learning a learning model (hereinafter referred to as the "first learning model") used by the dialogue AI engine. The action learning processing unit 118 executes processing for learning a learning model (hereinafter referred to as the "second learning model") used by the action AI engine. The learning model is a mechanism for learning the output tendency for input to the AI ​​engine and for making the AI ​​engine output according to the learned tendency.

[0037] The mode management unit 119 manages the character modes. The modes include a manual mode in which the actions, dialogues, etc. are controlled by the user's operation, and an automatic mode in which the actions, dialogues, etc. are controlled by the automatic action processing unit 114 and the automatic dialogue processing unit 116. The mode management unit 119 executes a process of switching between the manual mode and the automatic mode of the character that is the subject of the mode switching when a predetermined switching operation is performed by the user.

[0038] The user display unit 211 controls the display on a display, which is a display means included in the first user terminal 20. The user display unit 211 displays, for example, a web page or an application screen transmitted from the server display unit 111 on the display means. The operation receiving unit 212 receives an operation of the first user.

[0039] The user display unit 311 controls the display on a display, which is a display means included in the second user terminal 30. The user display unit 311 displays, for example, a web page or an application screen transmitted from the server display unit 111 on the display means. The operation receiving unit 312 receives user operations.

[0040] The sound processor 120 of the server device 10, the sound processor 213 of the first user terminal 20, and the sound processor 313 of the second user terminal 30 all perform processing related to sound. The sound processor 213 and the sound processor 313 generate sound data representing a sound input from a microphone and transmit it to the server device 10. The sound processor 120 converts the transmitted sound data into sound data representing a sound audible to a character, and transmits it to a terminal of a user associated with the character. The sound processor 120 also has a voice synthesis engine, and converts a character string into sound data and transmits it to a terminal of a user. The sound processor 213 and the sound processor 313 output the sound represented by the transmitted sound data from a speaker.

[0041] 3. Information Processing The flow of information processing executed by the metaverse-related processing system 1 will be outlined below with reference to Fig. 5. The metaverse-related processing system 1 includes a processor capable of executing a program to perform each step (each process) described below. The metaverse-related processing system 1 executes character control processing to control the actions and dialogue of characters existing in the metaverse.

[0042] Fig. 5 is an activity diagram showing an example of a character control process. The activity shown in Fig. 5 is started when both the first user and the second user are logged in to the metaverse related processing system 1. Either user may log in first. The server device 10 executes a display process for displaying a character corresponding to the logged-in user in the metaverse by the server display unit 111 (A11).

[0043] By executing the display process of A11, the first user terminal 20 displays a first character associated with the first user (character Z for first user Z shown in FIG. 1) in the metaverse by the user display unit 211 (A12). Also, by executing the display process of A11, the second user terminal 30 displays a second character associated with the second user (character A for second user A shown in FIG. 1, character B for second user B, etc.) in the metaverse by the user display unit 311 (A13).

[0044] The first character can be switched between a manual mode in which it is operated by the first user and an automatic mode in which it is automatically controlled without operation by the first user. Fig. 6 is a diagram showing an example of a mode switching operation of a first character. In the example of Fig. 6, the first user terminal 20 displays a metaverse screen G1 on the user display unit 211. A metaverse (virtual space) MV1, a character Z, and a balloon image D1 are displayed on the metaverse screen G1.

[0045] Character Z is a character associated with first user Z shown in FIG. 1. Speech bubble image D1 is displayed when first user Z performs a predetermined operation. The predetermined operation is, for example, an operation of pressing one or more predetermined keys or an operation of uttering a predetermined sound. Speech bubble image D1 includes the character string "Do you want to switch between manual mode and automatic mode?", a Yes button B11, and a No button B12. Speech bubble image D1 is displayed whether the current mode is manual and the mode is to be switched to automatic mode, or whether the current mode is automatic and the mode is to be switched to manual mode.

[0046] If the No button B12 is operated, the user display unit 211 ends the display of the balloon image D1. If the Yes button B11 is operated, the first user terminal 20 accepts the operation via the operation acceptance unit 212, and transmits switching data indicating a switch of the mode of the character Z to the server device 10 (A14). Upon receiving the switching data, the server device 10 switches the mode of the character Z indicated by the received switching data to a mode different from the current mode by the mode management unit 119, and updates it (A15). The mode management unit 119 performs the switching using a mode database that associates characters with modes.

[0047] The server display unit 111 displays the balloon image D1 and the switching image (the Yes button B11 and the No button B12 in the example of FIG. 6) only on the first user terminal 20 used by the first user, and does not display them on the second user terminal 30 used by the second user, even if the character Z is displayed. This makes it possible to prevent the second user from knowing that switching between the manual mode and the automatic mode is being performed.

[0048] Furthermore, the operation of switching between the manual mode and the automatic mode is not limited to the operation shown in Fig. 6. For example, the manual mode and the automatic mode may be switched without displaying an image for switching by simply performing a predetermined operation on the first user terminal 20 in a state in which the character Z is displayed. The predetermined operation is, for example, an operation of double-clicking or double-tapping the character Z. In this case, too, it is possible to make it impossible for the second user to know that the manual mode and the automatic mode are being switched.

[0049] Fig. 7 is a diagram showing an example of a mode database. In the mode database DB1 shown in Fig. 7, a character, a mode, and a start time are stored in association with each other. The character stores the name or identification information of the character. The mode stores "manual" or "automatic" as the character's current mode. The start time stores the time when the current mode started.

[0050] In the example of FIG. 7(a), a first character "character Z", a start time of "dd:hh:mm", and a mode of "manual" are stored in association with each other. Similarly, a first character "character Y", a start time of "dd:hh:mm", and a mode of "automatic" are stored in association with each other in the mode database DB1. Also, in the example of FIG. 7(a), a second character is also stored in the mode database DB1. For example, the second characters A, B, and C shown in FIG. 1 are all stored in association with a mode of "fixed manual".

[0051] For example, when switching data for character Z is transmitted, if the mode associated with character Z is "manual" as shown in Fig. 7(a), the mode management unit 119 updates it to "automatic" as shown in Fig. 7(b), and if it is "automatic" as shown in Fig. 7(b), the mode management unit 119 updates it to "manual" as shown in Fig. 7(a). The mode management unit 119 also updates the start time to the time when the mode is updated.

[0052] Next, a case will be described in which the second user performs an operation on the second character by approaching the first character and talking to the second character. First, the second user terminal 30 receives an operation by the second user instructing the second character to move (A21) through the operation receiving unit 312. The operation receiving unit 312 transmits to the server device 10 an instruction for a movement based on the received operation and movement instruction data indicating the character to move. Upon receiving the movement instruction data, the server device 10 determines, through the mode management unit 119, the mode of the character indicated by the received movement instruction data (A22).

[0053] The mode management unit 119 determines the mode associated with the character indicated by the action instruction data in the mode database DB1 as the mode of the character. Since the second character of the second user who uses the second user terminal 30 is associated with a mode called "fixed manual," the mode management unit 119 determines A22 as the manual mode.

[0054] When the server device 10 determines that the mode is the manual mode, the manual action processing unit 113 performs manual action processing to control the action of the second character associated with the second user based on the operation of the second user (A23). The manual action processing unit 113 generates an image of the second character performing the instructed action based on the instruction of the action of the second character indicated by the action instruction data received at A22. The server display unit 111 generates screen data indicating a screen displaying the generated image of the second character, and transmits the screen data to the second user terminal 30 and the first user terminal 20.

[0055] The second user terminal 30 displays the action of the operated second character by displaying a screen indicated by the transmitted screen data by the user display unit 311 (A24). In detail, the user display unit 311 displays the metaverse seen from a viewpoint corresponding to the second character. Meanwhile, the user display unit 211 of the first user terminal 20 displays the metaverse seen from a viewpoint corresponding to the first character. If the second character is present in the field of view seen from this viewpoint, the user display unit 211 displays the action of the second character by displaying a screen indicated by the transmitted screen data (A25).

[0056] Now, assume that the second character has come close enough to the first character by the operation of A23, and the second user has performed an operation to instruct the second character to have a conversation with the first character. The operation to instruct a conversation can be, for example, an operation to turn the second character toward the first character with whom the second character wants to have a conversation, an operation to input a question such as "Hello" by voice into a microphone, or an operation to input a character string indicating the question into a keyboard or the like.

[0057] When the second user terminal 30 receives a dialogue instruction by these operations through the operation receiving unit 312 and the sound processing unit 313, it generates dialogue instruction data indicating the character to dialogue with and the received instruction, and transmits it to the server device 10 (A31). When the server device 10 receives the dialogue instruction data, it determines the mode of the character indicated by the received dialogue instruction data (the second character in this case) through the mode management unit 119 (A32). The mode management unit 119 determines the mode associated with the character indicated by the dialogue instruction data in the mode database DB1 as the mode of that character.

[0058] In A32, as in A22, the second character is associated with a mode called "fixed manual", so the mode management unit 119 determines that the mode is manual. When the server device 10 determines that the mode is manual, the manual dialogue processing unit 115 executes a process of outputting data related to the dialogue of the second character (hereinafter referred to as "dialogue data") to each terminal as manual dialogue processing.

[0059] For example, the server device 10 stores, for each character, an image of the character with a mouth shape corresponding to the character to be spoken (for example, if the character to be spoken is "a", the mouth shape when "a" is spoken). When an instruction for a dialogue is given by inputting a voice, the manual dialogue processing unit 115 executes a process of instructing the server display unit 111 to recognize a character string indicated by the input voice, read out character images with mouth shapes corresponding to the recognized character strings, and display them by switching them in order, as the manual dialogue processing. When an instruction for a dialogue is given by inputting a character string to a keyboard or the like, the manual dialogue processing unit 115 executes a process of instructing the server display unit 111 to similarly display character images by switching them in order using the input character string, as the manual dialogue processing.

[0060] Furthermore, when an instruction to start a dialogue is given by inputting a voice, the manual dialogue processor 115 executes, as the manual dialogue processing, a process of instructing the sound processor 120 to output the voice as is on the terminal side. At that time, the manual dialogue processor 115 may instruct, for example, to vary the volume of the voice to be output according to the distance between the second character and the first character. Furthermore, when an instruction to start a dialogue is given by inputting a character string into a keyboard or the like, the manual dialogue processor 115 executes, as the manual dialogue processing, a process of instructing the sound processor 120 to output a voice indicating the input character string on the terminal side.

[0061] Furthermore, the manual dialogue processor 115 may convey words uttered by the user through the character not by voice but by an image including a character string. An image including a character string is, for example, an image of a speech bubble including words such as dialogue in a comic, or an image of a chat in which speech bubbles between people are arranged in chronological order. In this case, the manual dialogue processor 115 executes, as the manual dialogue process, a process of instructing the server display unit 111 to generate an image including a character string indicating the input words and display it around the second character.

[0062] The server display unit 111 transmits screen data indicating the interactive actions of the second character to the first user terminal 20 and the second user terminal 30 as instructed by the manual dialogue processing unit 115 (A33). Furthermore, if sound is to be output, the sound processing unit 120 transmits sound data indicating the interactive voice of the second character to the first user terminal 20 and the second user terminal 30 as instructed by the manual dialogue processing unit 115 (A33). These screen data and sound data are dialogue data (data related to the dialogue) by the second character.

[0063] The second user terminal 30 causes the user display unit 311 to display the second character indicated by the transmitted screen data on the display, and causes the sound processing unit 313 to output the sound indicated by the transmitted sound data from the speaker as the sound emitted by the second character. In this way, the second user terminal 30 can start a dialogue from the second character to the first character by outputting the dialogue data of the second character transmitted from the server device 10 (A34).

[0064] The first user terminal 20 causes the user display unit 211 to display the second character indicated by the transmitted screen data on the display, and causes the sound processing unit 213 to output from the speaker the sound indicated by the transmitted sound data as the sound emitted by the second character. In this way, the first user terminal 20 can start a dialogue from the second character to the first character by outputting the dialogue data of the second character transmitted from the server device 10 (A35).

[0065] When a dialogue from the second character to the first character is started, the server device 10 determines the mode of the first character (A41). The mode management unit 119 determines the mode associated with the first character in the mode database DB1 shown in Fig. 7 as the mode of the first character. First, it is assumed that the mode management unit 119 determines the mode as the automatic mode. If the automatic mode is determined, the server device 10 causes the automatic dialogue processing unit 116 to execute an automatic dialogue process for having the first character dialogue with the second character (A42).

[0066] The automatic dialogue processor 116 executes a process of outputting dialogue data for the first character to each terminal as the automatic dialogue process. The automatic dialogue processor 116 generates a character string indicating the content of a response to the content of a question posed by the second character to the first character, i.e., a character string indicated by the dialogue data output for the second character. The automatic dialogue processor 116 has, for example, LLM (Large Language Models), and generates a character string indicating the content of a response to the question using the LLM.

[0067] The automatic dialogue processor 116 executes, as the automatic dialogue processing, a process of instructing the server display unit 111 to display images of the character by switching them in order using the generated character string in the same manner as the manual dialogue processing described above. When dialogue is performed by voice, the automatic dialogue processor 116 executes, as the automatic dialogue processing, a process of instructing the sound processor 120 to output voice indicating the generated character string on the terminal side. When dialogue is performed by an image including a character string, the automatic dialogue processor 116 executes, as the automatic dialogue processing, a process of instructing the server display unit 111 to display an image including the generated character string around the second character.

[0068] The server display unit 111 transmits screen data showing the interactive actions of the first character to the first user terminal 20 and the second user terminal 30, as instructed by the automatic dialogue processing unit 116. Furthermore, when sound is output, the sound processing unit 120 transmits sound data showing the interactive voice of the first character to the first user terminal 20 and the second user terminal 30, as instructed by the automatic dialogue processing unit 116. These screen data and sound data are dialogue data (data related to the dialogue) by the first character.

[0069] The second user terminal 30 causes the user display unit 311 to display the first character indicated by the transmitted screen data on the display, and causes the sound processing unit 313 to output the sound indicated by the transmitted sound data from the speaker as the sound emitted by the first character. In this way, the second user terminal 30 can start a dialogue (response) from the first character to the second character by outputting the dialogue data of the first character transmitted from the server device 10 (A43).

[0070] The first user terminal 20 causes the user display unit 211 to display the first character indicated by the transmitted screen data on the display, and causes the sound processing unit 213 to output the sound indicated by the transmitted sound data from the speaker as the sound emitted by the first character. In this way, the first user terminal 20 can start a dialogue (response) from the first character to the second character by outputting the dialogue data of the second character transmitted from the server device 10 (A44).

[0071] Next, a case where the mode of the first character is determined to be the manual mode in A41 will be described. The manual mode is determined to be the case where the first user is operating the first character via the first user terminal 20. When the second character performs a dialogue action in A35, the first user performs an operation to instruct the first character to respond to the dialogue. Specifically, the first user instructs the second character to return a greeting or answer a question by voice, keyboard input, or the like.

[0072] When the first user terminal 20 receives a dialogue instruction by these operations through the operation receiving unit 212 and the sound processing unit 213, it generates dialogue instruction data indicating the character to dialogue and the received instruction, and transmits it to the server device 10 (A51). The server device 10 executes manual dialogue processing through the manual dialogue processing unit 115 based on the dialogue instruction indicated in the dialogue instruction data transmitted thereto (A52).

[0073] The manual dialogue processor 115 executes manual dialogue processing for having the first character dialogue based on the dialogue instruction of the first character indicated by the dialogue instruction data received in A51, similarly to A33. In A52, similarly to A33, the manual dialogue processor 115 executes processing for transmitting screen data indicating the dialogue action of the first character and sound data indicating the dialogue voice of the first character to the first user terminal 20 and the second user terminal 30 as the manual dialogue processing.

[0074] The second user terminal 30 displays the second character indicated by the transmitted screen data on a display and outputs the sound indicated by the transmitted sound data from a speaker as the sound uttered by the second character (A53). The first user terminal 20 displays the first character indicated by the transmitted screen data on a display and outputs the sound indicated by the transmitted sound data from a speaker as the sound uttered by the first character (A54).

[0075] As described above, in the metaverse-related processing system 1, the server display unit 111, the manual action processing unit 113, and the automatic action processing unit 114 work together to function as an example of a display unit that executes display processing for displaying a character moving in a virtual space. Also, the server display unit 111, the sound processing unit 120, the manual dialogue processing unit 115, and the automatic dialogue processing unit 116 work together to function as an example of a dialogue unit that executes dialogue processing for a character displayed in the display processing to dialogue with another character.

[0076] The dialogue processing includes a first dialogue processing (manual dialogue processing A52 in the example of FIG. 5) for a character to dialogue with other characters based on input from a person (e.g., a first user), and a second dialogue processing (automatic dialogue processing A42 in the example of FIG. 5) for a character to dialogue with other characters without any human input. The second dialogue processing is executed based on instructions from the automatic dialogue processor 116, not on human input.

[0077] When the first dialogue process is executed, the dialogue is performed based on the input of the first user, so the second user feels as if he is dialogue with a human. On the other hand, when the second dialogue process is executed, the dialogue is performed not based on the input of a human, so the second user feels as if he is dialogue with a machine. In the metaverse-related processing system 1, compared to the case where only the second dialogue process is executed, it is possible to provide a time period during which the first dialogue process gives the user the feeling of dialogue with a human, so that the feeling of dialogue with a human through a character can be improved.

[0078] The display process includes a first display process for displaying a character that moves based on input from a person (e.g., a first user), and a second display process for displaying a character that moves without human input. The second display process is executed based on an instruction from the automatic action processing unit 114, not on human input. In the example of Fig. 5, the first display process is executed when the manual mode is determined in A41, and the second display process is executed when the automatic mode is determined in A41.

[0079] The manual dialogue processor 115 executes the first dialogue process when a character is displayed by the first display process. The automatic dialogue processor 116 executes the second dialogue process when a character is displayed by the second display process. Between a manual state in which the first dialogue process and the first display process are executed and the character moves and dialogues in response to human input, and an automatic state in which the second dialogue process and the second display process are executed and the character moves and dialogues in response to machine (server device 10) processing, the manual state tends to give a more human feel than the automatic state, and the automatic state tends to give a more mechanical feel than the manual state.

[0080] For example, when a dialogue is performed by the second dialogue process while the first display process is being executed, the dialogue gives the impression that the user is speaking with a machine despite the human-like movements, which is likely to cause a sense of incongruity due to the difference in the subjects (human or machine) that control the movements and dialogue. Conversely, when a dialogue is performed by the first dialogue process while the second display process is being executed, the dialogue gives the impression that the user is speaking with a human despite the mechanical movements, which is likely to cause a sense of incongruity due to the difference in the subjects that control the movements and dialogue.

[0081] In the metaverse-related processing system 1, as described above, the first dialogue process is executed during the first display process, and the second dialogue process is executed during the second display process, so that the entity that controls the actions and dialogue is the same, and therefore the sense of incongruity between the character's actions and dialogue can be reduced compared to when these entities are not the same.

[0082] In the metaverse-related processing system 1, in order to reduce the sense of incongruity caused by the difference in the entity controlling the dialogue of the first character, the machine is made to learn the tendency of the dialogue by the first user. Specifically, the dialogue learning processing unit 117 functions as an example of a first learning unit that makes the above-mentioned first learning model (a learning model used by an AI engine for dialogue) learn the tendency of the dialogue using the dialogue performed in the first dialogue processing as teacher data. In this case, the server display unit 111, the sound processing unit 120, and the automatic dialogue processing unit 116 (an example of a dialogue unit) execute a process for performing a dialogue using the first learning model as the second dialogue processing.

[0083] The dialogue learning processing unit 117, for example, accumulates training data in which the dialogue content from the second character in the first dialogue process is input and the dialogue content by the first character operated by the first user is output, and makes the first learning model learn the dialogue tendency of the first user using a method such as DL (Deep Learning) using the accumulated training data. Note that the method of learning the dialogue tendency is not limited to this, and any well-known machine learning technique may be used.

[0084] As described above, the dialogue learning processing unit 117 learns the dialogue content of the first character operated by the first user, so that the dialogue by the second dialogue processing (automatic dialogue) can be made to approach the dialogue by the first dialogue processing (manual dialogue), that is, the automatic dialogue of the first character when the first user is absent can be made to approach the manual dialogue input by the first user. Also, the longer the first user engages in dialogue with the first character, the more the learning progresses and the closer the automatic dialogue can be to the manual dialogue.

[0085] The first user who operates the first character is not limited to one person, and may be two or more people. In this case, the dialogue learning processing unit 117 may cause the first learning model to learn the dialogue tendency using all dialogues performed in the first dialogue processing as teacher data, or may cause the first learning model to learn the dialogue tendency using only dialogues by the same user as teacher data. In the latter case, since the first learning model is generated individually for each user, it is possible to have individual dialogues even in automatic dialogues.

[0086] In addition, in the metaverse-related processing system 1, in order to reduce the sense of incongruity caused by the difference in the entity controlling the movement of the first character, the machine is made to learn the tendency of the movement of the first character by the operation of the first user. Specifically, the movement learning processing unit 118 functions as an example of a second learning unit that makes the above-mentioned second learning model (a learning model used by the AI ​​engine for movement) learn the tendency of the movement by using the movement performed in the first display process as teacher data. In this case, the server display unit 111 and the automatic movement processing unit 114 (an example of a display unit) execute a process for performing the movement using the second learning model as the second display process.

[0087] The action learning processing unit 118, for example, receives the position of the first character and the positions of the surrounding characters in the first display process, accumulates teacher data having the operation content of the first character by the first user as an output, and uses the accumulated teacher data to make the second learning model learn the tendency of the action of the first user by a method such as DL. Note that the method of learning the tendency of the action is not limited to this, and any well-known machine learning technique may be used.

[0088] As described above, the action learning processing unit 118 learns the content of the action of the first character operated by the first user, so that the action by the second action process (automatic action) can be made to approach the action by the first action process (manual action), that is, the automatic action of the first character when the first user is absent can be made to approach the manual action input by the first user. Also, the longer the first user moves the first character, the more the learning progresses and the more the automatic action can be made to approach the manual action.

[0089] In addition, when there are two or more first users who operate the first character, the action learning processing unit 118 may learn the movement tendency in the second learning model using all the actions performed in the first action processing as teacher data, or may learn the movement tendency in the second learning model using only the actions by the same user as teacher data. In the latter case, the second learning model is generated individually for each user, so that individual movements can be made even in automatic movements.

[0090] The mode management unit 119 limits the timing at which switching between the manual mode and the automatic mode is possible in A14 shown in Fig. 5. For example, when the automatic dialogue processor 116 is executing the second dialogue process, the mode management unit 119 does not switch to a state in which the manual dialogue processor 115 executes the first dialogue process. Also, when the first dialogue process is being executed, the mode management unit 119 does not switch to a state in which the second dialogue process is executed. In these cases, the mode management unit 119 functions as an example of a dialogue unit.

[0091] For example, the mode management unit 119 does not accept a switching operation in A14 while the second dialogue process is being executed, and accepts a switching operation to the first dialogue process after the execution of the second dialogue process is completed. Also, the mode management unit 119 does not accept a switching operation in A14 while the first dialogue process is being executed, and accepts a switching operation to the second dialogue process after the execution of the first dialogue process is completed. Note that the mode management unit 119 may accept a switching operation after a certain time has elapsed after the end of the first dialogue process and the second dialogue process.

[0092] As described above, even if the subject that controls the operation and the dialogue is unified to either a human or a machine, if the subject switches during the dialogue, the other person in the dialogue is likely to feel uncomfortable. According to the above-mentioned aspect of limiting the timing of switching between the manual mode and the automatic mode, it is possible to make the other person in the dialogue feel less uncomfortable than when a mode switching operation is accepted during the first dialogue process and the second dialogue process.

[0093] Note that the method of limiting the timing of switching between the manual mode and the automatic mode by the mode management unit 119 is not limited to this. For example, the mode management unit 119 (an example of a dialogue unit) may not switch between the first dialogue process and the second dialogue process when other characters are crowded around the first character. In this case, the mode management unit 119, for example, repeatedly calculates the distance between the first character and the other characters, and does not accept a switching operation at A14 shown in FIG. 5 when there are other characters whose calculated distance is less than a threshold.

[0094] Then, when there are no other characters whose calculated distance is less than the threshold, the mode management unit 119 accepts a switching operation in A14 and switches between the manual mode and the automatic mode. Even if a character near the first character does not directly interact with the first character, the character can see and hear the first character's actions and conversation, which may cause a sense of incongruity due to the difference in the subjects that control the actions and the conversation. By restricting the switching timing as described above, it is possible to make it less likely that a user operating a nearby character will feel uncomfortable due to switching between the manual mode and the automatic mode, compared to when the switching timing is not restricted.

[0095] Furthermore, when the above-mentioned switching restriction is performed, the automatic action processing unit 114 may function as an example of a moving unit that, when a predetermined instruction is given in a state in which other characters are crowded around a character, moves the character to eliminate the crowded state. The predetermined instruction is an instruction given while the character is displayed, and may be given by a voice indicating direct content such as "move away from the crowd", or may be given by associating this instruction with a predetermined operation such as double-clicking the first character.

[0096] When a predetermined instruction is given, the automatic action processing unit 114 may temporarily switch control from the manual action processing unit 113 to control by the automatic action processing unit 114, not only in the automatic mode but also in the manual mode, and move the characters so as to eliminate the crowded state. According to such an embodiment, since the restriction on switching is eliminated by giving a predetermined instruction, it is possible to switch to the manual mode more quickly than when movement control by the automatic action processing unit 114 is not performed.

[0097] The configurations (overall configuration, hardware configuration, functional configuration, etc.) shown in FIG. 1 and the like are examples, and other configurations may be used as long as there is no inconvenience in implementation. For example, one device may be distributed to two or more devices, or may be replaced by a cloud computing system. In addition, the functions of one device may be distributed and realized in two or more devices, or the functions of two or more devices may be concentrated and realized in one device. In addition, the operation performed by one function may be distributed to two or more functions, or two or more functions may be integrated into one function. In short, as long as each function required in the entire metaverse-related processing system 1 is realized, any configuration may be used to realize those functions.

[0098] The output destination of information or data (hereinafter referred to as "information, etc.") may be another device, a display, a memory unit (including a built-in memory unit and an external memory unit), etc. Acquisition of information, etc. includes acquiring information, etc. generated by the device itself, in addition to acquiring information, etc. transmitted from another device. Tables, etc. (tables, databases, etc.) that associate parameters are not limited to the tables, etc. shown in the figures, and the number of parameters may be reduced or increased. Furthermore, information, etc. corresponding to parameters may be obtained by a formula, a conditional formula, etc., without using a table, etc.

[0099] The aspect of the above-mentioned embodiment is an information processing system such as the metaverse-related processing system 1 including an information processing device such as the server device 10, but may be an information processing method. The information processing method includes steps of each process executed by the information processing system. The aspect of the above-mentioned embodiment may be a program. The program causes a computer to execute each step (each process) executed by the information processing system.

[0100] <Additional Notes> Furthermore, it may be provided in the following aspects:

[0101] (1) An information processing system comprising a processor, wherein in a display step, the processor executes a display process for displaying a character operating in a virtual space, and in an interaction step, the processor executes an interaction process for the character displayed in the display process to have an interaction with another character, the interaction process including a first interaction process for the character to have the interaction based on human input, and a second interaction process for the character to have the interaction even without human input.

[0102] According to this embodiment, it is possible to improve the feeling of interacting with a person through a character.

[0103] (2) In the information processing system described in (1) above, the display processing includes a first display processing for displaying the character that operates based on human input, and a second display processing for displaying the character that operates without human input, and in the dialogue step, the processor executes the first dialogue processing when a character is displayed by the first display processing, and executes the second dialogue processing when a character is displayed by the second display processing.

[0104] According to this embodiment, it is possible to improve the feeling of interacting with a person through the character.

[0105] (3) An information processing system according to any one of (1) to (2) above, wherein in a first learning step, the processor trains a first learning model to learn a tendency of a dialogue using the dialogue conducted in the first dialogue processing as teacher data, and in the dialogue step, executes a process for conducting a dialogue using the first learning model as the second dialogue processing.

[0106] According to this embodiment, the dialogue through the second dialogue processing can be made closer to the dialogue through the first dialogue processing.

[0107] (4) In the information processing system described in (2) above, in a second learning step, the processor uses the movements performed in the first display process as teacher data to have a second learning model learn the movement trends, and in the display step, executes a process for performing movements using the second learning model as the second display process.

[0108] According to this aspect, the action displayed by the second display process can be made closer to the action displayed by the first display process.

[0109] (5) In the information processing system according to any one of (1) to (4) above, in the dialogue step, when the processor is executing the second dialogue process, the processor does not switch to a state in which the first dialogue process is executed.

[0110] According to this embodiment, it is possible to prevent the other person in the conversation from feeling uncomfortable.

[0111] (6) In the information processing system according to any one of (1) to (5) above, in the dialogue step, the processor does not switch to a state in which the second dialogue process is being executed while the processor is executing the first dialogue process.

[0112] According to this embodiment, it is possible to prevent the other person in the conversation from feeling uncomfortable.

[0113] (7) In the information processing system described in any one of (1) to (6) above, in the dialogue step, the processor does not switch between the first dialogue processing and the second dialogue processing when other characters are crowded around the character.

[0114] According to this aspect, it is possible to prevent users operating surrounding characters from feeling uncomfortable due to switching.

[0115] (8) In the information processing system described in (7) above, in the movement step, when a specified instruction is given when other characters are crowded around the character, the processor moves the character to resolve the situation.

[0116] According to this aspect, it is possible to speed up the switching between the first dialogue process and the second dialogue process.

[0117] (9) A program that causes a computer to execute each step of the information processing system according to any one of (1) to (8) above.

[0118] According to this embodiment, it is possible to improve the feeling of interacting with a person through a character.

[0119] (10) An information processing method comprising the steps of the information processing system according to any one of (1) to (8) above.

[0120] According to this embodiment, it is possible to improve the feeling of interacting with a person through a character. Of course, this is not the case. Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

[0121] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0122] 1: Metaverse related processing system 2: Communication lines 10: Server device 11: Control section 20: First user terminal 21: Control section 30: Second user terminal 31: Control section 111: Server display section 112: User authentication section 113: Manual operation processing section 114: Automatic operation processing unit 115: Manual dialogue processor 116: Automatic dialogue processor 117: First learning processing unit 118: Second learning processing unit 119: Mode Management Unit 120: Sound processing section 211: User display section 212: Operation reception section 213: Sound processing section 311: User display section 312: Operation reception section 313: Sound processing section

Claims

1. An information processing system, a processor, the processor comprising: In the display step, a display process is executed to display a character moving in a virtual space; In the dialogue step, dialogue processing is executed so that the character displayed in the display processing can have a dialogue with another character; The interactive process includes: a first dialogue process for causing the character to carry out the dialogue based on a human input; and a second dialogue process for causing the character to carry out the dialogue without human input. Information processing system.

2. 2. The information processing system according to claim 1, The display process includes: a first display process for displaying the character moving based on a human input; and a second display process for displaying the character that moves without any human input, The processor, In the dialogue step, executes the first dialogue process when a character is displayed by the first display process; executing the second dialogue process when a character is displayed by the second display process; Information processing system.

3. 2. The information processing system according to claim 1, The processor, In the first learning step, a first learning model is made to learn a tendency of a dialogue using the dialogue performed in the first dialogue processing as training data; In the dialogue step, a process for conducting a dialogue using the first learning model is executed as the second dialogue process. Information processing system.

4. 3. The information processing system according to claim 2, The processor, In the second learning step, the movement performed in the first display process is used as training data to cause a second learning model to learn a tendency of the movement; In the display step, a process for performing a movement using the second learning model is executed as the second display process. Information processing system.

5. 2. The information processing system according to claim 1, The processor, In the dialogue step, while the second dialogue process is being executed, switching to a state in which the first dialogue process is being executed is not performed. Information processing system.

6. 2. The information processing system according to claim 1, The processor, In the dialogue step, while the first dialogue process is being executed, switching to a state in which the second dialogue process is being executed is not performed. Information processing system.

7. 2. The information processing system according to claim 1, The processor, In the dialogue step, switching between the first dialogue process and the second dialogue process is not performed in a state in which other characters are crowded around the character. Information processing system.

8. 8. The information processing system according to claim 7, The processor, In the moving step, when a predetermined instruction is given in a state where other characters are crowded around the character, the character is moved so as to eliminate the state. Information processing system.

9. A program, A program for causing a computer to execute each step of the information processing system according to any one of claims 1 to 8.

10. 1. An information processing method, comprising: The information processing system according to any one of claims 1 to 8, Information processing methods.

Images