Program

Abstract

To evaluate a speaker (user) via interactions with an avatar.SOLUTION: A program causes a computer to execute: a first process to visually present an avatar which acts based on a language model on a screen of an output device configured to output a display and voice viewable and audible to a user, and acoustically output a speech as a part of action of the avatar via the output device; a second process to acquire a speech from the user toward the avatar and generate an input to the language model based on the speech acquired from the user; and a third process to evaluate the user. The first process generates, via the avatar, a predetermined event related to an inner state of a person so as to be noticed by the user. The third process evaluates the user on the basis of a reaction of the user to the predetermined event.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a program and the like. [Background technology]

[0002] A method for providing lessons in a virtual environment is known, the method including: storing a plurality of game elements in a memory, each game element being associated with one or more activities within the virtual environment; receiving a request to create a lesson transmitted from a user device over a communications network, the request specifying how one or more selected game elements from the stored game elements are to be incorporated into lesson content; generating a lesson file based on the request, the lesson file including the selected game elements incorporated into the lesson content as specified in the request; and initiating playback of the lesson file in the virtual environment including one or more avatars, the selected game elements being presented for interaction by the avatars within a presentation of the lesson content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-27757 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned conventional techniques, it is difficult to evaluate a conversation partner (user) through a conversation with an avatar.

[0005] Therefore, an object of the present disclosure is to evaluate a conversation partner (user) through a conversation with an avatar. [Means for solving the problem]

[0006] a first process of visually expressing an avatar that moves based on a language model on a screen of an output device that is capable of outputting a display and sound that can be viewed by a user, and acoustically outputting speech as part of the movement of the avatar via the output device; a second process of acquiring an utterance from the user directed to the avatar, and generating an input to the language model based on the acquired utterance from the user; and a third process of evaluating the user. The first process causes a predetermined event related to an internal state of a person to occur via the avatar in a manner that the user can notice; The third process evaluates the user based on the user's reaction to the specified event. Effect of the Invention

[0007] According to the present disclosure, it is possible to evaluate a conversation partner (user) through a conversation with an avatar. [Brief description of the drawings]

[0008] [Figure 1] 1 is a configuration diagram illustrating an example of an information processing device according to an embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of a processing unit of the information processing device illustrated in FIG. 1. [Diagram 3] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 4] FIG. 2 is an explanatory diagram of the present embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 6] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 7] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 8] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 9] FIG. 2 is an explanatory diagram of the present embodiment. [Figure 10]FIG. 2 is an explanatory diagram of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment will be described in detail below with reference to the accompanying drawings.

[0010] FIG. 1 is a configuration diagram showing an example of a system 1 including an information processing device 20 according to the present embodiment.

[0011] The system 1 includes an information processing device 20, an avatar generation unit 40, a large-scale language model (LLM) 42, and a camera 50.

[0012] The information processing device 20 is, for example, a PC (Personal Computer), but may also be realized by a mobile phone, a smartphone, a tablet terminal, etc. The information processing device 20 may also be realized by a combination of a plurality of PCs communicably connected via a network. The information processing device 20 may also be realized by a combination of a PC, a server computer, etc. communicably connected via a network.

[0013] The information processing device 20 realizes a user listening function according to this embodiment. The user listening function is a function for evaluating a user through a conversation (dialogue) with an avatar, and the details thereof will be described later with reference to FIG. 3 and subsequent figures.

[0014] In the example shown in FIG. 1, the information processing device 20 includes a communication unit 21, a storage unit 22, a display unit 23, an input unit 24, and a control unit 25.

[0015] The communication unit 21 includes an interface for wirelessly or wired communication with an external device to transmit and receive information. The communication unit 21 is capable of communicating with the avatar generation unit 40 and the camera 50 via a network.

[0016] The storage unit 22 includes, for example, a primary storage device and a secondary storage device. For example, the storage unit 22 may include a semiconductor memory, a magnetic memory, an optical memory, or the like. The storage unit 22 stores various information and programs related to the user listening function. The information and programs related to the user listening function may be acquired from an external device via an external memory or the communication unit 21. For example, a user listening application program may be acquired from a predetermined application distribution server. Hereinafter, the application program may also be simply referred to as an application. The user listening application may be in the form of a WEB application.

[0017] The display unit 23 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 can display a variety of images. The display unit 23 may be configured as a touch panel, for example. The display unit 23 may be built into a head-mounted display. The display unit 23 may include a speaker (not shown) or a microphone (not shown).

[0018] The input unit 24 may include physical keys, and may further include any input interface including a pointing device such as a mouse.

[0019] The control unit 25 controls the overall operation of the information processing device 20. The control unit 25 executes a user listening application to execute processing related to the user listening function. For example, the control unit 25 outputs a GUI (Graphical User Interface) (described later with reference to FIG. 4) that detects an operation input on the screen of the display unit 23. The control unit 25 receives various inputs by touch operations, input operations, etc. via the input unit 24, and executes processing related to the user listening function in response to the various inputs.

[0020] The avatar generation unit 40 is realized by, for example, a server computer. Note that some or all of the functions of the avatar generation unit 40 may be realized by the information processing device 20.

[0021] The camera 50 captures an image of the user. The camera 50 may have a function for detecting various movements of the user (such as eye movements).

[0022] FIG. 2 is a diagram showing an example of a hardware configuration of the processing unit 10 of the information processing device 20 shown in FIG.

[0023] In the example shown in FIG. 2, the processing unit 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 11A, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14, a drive device 15, and a communication interface 17, all connected by a bus 19, as well as a wired transceiver unit 18A and a wireless transceiver unit 18B connected to the communication interface 17.

[0024] The GPU 11A performs image processing such as drawing various images.

[0025] The auxiliary storage device 14 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software and the like.

[0026] The wired transceiver 18A includes a transceiver capable of communicating using a wired network. Peripheral devices forming the input unit 24, the display unit 23, and the like shown in Fig. 1 are connected to the wired transceiver 18A. However, some or all of the peripheral devices may be connected to the bus 19 or may be connected to the wireless transceiver 18B.

[0027] The wireless transmission / reception unit 18B is a transmission / reception unit capable of communicating using a wireless network. The wireless network may include a wireless communication network of a mobile phone, the Internet, a Virtual Private Network (VPN), a Wide Area Network (WAN), etc. The wireless transmission / reception unit 18B may also include a Near Field Communication (NFC) unit, a Bluetooth (Bluetooth, registered trademark) communication unit, a Wireless-Fidelity (Wi-Fi, registered trademark) transmission / reception unit, an infrared transmission / reception unit, etc.

[0028] The wired transceiver 18A and the wireless transceiver 18B form, for example, the communication unit 21 shown in FIG.

[0029] The processing unit 10 may be connectable to a recording medium 16. The recording medium 16 stores a predetermined program. The program stored in the recording medium 16 is installed in the auxiliary storage device 14 of the processing unit 10 via the drive device 15. The installed predetermined program can be executed by the CPU 11 of the processing unit 10. For example, the recording medium 16 may be a recording medium that records information optically, electrically or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory. The recording medium 16 does not include a carrier wave.

[0030] 3 to 10 are explanatory diagrams of this embodiment.

[0031] The flow of the system may be implemented as shown in FIG.

[0032] The programs in the following disclosures 1 to 14 can be executed by a computer such as an information processing device 20 as shown in FIG.

[0033] As a premise, a user converses with an avatar displayed on the display unit 23 of the information processing device 20 while listening to the avatar. Here, the avatar corresponds to a patient who may have a mental illness, but other assumptions may also be made. The avatar is preferably drawn relatively realistically as a person. This is suitable when the present system 1 is used for training in listening to real people.

[0034] For example, a user may set a scenario regarding an avatar's conversation by making various selections via a scenario selection screen as shown in Fig. 5. This setting may be performed by another user who tests the user. When a scenario or persona is selected (step S302 in Fig. 3), a conversation simulation is performed (step S304 in Fig. 3).

[0035] The avatar operates based on the LLM. An LLM may be prepared for each avatar. The LLM may be constructed through machine learning or the like and updated afterwards. For example, as shown in FIG. 6, a persona and an avatar may be associated with each LLM. In addition, a predetermined event to be generated may also be set in advance for each LLM (avatar). The predetermined event may be an event related to the corresponding persona.

[0036] The avatar generates a predetermined event during a conversation with the user. The predetermined event can be any event as long as it is related to the internal state of the person, but it can be, for example, bowing one's head, trembling of the hands, etc. The predetermined event can occur once or more than once during a series of conversations.

[0037] Typically, the number of conversation partners is one, but there may be two or more at the same time. In this case, each avatar operates based on its own LLM or a common LLM, and the avatars may also converse with each other. Each avatar may generate one or more predetermined events based on its own LLM during a series of conversations.

[0038] Here, a predetermined event occurs via the avatar in a manner that the user can notice.

[0039] 7 shows the screen (user interface) of the display unit 23 during a series of conversations. The avatar is shown in a sitting position at the top left of the screen. Note that the background of the avatar is arbitrary, and the posture, etc. are also arbitrary and may be based on the LLM.

[0040] The LLM is preferably configured to output stage directions together with the speech content. The speech content is what is called a line, and the stage directions include instructions for actions such as "speak while looking down," "speak with trembling lips," and "look up after speaking." In this case, the corresponding avatar can speak according to the speech content and move according to the stage directions based on the output of the language model.

[0041] Furthermore, the LLM may preferably be configured to output at least one of a position, an angle, and a direction of a specific body part together with the speech content. The specific body part is arbitrary and may be the whole body or a part of the body. The angle is a concept including the angle of a joint, etc. For example, in the case of an avatar having joints, the position and angle of each joint may be specified. In this case, the corresponding avatar can also make an utterance according to the speech content based on the output of the language model, and perform a movement according to the specification.

[0042] When the user notices a predetermined event during a series of conversations, the user inputs that fact by, for example, pressing a button, which is an example of the input unit 24. Note that the method of inputting that the user notices a predetermined event may be realized via voice, gestures, or the like.

[0043] In this embodiment, there may be a textbook mode and a practical mode. The textbook mode is a tutorial mode or a practice mode. In this mode, as shown in FIG. 8, immediately after a predetermined event occurs, a query as to whether the predetermined event was noticed may be output. Note that FIG. 8 is a screen (user interface) of the display unit 23 during a series of conversations in the textbook mode, similar to FIG. 7.

[0044] In the practical mode, feedback is given to the user after a series of conversations has ended as to whether or not the user has noticed a predetermined event.

[0045] The length of the series of conversations is arbitrary and may be set appropriately depending on the selection of the scenario, etc.

[0046] In this embodiment, after the series of conversations is completed (after the conversation simulation), the user is evaluated by AI (artificial intelligence) (step S306). Note that the user may be evaluated by other methods (e.g., humans) in addition to or instead of AI.

[0047] The evaluation relates to whether or not the user noticed the predetermined event. Basically, if the user noticed the predetermined event, the evaluation will be higher than if the user did not. Evaluation points may be assigned to each predetermined event according to the level of difficulty, etc. Also, if the user mistook a part that is not a predetermined event for a predetermined event, the evaluation may be lower than if the user did not notice the predetermined event.

[0048] Specifically, the evaluation may be performed as follows. Whether or not the user has noticed the predetermined event may be determined based on at least one of the presence or absence of a predetermined input from the user for the predetermined event and the contents of the utterance from the user following the predetermined event. The predetermined input corresponds to an input indicating that the user has noticed the predetermined event described above. The contents of the utterance from the user following the predetermined event are utterance contents that clearly indicate or suggest that the user has noticed the predetermined event. For example, if the predetermined event is "hand trembling," it may be determined that the user has noticed the predetermined event if the user utters, "My hands are shaking, are you okay?". In addition, the predetermined input may include an input based on the movement of the user, such as eye movement. For example, if the predetermined event is "hand trembling," it may be determined that the user has noticed the predetermined event if the user's gaze is directed toward the trembling hands.

[0049] The evaluation may be realized based on the user's reaction to a specified event, as well as on at least one of the user's movements at times other than the specified event, the content of the user's speech, the attributes of the questions posed by the user, the number or frequency of questions of a particular attribute asked by the user, the intonation of the user's speech, and the manner in which the user conducts the conversation.

[0050] The user's movements may be analyzed based on the user image captured by the camera 50. The user's movements may include the movements of the hands and feet, the movements of the eyes, facial expressions, etc. The user's movements may also include the presence or absence of laughter, etc.

[0051] Here, the attribute of a question may be an attribute that indicates the difference between a free recall question, a focused question, a closed question, a leading question, etc. Also, a question with a specific attribute may be an undesirable question such as a closed question or a leading question, or a preferable question. A question with a specific attribute may be set for each scenario.

[0052] The evaluation screen may be output in the form shown in Fig. 9 or Fig. 10. Fig. 9 is a time series of scores as evaluation results. Fig. 10 is a radar chart of various evaluation items. An interface is generated that allows the user to review the evaluation results. The review may be realized through a video that includes important points such as the points where a specified event occurred.

[0053] Although each embodiment has been described above in detail, the present invention is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of the components of the above-described embodiments.

[0054] In relation to the above embodiments, the following is further noted.

[0055] [Appendix 1] a first process of visually expressing an avatar that moves based on a language model on a screen of an output device that is capable of outputting a display and sound that can be viewed by a user, and acoustically outputting speech as part of the movement of the avatar via the output device; a second process of acquiring an utterance from the user directed to the avatar, and generating an input to the language model based on the acquired utterance from the user; and a third process of evaluating the user. The first process causes a predetermined event related to an internal state of a person to occur via the avatar in a manner that the user can notice; The third process evaluates the user based on the user's reaction to the specified event.

[0056] Regarding Supplementary Note 1, an output device capable of outputting a display viewable by a user can be realized by the display unit 23. Also, an output device capable of outputting audio can be realized by a speaker (not shown) or a microphone (not shown) that the information processing device 20 may be equipped with.

[0057] The first process to the third process can be executed by the control unit 25.

[0058] [Appendix 2] The program according to claim 1, wherein the first process generates the specified event through at least one of a movement of the avatar and a speech content output via the avatar.

[0059] Supplementary Note 2 relates to a method of generating a predetermined event. The method of generating a predetermined event is realized through an avatar. The movement of the avatar is based on the LLM. The avatar may be generated by the avatar generation unit 40, or may be rendered by the GPU based on a generation instruction from the avatar generation unit 40.

[0060] [Appendix 3] The program according to claim 1 or 2, wherein the user's reaction is related to whether or not the user noticed the specified event.

[0061] [Appendix 4] The program described in any one of Appendices 1 to 3, wherein the third process determines whether or not the user has noticed the specified event based on at least one of the presence or absence of a specified input from the user in response to the specified event and the content of an utterance from the user following the specified event.

[0062] [Appendix 5] The third process evaluates the user based on at least one of the following in addition to the user's reaction: the user's movements at timings other than the specified event, the content of the user's speech, the attributes of the questions posed by the user, the number or frequency of questions of a specific attribute posed by the user, the intonation of the user's speech, and the way in which the user conducts a conversation.

[0063] [Appendix 6] The program according to any one of appendices 1 to 5, further comprising a fourth process of generating an interface on the screen that enables the user to reflect on the user's evaluation results.

[0064] [Appendix 7] 7. The program according to claim 1, wherein the avatars are two or more people, each of which is associated with a different language model.

[0065] [Appendix 8] The program according to any one of appendixes 1 to 7, wherein at least one of a different scenario and a different persona is associated with each of the avatars or language models.

[0066] [Appendix 9] The program according to any one of appendices 1 to 8, further comprising a fifth process of acquiring an utterance from one of the avatars, and generating an input for the language model related to an avatar other than the one of the avatars based on the acquired utterance from the one of the avatars.

[0067] [Appendix 10] 10. The program of claim 1, wherein each of the language models is configured to generate the predetermined event via the first process in a different manner.

[0068] [Appendix 11] the language model is configured to output stage directions along with the dialogue; 11. The program of claim 1, wherein the avatar speaks in accordance with the content of the speech based on the output of the language model, and moves in accordance with the stage directions.

[0069] [Appendix 12] the language model is configured to output, together with the speech content, at least one of a position, an angle, and a direction of a specific body part; 12. The program according to any one of claims 1 to 11, wherein the avatar speaks in accordance with the content of the speech based on the output of the language model, and moves in accordance with the specification.

[0070] [Appendix 14] 14. The program of any one of claims 1 to 13, wherein the predetermined event has two or more modes.

[0071] [Appendix 15] The program according to any one of appendices 1 to 14, wherein the third process generates an evaluation result based on artificial intelligence.

[0072] [Appendix 16] generating a predetermined event related to an internal state of a person through an avatar that operates based on a language model during a conversation between the user and the avatar; A program that causes a computer to execute a process of evaluating the user based on the user's reaction to the predetermined event.

[0073] Although each embodiment has been described above in detail, the present invention is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of the components of the above-described embodiments. [Explanation of symbols]

[0074] 1 System

Claims

1. a first process of visually displaying an avatar that moves based on a language model on a screen of an output device that is capable of outputting a display and sound that can be viewed by a user, and acoustically outputting speech as part of the movement of the avatar via the output device; a second process of acquiring an utterance from the user directed to the avatar, and generating an input to the language model based on the acquired utterance from the user; and a third process of evaluating the user. The first process causes a predetermined event related to an internal state of a person to occur via the avatar in a manner that the user can notice; The third process evaluates the user based on a reaction of the user to the predetermined event.

2. The computer-readable medium according to claim 1 , wherein the first process generates the predetermined event through at least one of a movement of the avatar and a speech output via the avatar.

3. The program according to claim 1 , wherein the user's reaction relates to whether or not the user noticed the predetermined event.

4. The program according to claim 3 , wherein the third process determines whether or not the user has noticed the specified event based on at least one of the presence or absence of a specified input from the user in response to the specified event and the content of an utterance from the user following the specified event.

5. The program of claim 1, wherein the third process evaluates the user based on at least one of the user's reaction, the user's movements at timings other than the specified event, the content of the user's speech, the attributes of the questions posed by the user, the number or frequency of questions of a particular attribute posed by the user, the intonation of the user's speech, and the manner in which the user conducts a conversation.

6. The program according to claim 1 , further comprising a fourth process of generating, on the screen, an interface that enables the user to review an evaluation result of the user.

7. The program according to claim 1 , wherein the avatars are two or more people, and each avatar is associated with a different language model.

8. The program according to claim 1 , wherein at least one of a different scenario and a different persona is associated with each of the avatars or language models.

9. The program according to claim 7 or 8, further comprising a fifth process of acquiring an utterance from one of the avatars, and generating an input to the language model related to an avatar other than the one of the avatars based on the acquired utterance from the one of the avatars.

10. The computer-readable medium according to claim 7 or 8, wherein each of the language models is configured to generate the predetermined event via the first process in a different manner.

11. the language model is configured to output stage directions along with the dialogue; The program according to claim 1 , wherein the avatar speaks in accordance with the speech content based on the output of the language model, and moves in accordance with the stage directions.

12. the language model is configured to output, together with the speech content, at least one of a position, an angle, and a direction of a specific body part; The program according to claim 1 , wherein the avatar speaks in accordance with the content of the utterance based on the output of the language model, and moves in accordance with the specification.

13. The program product according to claim 1 , wherein the predetermined event has two or more modes.

14. The program according to claim 1 , wherein the third process generates an evaluation result based on artificial intelligence.

15. generating a predetermined event related to an internal state of a person through an avatar that operates based on a language model during a conversation between the user and the avatar; A program that causes a computer to execute a process of evaluating the user based on the user's reaction to the predetermined event.

Images