Information processing method, information processing device, program, and language ability diagnostic system

Abstract

A user's language ability is diagnosed from multiple perspectives based on the diverse linguistic features contained in the user's multimodal speech data. [Solution] The information processing method is an information processing method executed by an information processing device in a language ability diagnostic system, and includes an acquisition process for acquiring input data including speech by a user, a feature extraction process for extracting features from the acquired input data, and a language ability measurement process for measuring the user's language ability based on an arbitrary index from the extracted features, wherein the language ability measurement process includes a score measurement process for calculating a score, and a basis extraction process for extracting the basis for the score.

Description

[Technical field]

[0001] The present invention relates to an information processing method, a program, and a language ability diagnostic system. [Background technology]

[0002] Cited Document 1 proposes a method for predicting a language's speaking proficiency by machine learning based on phoneme information obtained by speech recognition of a user's speech. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,947,322 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to proceed with second language learning, it is essential to assess the proficiency level of a learner (user) in order to grasp the current state of the learner (user). In second language learning, there are countless items that must be mastered, such as grammar, vocabulary, and pronunciation. However, as in the above-mentioned conventional technology, features (such as pitch, speech rate, and pause length) specialized for measuring specific language abilities (e.g., fluency) are extracted from the user's speech and proficiency is often predicted based on these features. However, in order to measure language abilities from various perspectives (such as consistency and goodness of interaction), it is necessary to consider various language features (linguistic features) contained in the multimodal information (speech data) obtained when the user speaks, so it is difficult to accurately diagnose individual language abilities with only limited features extracted from the speech voice as in the above-mentioned conventional technology. The speech data is also called input data.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to diagnose language ability from multiple perspectives based on various linguistic features contained in a user's multimodal speech data. [Means for solving the problem]

[0006] According to one embodiment, an information processing method is executed by a language ability diagnostic system equipped with an information processing device, and includes an acquisition process for acquiring input data including speech by a user, a feature extraction process for extracting features from the acquired input data, and a measurement process for measuring a score based on an arbitrary index for the speech by the user from the extracted features, wherein the measurement process includes a process for calculating a score and a process for diagnosing the language ability of the user. Effect of the Invention

[0007] According to one embodiment, language ability can be diagnosed based on various linguistic features contained in a user's multimodal input data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system (1000). [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of a language ability diagnosis device (1). [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a language ability diagnosis device. [Diagram 5] FIG. 1 is a diagram illustrating an example of the concept of a language ability diagnosis device. [Figure 6] FIG. 2 illustrates an example of a multi-modal feature extractor. [Figure 7] FIG. 2 illustrates an example of a lexical difficulty feature extractor. [Figure 8] FIG. 2 illustrates an example of a pronunciation feature extractor. [Figure 9] FIG. 2 illustrates an example of a discourse feature extractor. [Figure 10] FIG. 13 is a diagram showing an example of a relationship between discourse markers and classes. [Figure 11]FIG. 13 is a diagram showing an example of a sentence in which a part of a discourse marker is masked. [Figure 12] FIG. 2 illustrates an example of a co-reference feature extractor. [Figure 13] FIG. 13 is a diagram illustrating an example of a coreference relationship. [Figure 14] FIG. 13 is a diagram showing an example of scores for each index calculated by a score measurement unit. [Figure 15] FIG. 1 is a diagram illustrating an example of a concept of normalization. [Figure 16] FIG. 11 is a diagram illustrating an example of a processing flow of a complex feature model. [Figure 17] FIG. 2 illustrates an example process flow for a multi-modal feature encoder. [Figure 18] FIG. 2 illustrates an example of a general process flow for an encoder other than a multi-modal feature encoder. [Figure 19] A figure showing an example of the processing flow of an ensemble model that outputs an overall proficiency level in a complex feature model. [Figure 20] FIG. 11 is a diagram illustrating an example of a processing flow of a simple feature model. [Figure 21] FIG. 13 is a diagram showing an example of visualization using a radar chart. [Figure 22] FIG. 13 is a diagram illustrating an example of visualization of grounds. [Diagram 23] FIG. 13 is a diagram illustrating an example of visualization of grounds. [Figure 24] FIG. 1 is a diagram showing an example of visualization of the CEFR level of vocabulary included in a user utterance. [Diagram 25] FIG. 13 is a diagram illustrating an example of visualization of grammatical error features. [Figure 26] FIG. 13 is a diagram illustrating an example of visualization of discourse features. [Figure 27] FIG. 13 is a diagram showing an example of visualization of pronunciation quality. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. In the description of the specification and drawings of each embodiment, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0010] <System configuration> First, an overview of an information processing system 1000 according to this embodiment will be described. The information processing system 1000 diagnoses the functional language ability of a user who is learning a second language, a third language, or the like, that is, a language learner, by using a technique such as machine learning. In the present invention, functional language ability is the ability to sufficiently and appropriately achieve non-verbal results given by a situation (the ability to sufficiently and appropriately achieve specific tasks in daily life through the language to be learned). Here, functional language ability is also simply referred to as language ability. For more information on functional language abilities, please refer to (Reference 1) F. Kuiken and I. Vedder, “TASK. Journal on Task-Based Language Teaching and Learning”, Volume 2, Number 1, (Netherlands) John Benjamins, Amsterdam, 2022, and (Reference 2) N. Taguchi and Y. Kim, Eds., “Task-Based Approaches to Teaching and Assessing Pragmatics”, Chap. 11, (USA), Longmans Green and Co., 2018, Chap. 11, pp. 266-285.

[0011] The information processing system 1000 is used to help a language learner efficiently learn a second language. The information processing system 1000 can also support the language learner's linguistic performance. A linguistic performance is an actually observable utterance that focuses on the characteristics of the utterance itself (e.g., fluency). For more information on linguistic performance, please refer to References 1 and 2 above.

[0012] Fig. 1 is a diagram showing an example of the configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes a language ability assessment device 1 and a user terminal 2, which are communicably connected to each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination of these. In the example of Fig. 1, the information processing system 1000 includes one each of the language ability assessment device 1 and the user terminal 2, but may include multiple each.

[0013] The language ability diagnosis device 1 is an information processing device that diagnoses a user's language ability based on user input data. The language ability diagnosis device 1 is, for example, a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a robot, but is not limited thereto. In the example of FIG. 1, the language ability diagnosis device 1 is one information processing device, but may be realized as a system consisting of multiple information processing devices connected via a network N. In addition, the language ability diagnosis device 1 may diagnose the user's language ability by inputting an utterance by the user (user utterance) during a series of dialogues (tasks) after the user has finished all of the dialogues. In addition, the language ability diagnosis device 1 may analyze the user's utterance in real time during the user's dialogue.

[0014] The user terminal 2 is an information processing device used by a user, which transmits input data to the language ability diagnosis device 1 and displays the evaluation result of the language ability. The user terminal 2 is, for example, a PC, a smartphone, a tablet terminal, a server device, AR (Augmented Reality) goggles, VR (Virtual Reality) goggles, or a microcomputer, but is not limited thereto. The user terminal 2 may acquire input data from an external device and transmit the acquired input data to the language ability diagnosis device 1. The user terminal 2 may also include a camera and a microphone, and may transmit input data photographed by the camera or input data collected by the microphone to the language ability diagnosis device 1. In this case, the user may photograph or collect the input data himself or herself, or may have a person other than the user photograph or collect the input data himself or herself. The user terminal 2 that transmits the input data and the user terminal 2 that displays the evaluation result may be different information processing devices.

[0015] The input data is information generated by a user, and is any one of text information, audio information, and video information, or a combination of these. The input data will be described later with reference to FIG.

[0016] <Hardware configuration of language ability diagnosis device 1> Next, a description will be given of the hardware configuration of the language ability diagnosis device 1. Fig. 2 is a diagram showing an example of the hardware configuration of the language ability diagnosis device 1. As shown in Fig. 2, the language ability diagnosis device 1 includes a processor 101, a memory 102, a storage 103, a communication I / F 104, an input / output I / F 105, a drive device 106, an input device 107, and an output device 108, which are connected to each other via a bus B1.

[0017] The processor 101 loads into the memory 102 various programs including an OS (Operating System) and a language ability diagnostic program stored in the storage 103 and executes them, thereby controlling each component of the language ability diagnostic device 1 and realizing the functions of the language ability diagnostic device 1. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit) or a DSP (Digital Signal Processor), but is not limited to these.

[0018] The memory 102 is, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), or a combination thereof. The ROM is, for example, a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), or a combination thereof. The RAM is, for example, but is not limited to, a Dynamic RAM (DRAM) or a Static RAM (SRAM).

[0019] The storage 103 stores various programs and data including an OS and a language ability diagnostic program. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM), but is not limited to these.

[0020] The communication I / F 104 is an interface for connecting the language ability diagnosis device 1 to an external device via the network N and controlling communication. The communication I / F 104 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited to these.

[0021] The input / output I / F 105 is an interface for connecting an input device 107 and an output device 108 to the language ability diagnosis apparatus 1. The input device 107 is a device for inputting information to the language ability diagnosis apparatus 1. The input device 107 is, for example, but not limited to, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, an operation button, or a combination of these. The output device 108 is a device for outputting information from the language ability diagnosis apparatus 1. The output device 108 is, for example, but not limited to, a display, a projector, a printer, a speaker, a vibrator, or a combination of these.

[0022] The drive device 106 is a device that reads and writes data to the recording medium 109. The drive device 106 is, for example, but not limited to, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader. The recording medium 109 is, for example, but not limited to, a CD (Compact Disc), a DVD (Digital Versatile Disc), a FD (Floppy Disk), an MO (Magneto-Optical disk), a BD (Blu-ray (registered trademark) Disc), a USB (registered trademark) memory, or an SD card.

[0023] In this embodiment, the language ability diagnostic program may be written into memory 102 or storage 103 during the manufacturing stage of the language ability diagnostic device 1, or may be provided to the language ability diagnostic device 1 via the network N, or may be provided to the language ability diagnostic device 1 via a non-transitory computer-readable recording medium such as a recording medium 109.

[0024] <Hardware configuration of user terminal 2> Next, a description will be given of the hardware configuration of the user terminal 2. Fig. 3 is a diagram showing an example of the hardware configuration of the user terminal 2. As shown in Fig. 3, the user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, a drive device 206, an input device 207, and an output device 208, which are mutually connected via a bus B2.

[0025] The processor 201 controls each component of the user terminal 2 and realizes the functions of the user terminal 2 by expanding the OS and various programs stored in the storage 203 into the memory 202 and executing them. The processor 201 is, for example, a CPU, an MPU, a GPU, an ASIC, or a DSP, but is not limited to these.

[0026] The memory 202 may be, for example, a ROM, a RAM, or a combination thereof. The ROM may be, for example, a PROM, an EPROM, an EEPROM, or a combination thereof. The RAM may be, for example, but is not limited to, a DRAM or an SRAM.

[0027] The storage 203 stores the OS and various programs and data. The storage 203 is, for example, a flash memory, a HDD, an SSD, or an SCM, but is not limited to these.

[0028] The communication I / F 204 is an interface for connecting the user terminal 2 to an external device via the network N and controlling communication. The communication I / F 204 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited to these.

[0029] The input / output I / F 205 is an interface for connecting an input device 207 and an output device 208 to the user terminal 2. The input device 207 is a device for inputting information to the user terminal 2. The input device 207 is, for example, but not limited to, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, an operation button, or a combination of these. The output device 208 is a device for outputting information from the user terminal 2. The output device 208 is, for example, but not limited to, a display, a projector, a printer, a speaker, a vibrator, or a combination of these.

[0030] The drive device 206 is a device that reads and writes data to the recording medium 209. The drive device 206 is, for example, but not limited to, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader. The recording medium 209 is, for example, but not limited to, a CD, a DVD, a FD, an MO, a BD (registered trademark), a USB (registered trademark), or an SD card.

[0031] The output device 208 is a device for outputting information from the user terminal 2. The output device 208 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a photographing device (camera), various sensors, or an operation button, but is not limited to these.

[0032] <Functional configuration of language ability diagnosis device 1> Next, a description will be given of the functional configuration of the language ability diagnosis device 1. Fig. 4 is a diagram showing an example of the functional configuration of the language ability diagnosis device 1. As shown in Fig. 4, the language ability diagnosis device 1 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0033] The communication unit 11 is realized by the communication I / F 104. The communication unit 11 transmits and receives information to and from the user terminal 2 via the network N. The communication unit 11 receives input data from the user terminal 2. In addition, the communication unit 11 transmits information such as a diagnosis result to the user terminal 2.

[0034] The storage unit 12 is realized by the memory 102 and the storage 103. The storage unit 12 stores user information and input data. The storage unit 12 may be constructed by a database.

[0035] The input data is information issued by the user, and is any one of character information, voice information, and video information, or a combination of these. The character information may include a character string converted from the user's speech through a voice recognition system (not shown) or a character string input by the user using an input device 207 such as a keyboard. The character information may also be a text transcribed from a voice issued by a dialogue agent to the user through an output device 208 such as a speaker of the user terminal 2, or a text displayed by a dialogue agent to the user through an output device 208 such as a display of the user terminal 2. The dialogue agent refers to a party that dialogues with the user (learner) instead of a human teacher, and may be a system or a program. The dialogue agent may be, for example, a robot or a three-dimensional character, or may be realized only by a disembodied voice.

[0036] The voice information includes the voice emitted from the vocal cords of the user, the voice of the user input to an input device 207 such as a microphone of the user terminal 2, and the voice emitted by the dialogue agent to the user through an output device 208 such as a speaker of the terminal.

[0037] The video information is video of the user's facial expressions, mouth movements, gestures, and hand movements inputted from an input device 207 such as a camera of the user terminal 2. The video information may also be video of actions shown to the user by the dialogue agent through an output device 208 such as a display of the user terminal 2.

[0038] The input data may be a combination of text information, audio information, and video information obtained by the above-mentioned method. The input data may be dialogue data exchanged among multiple people. The input data may be dialogue (conversation) data between a human and a dialogue agent, or dialogue data between multiple dialogue agents. In this case, the input data may include both text information, audio information, and video information generated by the interaction between the dialogue agent and the user. The input data may also include output data generated by the dialogue agent. The input data is not limited to dialogue data, and may be monologue speech data. The input data may be text information input by the input device 207 such as a keyboard, that is, text information without speech. The input data may also include additional information stored in the storage unit 12, which is information related to the user, and may be, for example, the user's properties or learning objectives.

[0039] Each of the above-mentioned input data may be encrypted. Furthermore, in the input data, character information, audio information, and video information may be arranged in chronological order, but is not limited to this.

[0040] The control unit 13 is realized by the processor 101 reading and executing a language ability diagnostic program from the memory 102 and cooperating with other hardware configurations. The control unit 13 controls the overall operation of the language ability diagnostic device 1. The control unit 13 includes a language feature extraction unit 131, a language ability measurement unit 132, a score measurement unit 133, a reason extraction unit 134, a visualization unit 135, and a translation unit 136.

[0041] The language feature extraction unit 131 will be described with reference to FIG. 5. FIG. 5 is an example of a conceptual diagram of the language ability diagnostic device 1. As illustrated in FIG. 5, the language feature extraction unit 131 receives one or more pieces of character information, audio information, and video information included in the input data, and outputs a specific language feature (feature amount) to the language ability measurement unit 132. The language feature extraction unit 131 includes one or more feature extractors. FIG. 5 illustrates an example in which n (n is an integer equal to or greater than 1) feature extractors are included, but the language feature extraction unit 131 may include one feature extractor. The input data may be subjected to preprocessing such as noise removal and division into multiple data. The language feature includes a text feature, which is a feature amount obtained by converting character information into numerical information. In other words, the language feature is a vector obtained by converting character information, audio information, and video information included in the input data using a statistical method, a machine learning model, or the like.

[0042] The present system is multilingual and includes feature extractors that support multiple languages. FIG. 5 illustrates that feature extractors 1 to n each support multiple languages, such as English and French. Language i, language j, and language k in FIG. 5 are, for example, French, German, and English, respectively. FIG. 5 illustrates that input data including language i (French) and language j (German) is input as is to feature extractor 1 and feature extractor 2, and language k (English) is input to feature extractor n via translation unit 136. This assumes that, for example, a user utterance is in French, and when translation unit 136 has a function of translating from another language (for example, French) to English, only English (language k) is input to feature extractor n. Note that the definition of the language and the feature extractor are not limited to those described above.

[0043] The language feature extraction unit 131 also includes one or more feature extractors for the complex feature model and the simple feature model. The complex feature model and the simple feature model will be described later. Here, the machine learning model is also simply referred to as the model.

[0044] In this system, when the publicly known CEFR (Common European Framework of Reference for Languages: Learning, teaching, assessment) index (perspective) is used, each feature extractor outputs all or any of the text feature, multimodal feature, lexical difficulty feature, grammatical error feature, pronunciation feature, discourse feature, coreference feature, common knowledge feature, and manually designed feature as language features (feature amount). These feature extractors are respectively called text feature extractor, multimodal feature extractor, lexical difficulty feature extractor, grammatical error feature extractor, pronunciation feature extractor, discourse feature extractor, coreference feature extractor, common knowledge feature extractor, and manually designed feature extractor. Feature extractors 1 to n illustrated in FIG. 5 respectively indicate the above extractors, and for example, feature extractor 1 indicates a text feature extractor and feature extractor 2 indicates a multimodal feature extractor. Note that the above-mentioned feature extractors are merely examples, and other features may be extracted (output). Each of these feature extractors will be described.

[0045] A feature extractor that extracts text features, that is, a text feature extractor, will be described. The text feature extractor receives character information of input data as input, and outputs text features as feature quantities. The text feature extractor may be obtained as the output of the final layer or intermediate layer of a pre-trained model (a model trained using a large-scale data set). In this case, the text feature extractor may use the well-known ALBERT (A Lite BERT) as an example of a pre-trained model, and may provide ALBERT with sentence-based character information in a dialogue between a user and a dialogue agent in a time series. In this case, the text feature extractor outputs the output of the final layer of ALBERT (a vector sequence in token units) from the feature extractor. In a language model such as ALBERT, text is processed in units called tokens. A token refers to a word, character, or subword (a unit smaller than a word and larger than a character), which is a component of a sentence, and the definition of a token varies depending on the language model. Hereinafter, the unit of a token will be described as a word.

[0046] Also, as the pre-trained model, for example, the well-known Word2Vec (registered trademark) or BERT (Bidirectional Encoder Representations from Transformers, registered trademark) may be used, and the output of the final layer or intermediate layer of these models may be obtained as the text feature. Word2Vec is a model that learns the meaning and grammatical features of words by a task such as predicting surrounding words from a certain word in a sentence, and when a word is input, it outputs a vector in which the meaning and grammatical functions of the word are embedded. BERT is a bidirectional encoding model with a structure in which multiple layers of Transformer Encoders are stacked, and outputs an embedding vector that reflects the meaning of the word taking into account the context through a task of masking a certain word in a sentence and predicting the masked word from surrounding words, and a task of predicting whether two input sentences are consecutive sentences. ALBERT is a lightweight model of BERT that has been improved by decomposing the matrix of the word embedding representation used in BERT into two smaller matrices.

[0047] The text features may be obtained as the frequency of occurrence of words when the character information of the input data is analyzed using a known method such as TF-IDF (Term Frequency - Inverse Document Frequency). TF-IDF is a statistical measure that indicates how important each word contained in each sentence is within that sentence. Using such a method, the text features output by the text feature extractor are features obtained by converting the character information of the input data into numerical information. A sentence is a unit consisting of one or more words and separated by periods (such as "." or "."). A sentence is a unit consisting of one or more sentences.

[0048] The output of the text feature extractor is numerical information such as a vector or a vector sequence. The text feature may be a multidimensional vector with units such as sentences or words. When the input to the text feature extractor is character information consisting of multiple sentences or words, the text feature may be obtained as a multidimensional vector sequence, and when the entire character information is input, the text feature may be obtained as a single vector.

[0049] A feature extractor that extracts multimodal features, that is, a multimodal feature extractor, will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of a multimodal feature extractor. As shown in Fig. 6, the multimodal feature extractor receives an input of a combination of two or more pieces of text information, audio information, and video information included in the input data, and outputs multimodal features as feature quantities.

[0050] The form of input data to the multimodal feature extractor and the multimodal feature extraction process are explained below. When the input data is a dialogue between a dialogue agent and a user, the input data, i.e., text information, audio information, or video information, may be in a form in which system utterances, which are the utterances of the dialogue agent, and user utterances appear alternately. Figure 6 illustrates this form, specifically showing that system utterance 1, user utterance 1, system utterance 2, ..., user utterance N (N is an integer equal to or greater than 1) occur in that order in the dialogue.

[0051] As shown in FIG. 6, the user's utterance is composed of one or more speech clauses. An utterance clause is a unit that is a position of a punctuation mark (e.g., ",", "?", ".", "!") in character information and a section in audio information where a pause is longer than a certain length (e.g., 0.5 seconds or longer). When dividing a user's utterance, the unit is not limited to an utterance clause. The utterances are input to the multimodal feature extractor in the order in which they were uttered. That is, first, utterance clause 1-1 is input to the multimodal feature extractor, then utterance clause 1-2 is input, and finally, utterance clause N-M is input.

[0052] When speech clause data is input to the multimodal feature extractor in the form of input data as described above, the multimodal feature extractor outputs multimodal features in which the character information, audio information, and video information contained in the speech clause data are blended (interactions between features are taken across modalities). In generating multimodal features, first, text features are extracted from the character information contained in the speech clause data using ALBERT, a well-known pre-trained model. These text features are vector sequences in which information on a word-by-word basis constituting the character information of the speech clause is embedded.

[0053] In addition, audio-visual features are extracted from the combination of audio and video information contained in the speech clause data using AV-HuBERT (Audio-Visual Hidden Unit BERT), a well-known pre-trained model. The audio-visual features are vector sequences in which frame-by-frame information constituting the audio and video information of the speech clause is embedded. AV-HuBERT is an extended model of HuBERT that can extract features from video information in addition to audio information. Using a model in which AV-HuBERT is pre-trained with large-scale audio and video information, the output of the Transformer Encoder in the model is used as a feature that blends video and audio. The Transformer Encoder represents the encoder part of the Transformer model, which is an encoder-decoder model composed only of an attention mechanism and was proposed as a model for machine translation. The multimodal feature extractor then outputs multimodal features that are a blend of features of the three modalities of text information, audio information, and video information by interacting the text features and the audio-visual features through the Co-attention Transformer Layer (described below) of ViLBERT (described below).

[0054] Also, for example, the multimodal feature extractor uses a co-attention transformer layer used in ViLBERT (short for Vision-and-Language BERT), a known machine learning model, to output a multimodal feature in which the features of three modalities of character information, voice information, and video information are blended from a combination of features related to character information output from ALBERT and features related to voice information and video information output from AV-HuBERT. ViLBERT is a model that shows high performance in known tasks such as Visual Question Answering (VQA) and Visual Commonsense Reasoning (VCR: common sense reasoning based on visual information) by performing interaction between image and text features using a co-attention transformer layer (a method of calculating attention between image features and text features). This Visual Question Answering is a task of deriving a correct answer when an image and a question related to the image are presented. Visual Commonsense Reasoning is a task of deriving a correct answer and a reason for a question related to an image and answer options.

[0055] As illustrated in Fig. 6, the multimodal features thus obtained are input to a complex feature model or a simple feature model of the score measurement unit 133. Note that multimodal features may be obtained from a combination of input data other than those described above for the pre-trained model described above. Also, different feature extractors may be used for each of the character information, audio information, and video information to fuse features of different modalities in multiple stages as described above, or multimodal features may be extracted from a single trained model that has been pre-trained with large-scale data including character information, audio information, and video information.

[0056] A feature extractor that extracts lexical difficulty features, that is, a lexical difficulty feature extractor, will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a lexical difficulty feature extractor. The part surrounded by a solid line near the center of Fig. 7 is the lexical difficulty feature extractor, and the part surrounded by a dotted line is the score measurement unit 133. As illustrated in Fig. 7, the lexical difficulty feature extractor receives character information included in the input data as input, and outputs a feature amount that quantifies the difficulty of a term (word or phrase) included in a user utterance as a lexical difficulty feature.

[0057] The form of input data to the lexical difficulty feature extractor and the lexical difficulty feature extraction process are explained below. When the input data is a dialogue between a dialogue agent and a user, the text information included in the input data may be in a form in which system utterances, which are the utterances of the dialogue agent, and user utterances appear alternately. Figure 7 illustrates this form, specifically showing that system utterance 1, user utterance 1, system utterance 2, ..., user utterance N (N is an integer equal to or greater than 1) occur in sequence in the dialogue.

[0058] As shown in Fig. 7, a user's utterance consists of one or more words. To quantify the difficulty of each word, the words are input to the lexical difficulty feature extractor in the order in which they were spoken. That is, first, word 1-1 is input to the multimodal feature extractor, followed by word 1-2, and finally word N-M. Note that the input to the lexical difficulty feature extractor is not limited to words, and may include phrases consisting of one or more words.

[0059] The lexical difficulty feature extractor trains a model that predicts the CEFR level (CEFR level) using Word2Vec, Glove (Global Vectors), or fastText, which are publicly known models that are pre-trained with distributed representations of words (large-scale text or audio). fastText is a publicly known pre-trained model that is trained with large-scale text on the web and is publicly available. Specifically, when input data is input to the lexical difficulty feature extractor in the above-mentioned form, the lexical difficulty feature extractor trains the model using the English Vocabulary Profile (EVP) to which CEFR levels are assigned, or a publicly known Japanese language teaching vocabulary table to which Japanese vocabulary is assigned levels from the early beginner to the late advanced. The above-mentioned English Vocabulary Profile is a dictionary in which CEFR levels are assigned to the meanings of words and phrases.

[0060] As illustrated in FIG. 7, the lexical difficulty feature extractor inputs the output of a model such as fastText into a linear function, passes through the well-known GELU (Gaussian Error Linear Units), and then inputs the result into an activation function, and passes through a Softmax function to output a vector of predicted probabilities (e.g., (0.1, 0.3, ..., 0.2)) corresponding to the CEFR level as a lexical difficulty feature. The predicted probabilities are the probabilities predicted by the model. The lexical difficulty features output by the lexical difficulty feature extractor are input to the score measurement unit 133. For example, the weights of the linear function are pre-trained using the English Vocabulary Profile. In addition, when a word is a polysemous word, the lowest level of the meanings may be trained as the correct level of the word.

[0061] In this way, the feature quantity output by the lexical difficulty feature extractor includes a lexical difficulty feature, which is a feature quantity that quantifies the difficulty of a phrase included in a user's utterance. When a phrase is polysemous and has a different level for each sense (for example, for the English word "support," the sense of "help" is B1, but the sense of "support" is B2), a publicly known word sense disambiguation may be performed to identify the sense and obtain a feature quantity according to the sense. Word sense disambiguation is a task to determine the sense (meaning) of a word or phrase that appears in a sentence in the context. As a method, for example, there is a model such as T5 (Text-to-Text Transfer Transformer) that is trained to generate a correct definition of the sense from the definition list by inputting a sentence that is the context, a target phrase in the sentence to be disambiguated, and a definition list of the phrase. In addition, the predicted probability of the above model or a vector of the intermediate layer output may be obtained as the lexical difficulty feature of the word, or a vector sequence obtained by collecting these vectors in units of turns (the time from when the speaker starts speaking to when the turn of the conversation is passed to the other person) may be obtained as the lexical difficulty feature. Note that the lexical difficulty feature extractor may receive audio information and video information included in the input data as input.

[0062] A feature extractor for extracting grammatical error features, that is, a grammatical error feature extractor, will now be described. The grammatical error feature extractor receives character information included in the input data divided into sentence units, identifies grammatical errors in the user's speech, and outputs a feature that quantifies information about the grammatical errors as a grammatical error feature. For example, the grammatical error feature extractor may use GECToR (Grammatical Error Correction: Tag, Not Rewrite) as a grammatical error corrector and obtain the output (word-by-word vector sequence) of the final layer of RoBERTa (Robustly optimized BERT approach) used inside GECToR as a grammatical error feature. GECToR is a model that solves grammatical error correction as sequence labeling. RoBERTa is a model that improves BERT by introducing a mechanism that dynamically changes the masked words according to the progress of learning in the task of predicting masked words performed in BERT. The grammatical error feature extractor may also receive audio information and video information included in the input data. In this way, the feature that is the output of the grammatical error feature extractor includes grammatical error features that are features that quantify information about grammatical errors.

[0063] A feature extractor that extracts pronunciation features, that is, a pronunciation feature extractor, will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of a pronunciation feature extractor. The part surrounded by a solid line in the center of FIG. 8 is the pronunciation feature extractor, and the part surrounded by a dotted line is the score measurement unit 133. As exemplified in FIG. 8, the pronunciation feature extractor receives speech information included in the input data as input, and outputs a feature amount that quantifies the quality of pronunciation of the speech uttered by the user as a pronunciation feature. As an example, the pronunciation feature is the accuracy of pronunciation, the fluency of pronunciation, intonation and rhythm, and the overall quality of pronunciation.

[0064] The format of the data input to the pronunciation feature extractor and the pronunciation feature extraction process are described below. As with the multimodal feature extractor, the input data is input in the form of speech phrases in the order in which they were spoken.

[0065] When input data is input to the pronunciation feature extractor in the above-described form, the pronunciation feature extractor first extracts text features (word-unit vector strings) from character information included in the input data using ALBERT, a known learning model. The pronunciation feature extractor also extracts a phonetic symbol string (for example, the known ARPABET (ARPAbet) or IPA (International Phonetic Alphabet)) obtained by converting the character information as a feature of the phonemes that should originally be spoken. The extracted phonetic symbol string is converted into a one-hot vector (a vector in which the dimension of the corresponding phonetic symbol is 1 and the rest is 0) in time series, and the entire phonetic symbol string is converted into a vector string and used as a feature (phoneme feature) related to the phonetic symbols.

[0066] The pronunciation feature extractor extracts speech features as features from speech information included in the input data using HuBERT, a known pre-trained model. Note that HuBERT is a model trained with large-scale speech data to perform a task of masking a part of speech and predicting a pseudo-label (a label obtained by clustering) of the masked speech. The pronunciation feature extractor is constructed by training the model shown in FIG. 8 so as to predict the accuracy of pronunciation, the fluency of pronunciation, the quality of intonation and rhythm, and the overall quality of pronunciation from text features, phoneme features, and speech features using a data set, etc., which will be described next. The pronunciation feature extractor may also use video information included in the input data as an input.

[0067] In the above model of the pronunciation feature extractor, for example, the model is trained using the well-known speechocean762 dataset. In the case of speechocean762, 11-level scores are annotated for English speech by non-native speakers. The model is trained to maximize the evaluation index Quadratic Weighted Kappa (QWK; quadratic weighted kappa coefficient) using a loss function (described later) similar to that of the complex feature model, and the intermediate layer output (the output of Self-Attention in FIG. 8) may be obtained as the pronunciation feature.

[0068] As shown in Fig. 8, the pronunciation feature extractor generates pronunciation features from the above-mentioned text features, phoneme features, and speech features through a process described below. Specifically, the extracted pronunciation features are input to a linear function, and after passing through a known GELU, positional information in the sequence is added by positional encoding. Note that positional encoding is a process of adding positional information to an input token sequence in order to uniquely distinguish the position of each token in the sequence.

[0069] Furthermore, the pronunciation feature extractor inputs the output from the Positional Encoding to the Transformer Encoder. Then, the Transformer Encoder interacts with information obtained from the text features, phoneme features, and speech features for each output from the Transformer Encoder. As a result, a vector sequence is output from the Transformer Layer, and this vector sequence is converted into one vector by the well-known Self-Attention. Note that Self-Attention is a method for calculating the degree to which a vector constituting a target vector sequence should be noted with respect to another vector. The pronunciation feature extractor inputs the output from Self-Attention to a linear function and applies a Softmax function.

[0070] The pronunciation feature extractor outputs the predicted probability (label 0 is the worst) of 11-level labels (label 0,..., label 10) annotated in speechcean762 for the accuracy of pronunciation, fluency of pronunciation, intonation and rhythm, and overall quality of pronunciation through the above-mentioned Softmax function. In this system, vectors output from each of the four Self-Attentions illustrated in FIG. 8 are input as pronunciation features to the complex feature model and the simple feature model of the score measurement unit 133. The predicted probability, which is the final output of the model, may be input as a pronunciation feature to the complex feature model and the simple feature model of the score measurement unit 133. In this way, the feature amount output by the pronunciation feature extractor includes the pronunciation feature, which is a feature amount that quantifies the quality of pronunciation of the voice spoken by the user.

[0071] A feature extractor for extracting discourse features, that is, a discourse feature extractor, will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of a discourse feature extractor. The discourse feature extractor receives text information included in a dialogue (dialogue data) between a user and a dialogue agent as input, and outputs discourse features. The discourse features are, for example, feature quantities that quantify prosodic features, lexical features, syntactic features, semantic features, functional features, social features, or stylistic features at the discourse level across multiple sentences or clauses. The discourse feature extractor may output multiple types of discourse features as discourse features. For example, the discourse feature extractor outputs, as discourse features, intermediate layer outputs of a model that predicts discourse markers (discourse markers) that do not explicitly appear between sentences in a user utterance, as shown in FIG. 9.

[0072] FIG. 10 is a diagram showing an example of the relationship between discourse markers and classes. The right column of FIG. 10 shows discourse markers, and the left column shows examples of classes (class names) to which the discourse markers in the same column belong. Discourse markers and classes will be explained with reference to FIG. 10. Discourse markers are signs used to encourage listeners to interpret the meaning of the message a speaker wants to convey, and are classified into textual functions, informational functions, speech attitude functions, and interpersonal relationship adjustment functions, for example. For example, “in addition,” which is an additional function, and “but,” which is an adversative, contradictory, contradictory, and concessive function, are classified into textual functions. The classes of discourse markers are the classifications shown in the left column of FIG. 10. For example, when the discourse marker is “in addition,” the class of the discourse marker is “ADDITION” (additional function). In this system, classes may be defined to cover discourse markers that are frequently used in a large-scale text corpus. In this way, the discourse feature extractor learns the model illustrated in FIG. 9 for each class of discourse markers using a method described below, and outputs the intermediate layer output of each model as a discourse feature.

[0073] The form of input data to the discourse feature extractor and the discourse feature extraction process will be described. The form of input data is a summary of character information of user utterances divided into sentences, which are collected in units of turns. It is to be noted that the input data is not limited to user utterances, but system utterances may also be input, and user utterances and system utterances spanning multiple turns may also be input. The input data may be not only character information, but also audio information or video information. Furthermore, the character information may be character information after grammatical error correction. The unit of division of character information is not limited to sentences, but may be clauses, phrases, or words. The discourse feature extractor may also receive audio information and video information included in the input data, or character information other than dialogue. In this way, the feature amount that is the output of the discourse feature extractor includes discourse features that are feature amounts that quantify linguistic phenomena at the discourse level of the dialogue between the user and the dialogue agent.

[0074] When input data is input to the discourse feature extractor in the above-mentioned form, the discourse feature extractor extracts text features (word-unit vector strings) from character information included in the input data using ALBERT, a known learning model. In the model for acquiring discourse features, as an example, a sentence in which discourse markers (see FIG. 10) that appear in a known text corpus are masked is input. FIG. 11 is a diagram showing an example of a sentence in which discourse markers between certain sentences are masked. A sentence in which a part of a discourse marker is masked will be described with reference to FIG. 11. In this system, the model (the model shown in FIG. 9) is trained to predict the class of the masked discourse marker from the sentences existing before and after the masked part. Specifically, when the discourse marker shown in FIG. 10 is included in the sentence, the discourse marker is masked in this system as shown in FIG. 11. In this way, in the model for acquiring discourse features, the class of the discourse marker is predicted from the sentences existing before and after the masked part (the part surrounded by a square in FIG. 11). In FIG. 11, the masked discourse marker is “for example”, and the model is trained to predict “ILLUSTRATION”, which is the class of the discourse marker “for example”, from the sentences before and after the mask.

[0075] As shown in FIG. 9, the discourse feature extractor processes the discourse features extracted as described above for each class of discourse marker. Specifically, the text features (word-based vector sequence) extracted by ALBERT are input to a linear function, and after passing through a known GELU, positional information in the sequence is added by positional encoding. Furthermore, the pronunciation feature extractor inputs the output from the positional encoding to a transformer encoder, which interacts with the word-based features. This causes the transformer encoder to output a vector sequence, which is then converted into a single vector by known self-attention. The output from self-attention is then input to a linear function, and a sigmoid function is applied.

[0076] The discourse feature extractor outputs the output vector of Self-Attention as the discourse feature. The predicted probability obtained through the above-mentioned Sigmoid function may be used as the discourse feature. This trained model predicts the probability that two consecutive sentences included in the user utterance can be connected by a specific discourse marker. In this system, the predicted probability of the vector output from each of the Self-Attentions illustrated in FIG. 9 is output and input to the complex feature model and the simple feature model of the score measurement unit 133. As an example, in this system, it is assumed that there are 26 discourse marker classes shown in FIG. 10. Specifically, as illustrated in FIG. 9, the vector output from each of the Self-Attentions is, for example, the output vector of the Self-Attention prediction model for the 26 discourse marker classes related to sentence 1-1 and sentences 1-2, 1-3, and 1-4.

[0077] A feature extractor that extracts coreference features, that is, a coreference feature extractor, will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of a coreference feature extractor. The coreference feature extractor receives as input character information contained in a dialogue between a user and a dialogue agent on the same topic, and outputs coreference features. A topic is a subject such as "breakfast" or "travel" defined in a dialogue scenario.

[0078] The coreference relationship will be described with reference to FIG. 13. FIG. 13 is a diagram showing an example of the coreference relationship included in the dialogue between the user and the dialogue agent. In FIG. 13, the column "Speaker" indicates the speaker, and for example, in the case of "System", the speaker is the dialogue agent, and in the case of "User", the speaker is the user. The column "Sentence" indicates the sentence uttered by the speaker in the same row. The column "ID" indicates the identifier of the sentence written in "Sentence" in the same row. For example, the sentence with "ID" of "3-2-1" is "Tell me about your house.", which indicates that this sentence was uttered by the dialogue agent. Here, "ID" is composed of a topic number-turn number-sentence number, but is not limited to this. Note that the character information of the user's utterance to be input may be character information after grammatical error correction.

[0079] In FIG. 13, "your" with the code A in the utterance by the dialogue agent and "m(M)y" with the code A in the utterance by the user refer to the same object. Also, in FIG. 13, "your house" with the code B in the utterance by the dialogue agent and "My house" with the code B in the utterance by the user refer to the same object. Also, in FIG. 13, "my room" in two places with the code C in the utterance by the user refer to the same object. Such a pair of words that refer to the same object is called a coreference relationship. The coreference feature extractor identifies pairs of words that are in a coreference relationship (codes A, B, and C in FIG. 13) and outputs a feature that quantifies the identified coreference relationship as a coreference feature.

[0080] The form of input data to the coreference feature extractor and the coreference feature extractor are described below. Input data to the coreference feature extractor is a combination of system utterances and user utterances on a topic-by-topic basis, as shown in FIG.

[0081] The coreference feature extractor uses a data set annotated with coreference relationships to train a coreference analysis model (e.g., LingMess) that estimates coreference relationships between phrases. The well-known LingMess is a model that learns a function for determining whether a coreference relationship exists between two phrases that appear in a sentence for each category, such as the relationship between pronouns and pronouns or the relationship between proper nouns and pronouns, and uses a different function depending on which category the two phrases belong to during inference, thereby achieving high coreference analysis performance. The coreference feature extractor outputs coreference features using this trained coreference analysis model. The coreference feature extractor may also receive audio information and video information included in the input data as input. In this way, the feature amount that is the output of the coreference feature extractor includes coreference features that are features obtained by identifying the coreference relationship between phrases in a dialogue between a user and a dialogue agent.

[0082] Furthermore, the coreference feature extractor may receive as input the text information of the user's utterance after grammatical error correction, in addition to the text information transcribed from the dialogue agent, and output a set of embedded representations (e.g., the output of the final layer of the well-known Longformer used inside LingMess) of phrases (word strings) in a coreference relationship identified by the coreference analysis model described above, in order of appearance from earliest to latest. An embedded representation is a representation of symbols such as words as a high-dimensional real-valued vector. Note that Longformer is a model that has been improved to be able to process longer sentences compared to BERT and the like.

[0083] Here, the above-mentioned LingMess uses the word embedding representation of Longformer. Specifically, in the output of LingMess, after identifying words that have a coreference relationship, such as the symbols A, B, and C shown in Figure 13, LingMess obtains the word embedding representation of the corresponding words from the Longformer in LingMess. At this time, if a phrase is composed of multiple words, the embedding representation of each word that composes the phrase is summed to create a single vector representation.

[0084] A feature extractor that extracts common sense features, that is, a common sense feature extractor, will be described. The common sense feature extractor receives character information of a user utterance as input and outputs common sense features. Common sense features are features obtained when a person with common sense infers the intention and personality of a subject from a user utterance. For example, when there is an utterance of "I went to a convenience store," it is inferred that the subject's intention is "There was something I wanted to buy." Also, when there is an utterance of "I made coffee for a tired colleague," it is inferred that the subject's personality is "I am considerate." Note that the common sense feature extractor may receive the character information of a user utterance as it is (without processing), or may receive character information that has been subjected to grammatical error correction as a preprocessing step and divided into sentence units as input.

[0085] For example, the common sense feature extractor uses a data set annotated with the above-mentioned inference (inference) relationship to train a common sense inference model such as COMET, and outputs common sense features using the pre-trained common sense inference model. In this way, the feature value output by the common sense feature extractor is a feature value obtained when a person with common sense infers the intention and personality of a subject person from a user's utterance. As an example, the common sense feature extractor may train a common sense inference model using a knowledge graph such as ATOMIC as a data set, and output the intermediate layer output of the trained common sense inference model as a common sense feature. Note that ATOMIC is a database that describes common sense inference knowledge in everyday life in an if-then relationship, such as when X and Y have a conversation, if X compliments Y, Y is likely to return a compliment to X. Also, COMET is a model that trains a Transformer using data such as ATOMIC so that it outputs a then text (Y will return a compliment to X) in response to an input of an if text (X complimented Y).

[0086] A feature extractor that extracts manually-designed features, that is, a manually-designed feature extractor, will be described. The manually-designed feature extractor receives as input the utterance of a dialogue agent in each turn and the text information and speech information of the user's utterance in response to the utterance, and outputs manually-designed features. The manually-designed features are features designed based on human expert knowledge. As an example, the manually-designed feature extractor obtains manually-designed features based on knowledge of second language acquisition research. For example, for fluency, the length of the utterance, the length of a blank space (pause), the number of words spoken per second, and the number of fillers (e.g., "uh", "um") are extracted, and manually-designed features are obtained based on this. For vocabulary richness, for example, a dictionary in which levels are assigned to words is prepared, and the frequency and ratio of how many words of each level were included in the user's utterance are extracted, and manually-designed features are obtained based on this. For coherence, for example, the frequency and ratio of discourse markers that appear in the user's utterance are calculated, and manually-designed features are obtained based on this. In this way, the feature quantities output by the manual design feature extractor include manual design features, which are features designed based on human expert knowledge.

[0087] When outputting multilingual language features, the language feature extraction unit 131 may have a different feature extractor for each language and extract language features using input data in multiple languages ​​as input, or may extract language features from a common feature extractor using a model pre-trained with a large-scale multilingual data set. In this case, the input data may be data in a single language or data in which multiple languages ​​are mixed. For example, when it is desired to measure the ability to switch between multiple languages ​​depending on the situation (plurilingualism), a situation in which multiple languages ​​are mixed within the same dialogue task may be assumed.

[0088] Returning to Fig. 4, the language ability measurement unit 132 will be described. Referring to Fig. 5, the language ability measurement unit 132 receives the language features extracted by the language feature extraction unit 131 as input, and outputs a score and the basis of the score. The score is a numerical representation of the user's language ability in terms of a specific index (e.g., richness of vocabulary, correctness of grammar, fluency), and is defined in advance by a service provider or a system designer, but is not limited to this. The index and score will be described in detail later with reference to Fig. 14.

[0089] The language ability measurement unit 132 includes a score measurement unit 133 and a reason extraction unit 134. The score measurement unit 133 receives the language features extracted by the language feature extraction unit 131 as input and outputs a score. For example, the language ability measurement unit 132 inputs the output value of the language feature extraction unit 131 to a machine learning model for score measurement (score measurement machine learning model) to perform score measurement based on a predetermined index. As illustrated in FIG. 5, in the language ability measurement unit 132, a learning and evaluation process is executed between a human evaluator, a complex feature model, and a simple feature model, thereby improving the prediction accuracy of the score measurement machine learning model. Note that the algorithm for converting the language features into a score may be a statistical method or may be rule-based.

[0090] Before describing the functional configuration of the score measurement unit 133, the scores for each index calculated by the score measurement unit 133 will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the scores for each index calculated by the score measurement unit 133. The upper part of Fig. 14 shows the indexes, and the lower part shows the user's scores for each index. When CEFR is used as an index in this system, the score measurement unit 133 calculates the user's scores for seven indexes, namely, vocabulary richness (Range), grammatical accuracy (Accuracy), fluency (Fluency), pronunciation (Phonology), interaction (Interaction), coherence (Coherence), and overall proficiency (Overall), as shown in Fig. 14 as an example.

[0091] As illustrated in the lower part of FIG. 14, the score measurement unit 133 obtains a CEFR level (discrete score) and a continuous score (in parentheses in FIG. 14) as a score. In this system, as an example, the CEFR level is expressed as one of the discrete scores A1, A2, B1, B2, C1, and C2. This description is based on the description of the known CEFR level. A1 is the lowest CEFR level, and C2 is the highest CEFR level, and A1 to C2 indicate high proficiency in the indicator. As an example, the continuous score is expressed as a number between 0 and 6 (two decimal places), and 0 to 6 indicate high proficiency. Note that the above continuous score is only an example and is not limited to two decimal places.

[0092] There is a relationship between the CEFR level and the continuous score, and the CEFR level is defined as A1 when the continuous score is between 0.00 and 1.00, A2 when the continuous score is between 1.01 and 2.00, B1 when the continuous score is between 2.01 and 3.00, B2 when the continuous score is between 3.01 and 4.00, C1 when the continuous score is between 4.01 and 5.00, and C2 when the continuous score is between 5.01 and 6.00, but not limited thereto. Figure 14 shows that the user's CEFR level (discrete score) for the indicator of vocabulary richness is C2, and the discrete score is 5.54.

[0093] In addition, in this system, the language ability measurement unit 132 may output both the CEFR level and the discrete score, or may output either one of them. Also, the score output by the language ability measurement unit 132 may be a continuous value for a regression problem, a probability for a classification problem, or a discrete value indicating a specific level or category ID. For example, when using CEFR levels as an indicator, A1 can be defined as 0 to 1, A2 as 1 to 2, ..., and C2 as 5 to 6, and a continuous value such as 1.2 annotated by humans can be used as the correct answer to estimate the continuous value using a regression model. Alternatively, discrete labels such as A1, A2, ..., and C2 annotated by humans can be represented as one-hot vectors such as A1=[1,0,0,0,0,0], A2=[0,1,0,0,0,0], ..., C2=[0,0,0,0,0,1], the probability of each label can be predicted using a Softmax function, and the CEFR level with the highest probability (the CEFR level with the highest probability among A1, A2, B1, B2, C1, and C2) can be used as the score.

[0094] Furthermore, the language ability measurement unit 132 may obtain a score by decomposing a multi-class classification problem into a known One-vs-Rest binary class classification problem (e.g., A1 and other CEFR levels). Also, the multi-class classification problem may be decomposed into a known One-vs-One binary class classification problem (e.g., A1 and A2). Also, when a machine learning technique is used for the algorithm, a configuration may be used in which a model is provided for each indicator, or a configuration may be used in which scores for multiple indicators are output by a single model through multi-task learning. Furthermore, the final score is not limited to the output of one trained model, but may be the output of a model that is trained in ensemble learning using the output of that model as the input of yet another model.

[0095] In addition, in this system, the index is not limited to CEFR, and may be the degree of intimacy (rapport) between the interlocutor and the user. In this case, the interlocutor may be a human being, or may be an agent or robot equipped with a program. In addition, the index may be the user's personality characteristics, such as the well-known Big Five (five-factor theory of traits).

[0096] The functional configuration of the score measurement unit 133 will be described with reference to Fig. 5. As illustrated in Fig. 5, the score measurement unit 133 includes a complex feature model and a simple feature model as machine learning models for score measurement, and further includes a score converter.

[0097] The complex feature model is a machine learning model that predicts a score using various language features extracted by the language feature extraction unit 131. As illustrated in FIG. 5, in this system, a human evaluator checks whether the output of the score measurement unit 133, that is, the output of the complex feature model and the simple feature model, is correct. As an example, the human evaluator checks a video of a conversation between a user and a dialogue agent, and annotates a CEFR level of A1 to C2 as a score for each of the seven CEFR indicators (vocabulary richness, grammatical accuracy, fluency, good pronunciation, good interaction, consistency, and overall proficiency). A continuous score may be annotated for each of the seven CEFR indicators described above. The human evaluator may also annotate only the output of a specific indicator.

[0098] Here, the confidence level is defined as a scale that quantifies how reliable the output of the score measurement unit 133 is. The confidence level may be calculated by a known method, and a pair of output and input data with a low confidence level may be preferentially checked by a human evaluator. As an example, a probability distribution when the prediction (prediction result) by the complex feature model is correct and a probability distribution when it is not correct may be obtained, and the confidence level may be calculated based on the ratio of the degree to which the new prediction (prediction result) belongs to either probability distribution. Also, as illustrated in FIG. 5, in this system, a human evaluator checks the output of the complex feature model and the simple feature model, but the output of the complex feature model or the simple feature model may be regarded as the correct answer without providing (intervening) a human evaluator.

[0099] The learning of the complex feature model will now be described. As described above, the initial model of the complex feature model, which is the first cycle, is trained to predict the CEFR level (A1 to C2; correct answer) annotated by a human evaluator to the dialogue data between a user and a dialogue agent. In other words, the initial model of the complex feature model is trained with learning data (data set) on which a human evaluator has checked the input data and annotated the correct answer.

[0100] From the second cycle onwards, if an error is found in the output of the complex feature model in the confirmation process by the human evaluator, the human evaluator annotates the correct score and adds it to the training data. Note that the annotation data added to the training data is not limited to annotation data with the error corrected, and even if a human checks the output of the complex feature model and the result is correct, the annotation data may be newly added to the training data (dataset) as a score evaluated as correct by the human. The complex feature model is retrained using the newly added training data. In the retraining, the trained model may be fine-tuned using only the newly annotated data, or the model may be retrained from scratch using a dataset in which the newly annotated data is added to the existing training data.

[0101] In this system, the complex feature model and the simple feature model (hereinafter, the score measurement machine learning model) are trained to predict the CEFR level annotated by a human evaluator from input data using a loss function. This loss function will be described. In the score measurement machine learning model, training is performed to predict the CEFR level annotated by a human evaluator using a loss function. In this system, a cost-conscious loss function (LCS) having a known penalty term is introduced as the loss function. This is because the penalty is heavier when the output of the score measurement machine learning model is C2 for the correct answer A1 than when the output of the score measurement machine learning model is A2. The cost-conscious loss function will be described using Formulas 1 to 4. The overall loss function (L) is calculated using Formula 1.

[0102]

number

[0103]

number

[0104]

number

[0105]

number

[0106] In this system, a machine learning model for score measurement is trained using the loss functions calculated using Formulas 1 to 4. The trained model outputs each prediction probability A1 to C2 for the input data so that the sum is 1. In this system, as an example, this weighted sum is used as the score.

[0107] In this system, QWK is predefined as an evaluation index, and the score measurement unit 133 optimizes the boundaries when converting the above scores into CEFR levels, i.e., the boundary between A1 and A2, the boundary between A2 and B1, ..., and the boundary between C1 and C2, so as to maximize QWK (so as to best match the classification criteria of the human evaluator).

[0108] FIG. 15 is a diagram showing an example of the concept of a method for normalizing scores in a state where the intervals between score boundaries are unequal after the score boundaries have been optimized so that the intervals between the boundaries are equal. The normalization method in this system will be described with reference to FIG. 15. The upper part of FIG. 15 illustrates an example of the relationship between the CEFR level and the continuous value score after the score boundaries have been adjusted to be optimal (before normalization), and the lower part illustrates an example of the relationship between the CEFR level and the continuous value score after normalization. The relationship between the CEFR level and the continuous value score after normalization in the lower part of FIG. 15 has been normalized so that the intervals between the level boundaries are equal, compared to the relationship between the CEFR level and the continuous value score before normalization in the upper part.

[0109] Also, A1 to C2 in FIG. 15 are CEFR levels. Also, "a" in FIG. 15 is the boundary value (continuous value) between B1 and B2 before normalization, and "b" is the boundary value between B2 and C1 before normalization. Also, "a'" in FIG. 15 is the boundary value between B1 and B2 after normalization, and "b'" is the boundary value between B2 and C1 after normalization. Also, "1.00" and "2.10" in FIG. 15 illustrate the boundary values ​​of each CEFR level before and after normalization. Also, "x" in FIG. 15 is the score before normalization, and "x'" is the score after normalization. The user is presented with the score of "x'" after normalization.

[0110] The probability vector output from the score measurement machine learning model is converted into a score, and the imbalance boundary value is normalized based on Equations 5 to 7.

[0111]

number

[0112]

number

[0113]

number

[0114] In this system, as an example, the score measurement unit 133 compares the output of the ensemble model with the output of the core model (configuration from the encoder of each language feature in FIG. 16 to the Softmax function) for each CEFR index, and adopts the output with the higher performance. The ensemble model is a model that combines the outputs of each index of the core model and performs publicly known ensemble learning. As an example, the ensemble model and the core model of the complex feature model are trained on the training set after dividing a data set annotated by a human evaluator into a training set and a test set, and the performance (QWK) is evaluated on the test set. The QWK for each CEFR index on this test set is compared between the ensemble model and the core model, and for example, when the QWK of the ensemble model is larger than the QWK of the core model for the index of vocabulary richness during operation, the output of the ensemble model is adopted for the index of vocabulary richness during operation. In addition, when the QWK of the core model is larger than the QWK of the ensemble model, the output of the core model is used as the fluency index during operation. Note that this is only an example, and the above embodiment is not limited to the above.

[0115] In this system, based on the well-known Teacher-Student learning, the complex feature model is the Teacher model, the simple feature model is the Student model, and the simple feature model (Student model) is trained so as to reproduce (imitate, approximate) the output of the complex feature model (Teacher model). Note that the simple feature model (Student model) is lighter than the complex feature model (Teacher model). In other words, the simple feature model has fewer parameters than the complex feature model, consumes less memory during execution, and takes less processing time from inputting features to obtaining output. Also, the features input to the simple feature model are not necessarily the same as those in the complex feature model, and features output from a feature extractor for the simple feature model may be input.

[0116] Fig. 16 is a diagram showing an example of the concept of the complex feature model of the score measurement unit 133. An example of the concept of the complex feature model of the score measurement unit 133 will be described with reference to Fig. 16. Fig. 16 illustrates an example of a complex feature model including an ensemble model in which outputs from each encoder encoding various linguistic features are input to a known Transformer Encoder, interactions between different types of linguistic features output from each encoder are taken in the Transformer Encoder, and then the outputs are passed through a network (Self-Attention and a linear function) specialized for predicting each CEFR index, and the output of a core model having a structure for predicting the CEFR level of each index is received, and each index is predicted again.

[0117] When CEFR is used as an indicator in this system, as an example, the complex feature model includes various encoders, such as an Interactive Text Feature Encoder, a Multimodal Feature Encoder, a Vocabulary Difficulty Feature Encoder, a Grammarly Error Feature Encoder, a Pronunciation Feature Encoder, a Discourse Feature Encoder, a Coreference Feature Encoder, a Commonsense Feature Encoder, and an SLA Inspired Feature Encoder.

[0118] Each encoder of the score measurement unit 133 corresponds to each of the various feature extractors included in the language feature extraction unit 131, and for example, the output from the feature extractor that outputs a text feature is input to a text feature encoder. Each of the various encoders is a network that receives the language features output by the various feature extractors of the language feature extraction unit 131, encodes each language feature, and outputs a vector sequence. Each of these encoders will be described with reference to Figs. 17 and 18.

[0119] The multimodal feature encoder will be described with reference to FIG. 17. FIG. 17 is a diagram showing an example of a process flow of the multimodal feature encoder. The part surrounded by a dotted line is the multimodal feature extractor of the language feature extraction unit 131, and the part surrounded by a solid line is the multimodal feature encoder (see FIG. 16) of the complex feature model in the score measurement unit 133 (language ability measurement unit 132). As described above with reference to FIG. 6, the multimodal feature extractor inputs the feature amounts related to the input data (text information, audio information, and video information) divided into speech clause units in a time series to the multimodal feature encoder. The output from the top-level Transformer Encoder illustrated in FIG. 17 is input to the Transformer Encoder of the core model of the complex feature model illustrated in FIG. 16. The weights of the network of the multimodal feature encoder are updated during the learning of the complex feature model.

[0120] As illustrated in FIG. 17, the multimodal feature encoder inputs features related to character information acquired by ALBERT in the multimodal feature extractor of the language feature extraction unit 131 (text features, i.e., vector strings in units of words) and features related to audio information and video information acquired by AV-HuBERT (audio-visual features, i.e., vector strings in units of frames) into a linear function. Here, the text features and audio-visual features are converted into vectors of the same dimension. After that, through GELU, which is a known activation function, a vector string in which the text features and audio-visual features interact with each other is obtained by the above-mentioned Co-attention Transformer Layer and a Transformer Encoder provided in the upper layer.

[0121] After the above process, the vector sequence obtained from the lower-level Transformer Encoder in FIG. 17 is converted into one vector using well-known Self-Attention. Through the processing up to this point, a vector is obtained in which three modal pieces of information, namely, character information, audio information, and video information of the input speech clause, are fused. After that, the vectors generated from each piece of utterance data are input to the top-level Transformer Encoder in FIG. 17 in a time series manner while maintaining the order using Positional Encoding. The multi-modal feature encoder outputs a vector sequence in which features obtained from different speech clauses (vectors in which character information, audio information, and video information of the speech clause are embedded) have interacted with each other after processing up to this top-level Transformer Encoder.

[0122] FIG. 18 is a diagram showing an example of the process flow of an encoder other than the multimodal feature encoder described above. In FIG. 18, the part surrounded by a dotted line is a feature extractor other than the multimodal feature extractor in the language feature extractor unit 131, that is, a text feature extractor, a lexical difficulty feature extractor, a grammatical error feature extractor, a pronunciation feature extractor, a discourse feature extractor, a coreference feature extractor, a common sense feature extractor, and a manually designed feature extractor. The part surrounded by a solid line is a configuration of an encoder other than the multimodal feature encoder. These encoders are respectively encoders other than the multimodal feature encoder in the score measurement unit 133 (language ability measurement unit 132), that is, a text feature encoder, a lexical difficulty feature encoder, a grammatical error feature encoder, a pronunciation feature encoder, a lexical difficulty feature encoder, a discourse feature encoder, a coreference feature encoder, a common sense feature encoder, and a manually designed feature encoder.

[0123] In this system, interaction between features of the same type extracted from the same feature extractor is performed inside each encoder. Specifically, interaction between features of the same type is performed by the top-level Transformer Encoder illustrated in Fig. 17 if it is a multimodal feature, and by the Transformer Encoder illustrated in Fig. 18 if it is another feature. In addition, interaction between different types of features extracted from different feature extractors is performed by the Transformer Encoder of the core model of the complex feature model in Fig. 16.

[0124] As illustrated in FIG. 18, the encoders other than the multimodal feature encoder, i.e., the text feature encoder, the lexical difficulty feature encoder, the grammatical error feature encoder, the pronunciation feature encoder, the discourse feature encoder, the coreference feature encoder, the common knowledge feature encoder, and the manually designed feature encoder, each receive data on features from the corresponding feature extractor and input them to a linear function. Then, the vector sequence is input to a known Self-Attention via a known GELU activation function. Then, the vector output from the Self-Attention is input to a Transformer Encoder after being given position information by Positional Encoding. Note that the output from the Transformer Encoder illustrated in FIG. 18 is input to the Transformer Encoder of the core model of the complex feature model illustrated in FIG. 16. The network weights of each encoder are updated during the learning of the complex feature model. Each encoder other than the multimodal feature encoder will be described.

[0125] As described above, the text feature encoder receives the text features output from the text feature extractor (specifically, a set of word vector sequences obtained from all sentences, which are obtained by inputting sentences contained in the system utterance and user utterance into the text feature extractor (output of ALBERT)), inputs the vector sequences into a linear function in chronological order, and after completing processing up to Self-Attention, adds positional information by Positional Encoding, and interactions between features are performed in the Transformer Encoder. The weights of the text feature encoder's network are updated during learning of the complex feature model.

[0126] As described above, the lexical difficulty feature encoder receives the lexical difficulty features output from the lexical difficulty feature extractor (specifically, a vector sequence obtained by compiling the output (vector) of the lexical difficulty feature extractor for each word in the user utterance shown in FIG. 7 by turn, which is a set of vector sequences obtained from all user utterances), inputs the vector sequence into a linear function in chronological order, and after completing processing up to Self-Attention, adds positional information by Positional Encoding, and interactions between features are performed in the Transformer Encoder. The network weights of the lexical difficulty feature encoder are updated during learning of the complex feature model.

[0127] As described above, the grammatical error feature encoder receives the grammatical error features output from the grammatical error feature extractor (specifically, the word vector sequence (output of RoBERTa in GECToR) obtained by inputting the sentence spoken by the user into the grammatical error feature extractor is a set of vector sequences obtained from all sentences spoken by the user), inputs the vector sequence into a linear function in chronological order, and after completing processing up to Self-Attention, adds positional information using Positional Encoding, and interactions between features are performed in the Transformer Encoder. The network weights of the grammatical error feature encoder are updated during learning of the complex feature model.

[0128] As described above, the pronunciation feature encoder receives the pronunciation features output from the pronunciation feature extractor (specifically, the set of vector sequences obtained from all speech clauses, which are the output of Self-Attention (four vectors, i.e., vector sequences) illustrated in FIG. 8), inputs the vector sequences into a linear function in time series, and after completing the processing up to Self-Attention, adds positional information by Positional Encoding, and interactions between features are performed by the Transformer Encoder. The network weights of the pronunciation feature encoder are updated during learning of the complex feature model.

[0129] As described above, the discourse feature encoder receives discourse features output from the discourse feature extractor (specifically, the output of Self-Attention (26 vectors, i.e., vector sequence) when predicting the 26 discourse marker classes shown in FIG. 10 between sentences 1-1 and 1-2 as shown in FIG. 9, which is a set of vector sequences obtained from between all user utterances), inputs the vector sequences into a linear function in chronological order, and after completing the processing up to Self-Attention, adds positional information by Positional Encoding, and interaction between features is performed in the Transformer Encoder. The weights of the discourse feature encoder's network are updated during the learning of the complex feature model.

[0130] The coreference feature encoder will be described. As described above, the coreference feature encoder receives the coreference features output from the coreference feature extractor (specifically, as illustrated in FIG. 12, character information in units of topics is input to the coreference feature extractor, and a vector sequence in which the embedding vectors of phrases (word sequences) in a coreference relationship identified by the coreference analysis model are compiled for all phrases in the same coreference relationship is obtained as a set of vector sequences acquired for all phrases in a coreference relationship), inputs the vector sequence to a linear function in time series, and after completing the process up to Self-Attention, adds position information by Positional Encoding, and interaction between feature quantities is performed in the Transformer Encoder. The weights of the network of the coreference feature encoder are updated during the learning of the complex feature model.

[0131] The manually designed feature encoder will be described. As described above, the manually designed feature encoder receives manually designed features output from the manually designed feature extractor (specifically, for example, in the case of the number of fillers, the number of fillers included in the user utterance of a certain turn becomes the numerical value of a specific dimension of the vector, and similarly, a vector sequence in which the numerical values ​​of each dimension of the vector for all manually designed features are filled in for all turns of the same topic is compiled, and further a set of vector sequences obtained for all topics), inputs the vector sequence into a linear function in chronological order, and after completing the processing up to Self-Attention, positional information is added by Positional Encoding, and interaction between features is performed in the Transformer Encoder. The weights of the network of the manually designed feature encoder are updated during the learning of the complex feature model.

[0132] As shown in Fig. 16, the vector sequences output by the various encoders described above are input to a Transformer Encoder. The Transformer Encoder inputs vector sequences obtained by the mutual influence of linguistic features output by the various feature extractors to respective networks (combinations of Self-Attention and linear functions) specialized for each CEFR index. The network takes the vector sequences as input and uses a Softmax function to output the probability of each CEFR index level.

[0133] The ensemble model takes the predicted probability of each CEFR index level of the core model as input, and when predicting a specific CEFR index, outputs a reconsidered value taking into account the predicted probability of other CEFR indexes in the core model. FIG. 19 is a diagram showing an example of an ensemble model that outputs an overall proficiency in a complex feature model. The process flow of the ensemble model that outputs an overall proficiency in the complex feature model of the score measurement unit 133 will be described with reference to FIG. 19. As illustrated in FIG. 19, the score measurement unit 133 of this system combines the output of each index from the core model to recalculate the predicted probability of the overall proficiency. For example, the score measurement unit 133 of this system uses the above-mentioned GELU activation function for the recalculated predicted probability, and suppresses overlearning by known dropout. The score measurement unit 133 normalizes the mean and variance of the input data from the dropout using known layer normalization. Finally, the ensemble model applies a Softmax function to the output from the linear function to output the recalculated predicted probability of the overall proficiency. Similar ensemble models are constructed for vocabulary richness, grammatical correctness, etc.

[0134] Returning to FIG. 5, the simple feature model of the score measurement unit 133 will be described with reference to FIG. 20. FIG. 20 is a diagram showing an example of a processing flow of the simple feature model. As illustrated in FIG. 16, the complex feature model inputs a wide variety of language features and interacts with them to obtain high prediction performance, whereas the simple feature model inputs carefully selected features (for example, only text features or only specific features among manually designed features) and predicts the corresponding level. In addition, the predicted index may be narrowed down to one. For example, there is a simple feature model that inputs specific manually designed features that are effective for fluency and predicts only the CEFR level of fluency. Specifically, FIG. 20 illustrates an example of a simple feature model that predicts the level (A1 to C2) of vocabulary richness (Range) in the CEFR index. The part surrounded by the dotted line in FIG. 20 is a feature extractor of the simple feature model, that is, a text feature extractor, and the part surrounded by the solid line is the simple feature model. The feature extractor of the simple feature model takes as input the word sequence of the entire dialogue text between the dialogue agent and the user, and outputs the output of the final layer of the Longformer (a vector sequence of words) as the linguistic feature.

[0135] As illustrated in FIG. 20, the simple feature model inputs the linguistic features (word-unit vector sequence) output from the feature extractor of this simple feature model to a linear function. The output from this linear function is subjected to a known GELU, and the obtained vector sequence is given positional information by Positional Encoding, and then input to a Transformer Encoder. The vector sequence output from the Transformer Encoder is then converted into a single vector by known Self-Attention. The simple feature model inputs the output from Self-Attention to a linear function, and outputs a predicted probability of CEFR level (A1, A2, ..., C2) via a Softmax function.

[0136] As described above, the simple feature model may be trained based on the known Teacher-Student learning, with the complex feature model as the Teacher model and the simple feature model as the Student model, so that the output of the complex feature model (Teacher model) can be reproduced (imitated, approximated), or the simple feature model may be trained with annotation data by a human evaluator. The above-mentioned confirmation process by a human evaluator may also be applied to the simple feature model. The simple feature model may be configured to output one index for each model, or may be configured to output multiple indexes for each model, that is, a multi-task learning configuration. The model configuration and algorithm are not limited to this. The final output or intermediate layer output of the simple feature model (for example, the output of Self-Attention in FIG. 20) may be used as a feature extractor for the complex feature model, or the simple feature model itself (for example, the solid line part in FIG. 20) may be added to the encoder of the complex feature model.

[0137] The score converter converts the multidimensional probabilities (probabilities of A1, A2, ..., C2) obtained by the complex feature model and the simple feature model into real number (scalar) scores. For the scores converted by the score converter, see the example in Figure 14.

[0138] The basis extraction unit 134 receives the language features extracted by the score measurement unit 133 and the language feature extraction unit 131 as input and outputs the basis of the score. The basis of the score will be described later with reference to Figs. 21 to 27. The basis extraction unit 134 refers to the simple feature model and the complex feature model and calculates the contribution of the feature to the output score. For example, in a simple feature model that predicts the CEFR level of vocabulary richness (Range) using words as features, the basis is the contribution of each word included in the user's utterance to the prediction probability of each CEFR level of the simple feature model, and the explanation of the basis is performed by presenting this contribution. In addition, when the basis can be extracted from the complex feature model, the basis may be extracted directly from the complex feature model without going through the simple feature model. As an example, the Shapley value is used to calculate the contribution of the feature. The method using the Shapley value is a method of fairly allocating gains according to the contribution of players in conventional game theory. In this system, the Shapley value game is replaced with machine learning, the Shapley value players are replaced with the features of this system, and the Shapley value gains are replaced with the predicted values ​​of this system to find the contribution of the features to the predicted values ​​of the machine learning model. Note that the algorithm for calculating the contribution of the features is not limited to this.

[0139] In the method using Shapley value, the amount of calculation increases as the number of features increases. Therefore, as an example, this system uses the well-known SHAP (SHapley Additive exPlanations) as a method for approximating the Shapley value. In a multi-class classification model that uses word strings of the dialogue agent and the user's spoken text as features to identify CEFR levels (A1 to C2), SHAP is used to calculate the degree to which each word, which is a feature, contributes to the predicted probability of each CEFR level (SHAP value). The SHAP value calculated here shows whether each word contributed positively or negatively to the predicted probability of each CEFR level of A1 to C2. In other words, by applying SHAP to a simple feature model, it is possible to visually confirm to what extent each feature contributed to the determination of the CEFR level.

[0140] The visualization unit 135 visualizes the score obtained by the score measurement unit 133 and the basis of the score. FIG. 21 is a diagram showing an example of a radar chart. A case where a radar chart is used as one method of score visualization by the visualization unit 135 will be described with reference to FIG. 21. The visualization unit 135 displays the user's score obtained for each index by the score measurement unit 133 as a radar chart (part indicated by a thick line in FIG. 21) as exemplified in FIG. 21. From this radar chart, it can be seen that the user's overall proficiency level is C2. In addition, in the visualization, other indexes may be added, or one or more indexes may be omitted. In addition, the visualization method is not limited to a radar chart.

[0141] The visualization of the basis of the score by the visualization unit 135 will be described with reference to FIG. 22 and FIG. 23. FIG. 22 shows an example of visualization of the contribution (SHAP value) of each word to the probability of A2, 0.50, when the level of vocabulary richness in the CEFR index is A2 and the probability (A1, A2, B1, B2, C1, C2) is determined to be (0.10, 0.50, 0.30, 0.09, 0.01). In FIG. 22, the column "Speaker" indicates the speaker, and for example, "System" indicates that the speaker is a dialogue agent, and "User" indicates that the speaker is a user. In FIG. 22, for the sake of explanation, the sentences uttered by the dialogue agent are shown in italics. The column "Sentence" indicates the sentences uttered by the speaker in the same row. The column "ID" indicates the identifier of the sentence written in "Sentence" in the same row, and here consists of a topic number and a turn number.

[0142] In Fig. 22, thresholds are set for positive and negative SHAP values ​​(continuous values), and a positive SHAP value that is greater than or equal to the threshold is visualized as a "large plus", a value that is less than or equal to the threshold is visualized as a "small plus", and a negative SHAP value that is less than or equal to the threshold is visualized as a "large minus", and a value that is greater than or equal to the threshold is visualized as a "small minus". For the determination result A2, expressions (word strings) with a large minus contribution are displayed in bold, expressions with a small minus contribution are displayed in dotted shading, expressions with a small plus contribution are displayed in gray, and expressions with a large plus contribution are displayed underlined. The display mode illustrated in Fig. 22 is merely an example and is not limited thereto.

[0143] FIG. 23 is an example of visualization of the contribution (SHAP value) of each word to the probability of B1 of 0.30 when the level of vocabulary richness in the CEFR index is A2 and the probability (A1, A2, B1, B2, C1, C2) is determined to be (0.10, 0.50, 0.30, 0.09, 0.01). The display format of FIG. 23 is the same as that of FIG. 22, so the explanation is omitted. FIG. 22 illustrates the contribution to the determination result A2, while FIG. 23 displays the contribution to the determination result B1. By presenting (comparing) FIG. 22 and FIG. 23 to the user, when the determination result is A2, it is possible to confirm from FIG. 22 how much an expression (word string) contributed to the determination of A2 (contribution to the determination probability A2).

[0144] On the other hand, from Figure 23, although the overall judgment result was A2, it is possible to confirm the expression (word sequence) evaluated as B1 and its degree (contribution to judgment probability B1). Taking the last sentence (the sentence beginning with "In particular") of the utterance with "ID" 3-4 as an example, its contribution to judgment probability A2 is very low (large minus), while its contribution to judgment probability B1 is very large (large plus). From this display, it can be seen that the above expression is one of the reasons why it was judged to be more familiar (B1).

[0145] In addition, the above-mentioned language feature extraction unit 131 can be operated by itself without going through the language ability measurement unit 132. In that case, the visualization unit 135 may display the feature amount extracted by the language feature extraction unit 131. FIG. 24 is a diagram showing an example of visualization of the CEFR level of a phrase included in a user utterance. In FIG. 24, "ID" is composed of a topic number-turn number-sentence number. Other display forms in FIG. 24 are the same as those in FIG. 22 described above, and therefore will not be described. In FIG. 24, the result in which CEFR levels A1, A2, B1, B2, C1, and C2 are assigned to the phrase included in the user utterance is visualized. Specifically, in FIG. 24, as the CEFR level of an expression becomes higher (the proficiency level becomes higher), the expression is displayed in darker gray. From this display, it can be confirmed that "stubbornness" is an expression equivalent to C2 among the utterances with "ID" of 4-3-2, and it can be seen that this is one of the reasons why the proficiency level of vocabulary richness (Range) was determined to be high.

[0146] The visualization unit 135 may display the result of the grammatical error correction based on the grammatical error features extracted by the grammatical error feature extractor of the language feature extraction unit 131. FIG. 25 is a diagram showing an example of visualization of grammatical error features. In FIG. 25, after the grammatical error correction process, deleted expressions are displayed with strikethrough, and inserted expressions are displayed in bold. Other display forms in FIG. 25 are the same as those in FIG. 22 and FIG. 24 described above, and therefore will not be described. In the sentence with ID 2-3-1, since "an" is displayed in bold, it can be seen that "an" was inserted by the grammatical error correction process after the user's utterance. This indicates that a correction was made to the user's utterance for a preposition error. For example, the correct answer should be the preposition (an), but the user spoke without a preposition or using an incorrect preposition, so the corrected sentence is displayed from the perspective of the preposition.

[0147] Additionally, in the sentence with ID 2-5-1, "room" has been deleted and "rooms" is displayed in bold, which indicates that "rooms" was inserted in place of "room" after the user spoke as a result of grammatical error correction processing. This indicates that a correction was made to the grammatical error between the singular and plural forms of the user's speech. For example, the correct answer would be "rooms," but since the user spoke "room" in the singular form, the corrected sentence in which the singular form has been changed to the plural form is displayed. This display allows the user to visually confirm their own grammatical mistakes.

[0148] The visualization unit 135 may display the discourse features extracted by the discourse feature extractor of the language feature extraction unit 131. Fig. 26 is a diagram showing an example of visualization of discourse features. The display format of Fig. 26 is similar to that of Fig. 25 described above, and therefore a description thereof will be omitted. On the right side of Fig. 26, an estimated value (see Figs. 9 and 10) is displayed for each discourse marker class. This estimated value is a predicted value by a model for each discourse marker class, that is, the probability that the target sentence and the previous sentence can be connected by the discourse marker class.

[0149] For the sentence with "ID" 4-3-2, looking at the estimated class of discourse markers displayed on the right, the highest value is for "CONFLICT," a class to which discourse markers expressing adversative conjunctions such as "However," belong. This indicates that sentence 4-3-2 is in an adversative relationship with sentence 4-3-1. Also, for the sentence with "ID" 4-3-3, looking at the estimated class of discourse markers displayed on the right, the highest value is for "EFFECT," a class to which discourse markers expressing results or conclusions such as "Therefore," belong. This indicates that sentence 4-3-3 is a conclusion following sentence 4-3-2.

[0150] The results shown in FIG. 26 show that, as a result of prediction by the model of each discourse marker class shown in FIG. 10, the predicted value of the CONFLICT class for the discourse relation between sentences 4-3-1 and 4-3-2 exceeded the threshold probability of 0.5. Also, as a discourse relation between sentences 4-3-2 and 4-3-3, the predicted value of the EFFECT class exceeded the threshold probability of 0.5. This display allows the user to check how well he or she structures a story and speaks logically. When the predicted probabilities of multiple discourse marker classes exceed the threshold, multiple discourse marker classes may be visualized, or one discourse marker class with the highest predicted probability may be visualized.

[0151] The visualization unit 135 may display the coreference features extracted by the coreference feature extractor of the language feature extraction unit 131. The above-mentioned Fig. 13 is also a diagram showing an example of visualization of the coreference features. The display exemplified in Fig. 13 allows the user to check whether he or she is properly managing the topic and speaking coherently while referring to his or her own past statements and those of his or her interlocutor.

[0152] The visualization unit 135 may display the quality of pronunciation based on the pronunciation features extracted by the pronunciation feature extractor of the language feature extraction unit 131. In Fig. 27, "ID" is composed of topic number-turn number-speech clause number. Other display forms in Fig. 27 are the same as those in Fig. 22 described above, and therefore will not be described. On the right side of Fig. 27, values ​​are displayed that are normalized to 0 to 10 (scalar) from the predicted probability (probability of each of 11 levels from 0 to 10) by the model (see Fig. 8) using a score converter (Formulas 5 to 7).

[0153] The interpretation of the values ​​displayed on the right side of FIG. 27 is based on speechcean762, a known learning data set. Although FIG. 27 displays the speeches in chronological order, the display form is not limited to this, and the speeches may be displayed in order of the user's best pronunciation. This display allows the user to look back on the pronunciation in his / her own speech that was judged to be good or not so good in each index, such as pronunciation accuracy, pronunciation fluency, intonation and rhythm, and overall pronunciation quality. For example, the user can play and check his / her own speech using the button indicated by the play symbol shown in the figure. This display is an example, and the data may be arranged based on other indexes.

[0154] The translation unit 136 converts the language of input data when the language feature extraction unit 131 handles input data in multiple languages. By providing the translation unit 136, it becomes possible to execute feature extraction using an existing feature extractor even if the language feature extraction unit 131 does not have a different feature extractor for each language. Note that this translation unit 136 may be omitted, and for example, a translation system or translation application external to this system may be used.

[0155] <Summary> Next, the operation and effects of this embodiment will be described.

[0156] According to the information processing method of the present embodiment, when input data acquired from a user's speech is given, the input data is analyzed by a predetermined process to diagnose the user's language ability. Specifically, when input data acquired from a user's speech is given, features are extracted from the input data, and a score is output that measures the user's speech using an arbitrary index from the extracted features. Therefore, the language ability can be diagnosed based on various language features contained in the user's multimodal input data.

[0157] According to the information processing method of this embodiment, the method further includes a reason extraction unit 134 and a visualization unit 135, and the score calculated by the score measurement unit 133 and the reason for the score calculated by the reason extraction unit 134 are visualized by the visualization unit 135. Therefore, it is possible to not only show the score to the relevant person, but also to explain the reason for the score.

[0158] According to the information processing method of this embodiment, the feature amount extracted by the language feature extraction unit 131 may be displayed by the visualization unit 135. This allows visual feedback of the user's language features, allowing the user to reflect on his or her own utterance and the dialogue with the dialogue agent.

[0159] According to the information processing method of this embodiment, the multi-dimensional probabilities obtained by the complex feature model and the simple feature model are converted into real number (scalar) scores by the score converter. This allows the user to understand his / her own growth. For example, for a user whose CEFR level has progressed from A1 to A2, the language ability measurement results can be presented as A1(1.11)→A1(1.37)→A1(1.66)→A1(1.92)→A2(2.1) in order to understand his / her own growth in more detail, rather than A1→A1→A1→A1→A2.

[0160] According to the information processing method of the present embodiment, the algorithms and methods used in the present system are described as examples, but the implementation form is not limited to these. For example, other algorithms may be used instead of these algorithms and methods.

[0161] <Additional Notes> The present embodiment includes the following disclosure.

[0162] (Appendix 1) The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is indicated by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. The present invention is not limited to the above-mentioned embodiments, and various modifications are possible within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0163] (Appendix 2) The information processing device is characterized in that it includes at least one feature extractor that extracts the feature in the feature extraction process. 2. An information processing method as described in Appendix 1.

[0164] (Appendix 3) The input data includes one or more of text information, audio information, and video information; The feature amount includes a vector obtained by converting one or more of the character information, the audio information, and the video information into numerical information. 2. An information processing method as described in Appendix 1.

[0165] (Appendix 4) The feature extraction process includes a process of outputting all or any of text features, multimodal features, lexical difficulty features, grammatical error features, pronunciation features, discourse features, coreference features, common sense features, and manually designed features as features. 2. An information processing method as described in Appendix 1.

[0166] (Appendix 5) The score measurement process includes a process of measuring the score with respect to at least one or more indicators related to the language ability of the user. 2. An information processing method as described in Appendix 1.

[0167] (Appendix 6) The basis extraction process includes a process of extracting a contribution degree of each feature amount that explains the basis of the score. 2. An information processing method as described in Appendix 1.

[0168] (Appendix 7) a visualization unit that visualizes the basis of the score extracted in the basis extraction process, and the visualization unit visually presents the basis to the user. 2. An information processing method as described in Appendix 1.

[0169] (Appendix 8) a language feature extraction unit that acquires input data including a speech by a user and extracts features from the acquired input data; a language ability measurement unit that measures the language ability of the user based on an arbitrary index from the extracted feature amount, The language ability measurement unit includes a score measurement unit that measures a score, and a reason extraction unit that extracts a reason for the score. Information processing device.

[0170] (Appendix 9) On the computer, An acquisition process for acquiring input data including a speech by a user; A feature extraction process for extracting features from the acquired input data; A language ability measurement process for measuring the language ability of the user based on an arbitrary index from the extracted feature amount, The language ability measurement process includes a score measurement process for calculating a score, and a reason extraction process for extracting a reason for the score. A program for executing an information processing method.

[0171] (Appendix 10) a language feature extraction unit that acquires input data including a speech by a user and extracts features from the acquired input data; a language ability measurement unit that measures the language ability of the user based on an arbitrary index from the extracted feature amount, The language ability measurement unit includes a score measurement unit that measures a score, and a reason extraction unit that extracts a reason for the score. An information processing device; a user terminal for inputting a speech by the user, measuring the language ability of the user from the utterance by the user inputted from the user terminal; Language proficiency diagnostic system. [Explanation of symbols]

[0172] 1: Language Ability Diagnostic Device 2: User terminal 11: Communications Department 12: Storage section 13: Control unit 101: Processor 102: Memory 103: Storage 104: Communication I / F 105: Input / Output I / F 106: Drive device 107: Input device 108: Output device 109: Recording media 131: Language feature extraction unit 132: Language Ability Measurement Department 133: Score measurement section 134: Basis extraction unit 135: Visualization part 136: Translation Department 201: Processor 202: Memory 203: Storage 204: Communication I / F 205: Input / Output I / F 206: Drive device 207: Input device 208: Output device 209: Recording media 1000: Information Processing Systems

Claims

1. An information processing method executed by an information processing device in a language ability diagnostic system, comprising: An acquisition process for acquiring input data including a speech by a user; A feature extraction process for extracting features from the acquired input data; A language ability measurement process for measuring the language ability of the user based on an arbitrary index from the extracted feature amount, the language ability measurement process includes a score measurement process for calculating a score from the feature amount, and a reason extraction process for extracting a reason for the score from the feature amount, Information processing methods.

2. The information processing device is characterized in that it includes at least one feature extractor that extracts the feature in the feature extraction process. The information processing method according to claim 1 .

3. The input data includes one or more of text information, audio information, and video information; The feature amount includes a vector obtained by converting one or more of the character information, the audio information, and the video information into numerical information. The information processing method according to claim 1 .

4. The feature extraction process includes a process of outputting all or any of text features, multimodal features, lexical difficulty features, grammatical error features, pronunciation features, discourse features, coreference features, common sense features, and manually designed features as features. The information processing method according to claim 1 .

5. The score measurement process includes a process of measuring the score with respect to at least one or more indicators related to the language ability of the user. The information processing method according to claim 1 .

6. The basis extraction process includes a process of extracting a contribution degree of each feature amount that explains the basis of the score. The information processing method according to claim 1 .

7. a visualization unit that visualizes the basis of the score extracted in the basis extraction process, and the visualization unit visually presents the basis to the user. The information processing method according to claim 1 .

8. a language feature extraction unit that acquires input data including a speech by a user and extracts features from the acquired input data; a language ability measurement unit that measures the language ability of the user based on an arbitrary index from the extracted feature amount, the language ability measurement unit includes a score measurement unit that measures a score from the feature amount, and a reason extraction unit that extracts a reason for the score from the feature amount. Information processing device.

9. On the computer, An acquisition process for acquiring input data including a speech by a user; A feature extraction process for extracting features from the acquired input data; A language ability measurement process for measuring the language ability of the user based on an arbitrary index from the extracted feature amount, the language ability measurement process includes a score measurement process for calculating a score from the feature amount, and a reason extraction process for extracting a reason for the score from the feature amount, A program for executing an information processing method.

10. a language feature extraction unit that acquires input data including a speech by a user and extracts features from the acquired input data; a language ability measurement unit that measures the language ability of the user based on an arbitrary index from the extracted feature amount, The language ability measurement unit includes a score measurement unit that measures a score from the feature amount, and a reason extraction unit that extracts a reason for the score from the feature amount. An information processing device; a user terminal for inputting a speech by the user, measuring the language ability of the user from the utterance by the user inputted from the user terminal; Language proficiency diagnostic system.

Images