Foreign language acquisition support system and foreign language acquisition support method

Abstract

To improve non-language communication skill (expression, gesture or the like) other than language skill when learning a foreign language.SOLUTION: The camera (145) captures the non-verbal communication skill of the user during the conversation and transmits it to the central processor (121). The CPU (121) evaluates the non-language communication skill of the user by comparing the non-language communication skill of the user with an exemplary non-language communication skill pre-stored in the second database (132E).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a foreign language learning support system, a foreign language learning support method, and a program for causing a computer to execute the method, which support a user's acquisition of a foreign language.

Background Art

[0002] One of the effective means for foreign language learning is "Shadowing". "Shadowing" is primarily a simultaneous interpretation training method in which a person listens to a foreign language (for example, English) and imitates it to pronounce. Unlike "Repeat", which is performed after finishing listening to the foreign language, shadowing requires pronouncing immediately after the heard foreign language. It is named so because it chases right after the foreign language like a shadow. An example of a shadowing device for performing shadowing is described in Japanese Patent No. 5756555. This shadowing device records the voice of a learner during reading aloud or shadowing, and performs an objective evaluation on the recorded voice. It is stated that this can support independent learning and reduce the man-hours of instructors.

Prior Art Documents

Patent Documents

[0003] Japanese Patent No. 5756555 Publication

Summary of Invention

Problems to be Solved by the Invention

[0004] The ultimate goal of foreign language acquisition is not merely to acquire the ability to listen to and speak a foreign language, but to effectively communicate with others through language. In general, it is said that non-verbal elements (non-verbal communication skills) account for a larger proportion than language itself in communication. Nonverbal elements are means of expression other than language, such as facial expressions, gestures, and body language. These nonverbal elements not only complement and emphasize verbal messages, but also help us understand the other person's emotions and intentions. Furthermore, nonverbal elements are also important in avoiding misunderstandings with the other party. This is because the same words can be interpreted in completely different ways depending on facial expressions, gestures, and body language.

[0005] Thus, in order to communicate effectively with others, acquiring nonverbal communication skills is just as important as acquiring verbal skills. While there is no universal language, nonverbal communication skills are universal (for example, this is why Chaplin's silent films were accepted all over the world). However, the conventional shadowing devices mentioned above are solely aimed at improving language skills and are completely indifferent to nonverbal communication skills. This invention has been made in view of the problems with such conventional shadowing devices, and aims to provide a foreign language acquisition support system, a foreign language acquisition support method, and a program for the method that enable the improvement of nonverbal communication skills when acquiring a foreign language, regardless of the foreign language learning method. [Means for solving the problem]

[0006] To achieve this objective, the present invention, in a first aspect, is a foreign language acquisition support system (500) that supports a user's acquisition of a foreign language, comprising: an imaging means (145) for imaging the user as they pronounce according to the audio of a video; a first memory means (132D) for storing exemplary nonverbal communication skills of a speaker during a conversation; a second memory means (132E) for storing a learned evaluation model for evaluating the speaker's nonverbal communication skills during a conversation; an evaluation means (131D) for comparing exemplary nonverbal communication skills stored in the first memory means (132D) with the user's nonverbal communication skills captured by the imaging means (145) using the learned evaluation model (210B) stored in the second memory means (132E), and display means (131E, 142) for displaying the evaluation created by the evaluation means (131D), wherein the evaluation means (131D) is used to analyze facial expressions, gaze, gestures, body language, proxemics, Physical appearance, Visual focus, auditory information and cultural background The system provides a foreign language acquisition support system (500) which evaluates the user's nonverbal communication skills with respect to at least one of the evaluation items, and the evaluation means (131D) uses the user's conversation content as auxiliary language data when evaluating the user's nonverbal communication skills, and determines whether the user's nonverbal communication skills are consistent with the user's conversation content. 。

[0007] Preferably, the evaluation means (131D) extracts feature quantities that quantitatively represent the characteristics of each of the evaluation items from the data of the user's image captured by the imaging means (145), and compares these feature quantities with exemplary nonverbal communication skills stored in the first storage means (132D). 。 Preferably, the foreign language acquisition support system (500) further comprises a cultural background information database that stores cultural background information for each country or region, and an adjustment means that reads the user's cultural background information from the cultural background information database and makes adjustments to the evaluation of the user's nonverbal communication skills by the evaluation means (131D) taking the user's cultural background information into consideration.

[0008] Preferably, the trained evaluation model (210B) is generated by machine learning to evaluate the nonverbal communication skills of a speaker using exemplary nonverbal communication skills stored in the first memory means (132D) as a criterion, the input (220) to the trained evaluation model (210B) is image data of the user's nonverbal communication skills during the conversation captured by the imaging means (145), and the output (230) from the trained evaluation model (210B) is an evaluation of the user's nonverbal communication skills during the conversation using the exemplary nonverbal communication skills as a criterion, after performing a predetermined calculation process. The foreign language acquisition support system (500) preferably further comprises a curriculum creation means for creating a curriculum tailored to the user in order to compensate for the user's nonverbal communication skills, which were poorly evaluated by the evaluation means (131D). The foreign language acquisition support system (500) preferably further comprises a learning material creation means for creating learning materials in accordance with the curriculum created by the curriculum creation means.

[0009] For example, the learning material is preferably a video, and the video is preferably an interactive video in which the user (320) and the characters (310) engage in a dialogue. The foreign language acquisition support system (500) further comprises a subtitle formation means for displaying subtitles (330) on the video screen, and preferably the subtitle formation means displays at least one of the subtitles (330) on the screen, which is either a subtitle to that effect or an advice regarding the nonverbal communication skill, if the nonverbal communication skill of the user (320) that was poorly evaluated by the evaluation means (131D) appears during the dialogue. The foreign language learning support system (500) further comprises a subtitle-to-speech means for converting the subtitles (330) into audio, and it is preferable that the subtitle-to-speech means plays audio of the subtitles (330) from the screen in place of the subtitles (330) or together with the subtitles (330).

[0010] In a second aspect, the present invention provides a portable wireless communication device (100) incorporating the above-described foreign language acquisition support system (500). In a third aspect, the present invention is a program for causing a computer to execute a foreign language acquisition support method that assists a user in acquiring a foreign language, comprising: a first process (S140) of capturing images of the user pronouncing according to the audio of a video; a second process (S210) of using a trained evaluation model (210B) to evaluate the nonverbal communication skills of a speaker during a conversation, comparing exemplary nonverbal communication skills stored in a storage means (132D) that pre-stores exemplary facial expressions, gestures, and other exemplary nonverbal communication skills of a speaker during a conversation with the nonverbal communication skills of the user captured in the first process (S140), and evaluating the user's nonverbal communication skills; and a third process (S220) of displaying the evaluation created in the second process (S210), wherein the second process (S210) includes facial expressions, gaze, gestures, body language, proxemics, Physical appearance, Visual focus, auditory information and cultural backgroundThe present invention provides a program in which the user's nonverbal communication skills are evaluated with respect to at least one of the evaluation items, and in the second process (S210), when evaluating the user's nonverbal communication skills, the user's conversation content is used as auxiliary linguistic data, and it is determined whether or not the user's nonverbal communication skills are consistent with the user's conversation content.

[0011] The aforementioned second process In (S210), it is preferable that feature quantities that quantitatively represent the characteristics of each of the evaluation items are extracted from the user's image data, and these feature quantities are compared with exemplary nonverbal communication skills stored in the storage means (132D). 。 In the second process (S210) described above, it is preferable that the user's cultural background information is read from a cultural background information database that stores cultural background information for each country or region, and that the evaluation of the user's nonverbal communication skills is adjusted taking the user's cultural background information into account.

[0012] Preferably, the pre-trained evaluation model (210B) is generated by machine learning to evaluate the speaker's nonverbal communication skills using exemplary nonverbal communication skills stored in the memory means (132D) as a criterion, the input (220) to the pre-trained evaluation model (210B) is image data capturing the user's nonverbal communication skills during the conversation, and the output (230) from the pre-trained evaluation model (210B) is an evaluation of the user's nonverbal communication skills during the conversation, using the exemplary nonverbal communication skills as a criterion, after performing a predetermined calculation process. The foreign language acquisition support method is the second method described above. processa fourth step of creating a curriculum specialized for the user to compensate for the user's non-verbal communication skills that were evaluated low in (S210) process is further Execute preferably performed.

[0013] The program preferably further includes a fifth step of creating learning materials in accordance with the curriculum created in the fourth process step process to be further Execute performed. For example, the learning material is preferably a video, and the video is preferably an interactive video in a format where the user (320) and a character (310) have a dialogue. The program preferably further includes a fifth step of, when the user's non-verbal communication skills that were evaluated low in the second process (S210) appear during the dialogue, displaying at least a subtitle (330) indicating that fact, among the subtitle (330) to that effect and advice for the non-verbal communication skills, on the screen in a scrolling manner process to be further Execute performed. The program preferably further includes a sixth step of converting the subtitle (330) into speech, and a seventh step of outputting, from the screen, the speech obtained by vocalizing the subtitle (330) in place of or together with the subtitle (330) process , and process are further Execute preferably performed.

[0014] This invention provides, as a four aspect, a portable wireless communication device (100) incorporating the above program. Reference signs in parentheses are added to indicate correspondence with the embodiments described later, and are not intended to limit the scope of the claims thereto. Effects of the Invention

[0015] In order to communicate effectively with others, nonverbal communication skills (such as facial expressions, gestures, and body language) are just as important as verbal skills. However, conventional systems (foreign language acquisition support devices, foreign language learning materials, foreign language schools, etc.) have been completely indifferent to nonverbal communication skills. The foreign language acquisition support system according to the present invention enables users to improve their nonverbal communication skills. Specifically, it allows users to self-study exemplary nonverbal communication skills during conversations, thereby improving their communication abilities. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram showing an example of the structure of a portable wireless communication device incorporating a foreign language learning support system according to the first embodiment of the present invention. [Figure 2] This is a conceptual diagram showing the structure of the third program stored in external memory. [Figure 3] This is a conceptual diagram illustrating the functionality of a third program stored in external memory. [Figure 4] Figure 1 is a flowchart showing the operation of the foreign language learning support system. [Figure 5] This is a conceptual diagram of a foreign language acquisition support system according to the fifth embodiment of the present invention. [Modes for carrying out the invention]

[0017] (First embodiment) Figure 1 is a block diagram showing an example of the structure of a portable wireless communication device 100 incorporating a foreign language acquisition support system 500 according to the first embodiment of the present invention. The portable wireless communication device 100 comprises a communication unit 110, a control unit 120, an external memory (hard disk) 130, an input / output unit 140, and an antenna 150. Of these, the control unit 120, the external memory 130, and the input / output unit 140 constitute the foreign language learning support system 500. The portable wireless communication device 100 is configured as, for example, a portable telephone device such as a smartphone. The communication unit 110 is connected to the antenna 150 and transmits and receives data wirelessly with other wireless communication devices via the antenna 150. The communication unit 110 includes a wireless receiving unit 111, a wireless transmitting unit 112, and a changeover switch 113.

[0018] The wireless receiver 111 demodulates data received from other wireless communication devices and sends it to the control unit 120. The wireless transmitter 112 modulates the data output from the control unit 120 and transmits it to other mobile phones or other wireless communication devices via the antenna 150. The changeover switch 113 receives a signal from the control unit 120 and switches between transmitting and receiving according to that signal. The control unit 120 consists of a central processing unit (CPU) 121, a first memory 122 consisting of ROM, a second memory 123 consisting of RAM, an input interface 124 for transferring various commands and data input to the control unit 120 to the central processing unit 121, an output interface 125 for outputting processing results executed by the central processing unit 121 to the outside, and a bus 126 connecting the central processing unit 121 to the first memory 122, the second memory 123, the input interface 124, and the output interface 125, respectively.

[0019] The first memory 122 stores various control programs executed by the central processing unit 121 and other data that cannot be rewritten. The second memory 123 stores various data and parameters and also provides an operating area for the central processing unit 121. That is, data or programs temporarily required by the central processing unit 121 to execute various control programs are read from the external memory 130 and temporarily stored in the second memory 123. The central processing unit 121 works in cooperation with the OS (Operating System) to control the overall operation of the portable wireless communication device 100. Specifically, it reads the programs necessary for the operation of the portable wireless communication device 100 from the external memory 130 and executes those programs. In other words, the central processing unit 121 operates according to the programs stored in the external memory 130. As will be described later, the central processing unit 121 outputs a predetermined result for a predetermined input via a trained evaluation model.

[0020] The input / output unit 140 consists of an operation unit 141, a display 142, a speaker 143, a microphone (sound collector) 144, and a camera 145 as an imaging device. The operation unit 141 consists of, for example, a numeric keypad, and various data are input to the control unit 120 via the operation unit 141. The display 142 is, for example, a liquid crystal display and displays the results of calculations performed by the control unit 120 (such as evaluation results described later) and other data on the screen. The voice (synthesized voice) generated by the control unit 120 is output via the speaker 143. The audio data collected by the microphone 144 and the image data captured by the camera 145 are sent to the control unit 120.

[0021] The external memory (hard disk) 130 consists of an application storage section 131 and a data storage section 132. The data storage unit 132 consists of a video storage unit 132A which stores various videos collected to date, a video audio storage unit 132B which records the audio of a video selected by the user as audio data, a user audio recording unit 132C which records the audio of the user following the audio of the video as audio data, a first data storage unit 132D which stores data on exemplary language skills (such as accurate pronunciation, accent, and exemplary fluency) as well as exemplary nonverbal communication skills of the speaker during a conversation, a second data storage unit 132E which stores a trained first evaluation model that evaluates the user's language skills and a trained second evaluation model that evaluates the speaker's nonverbal communication skills during a conversation, a user image data storage unit 132F which records images of the user captured by the camera 145 (specifically, the user's nonverbal communication skills) as image data, and a sub-data storage unit 132G which stores various data other than these video data.

[0022] In this specification, the "nonverbal communication skills" of a speaker include at least the following items: (1) Facial expressions (focusing on changes in emotions and subtle movements of facial expressions) (2) Gaze (including the direction and duration of gaze) (3) Gestures (including hand and arm movements) (4) Body language (posture and body orientation) (5) Proxemics (focusing on interpersonal distance, interpersonal angle, etc.) (6) Physical appearance (including clothing, hairstyle, accessories, etc.) The application unit 131 stores the OS (Operating System) 131S, which controls the overall operation of the portable wireless communication device 100, as well as the first program 131A, the second program 131B, the third program 131C, the fourth program 131D, and the fifth program 131E.

[0023] The first program 131A constitutes a video audio recording means that causes the audio of a video selected by the user from among the videos stored in the video storage unit 132A to be recorded in the video audio storage unit 132B. The second program 131B comprises a user voice recording means that records the voice spoken by the user in accordance with the video selected by the user in the user voice storage unit 132C, and a user image data recording means that records the user's nonverbal communication skills captured by the camera 145 in the user image data storage unit 132F. The third program 131C uses the trained first evaluation model stored in the second data storage unit 132E to evaluate the user's language skills and generate evaluation results by comparing the exemplary language skills stored in the first data storage unit 132D with the user's voice data stored in the user voice storage unit 132C.

[0024] The fourth program 131D uses a trained second evaluation model stored in the second data storage unit 132E to create an evaluation result for evaluating the user's nonverbal communication skills during a conversation. This evaluation is done by comparing the exemplary nonverbal communication skills during a conversation stored in the first data storage unit 132D with the user's nonverbal communication skills during a conversation as captured by the camera 145 and sent to the control unit 120 (image data capturing the user's nonverbal communication skills during a conversation). In other words, the fourth program, 131D, assesses the user's nonverbal communication skills during shadowing (or reading aloud, conversation, etc.).

[0025] The fifth program 131E has the function of displaying the evaluation results created by the third program 131C and the fourth program 131D on the display 142. The fifth program 131E and the display 142 constitute the display means. Figure 2 is a conceptual diagram showing the configuration of the third program 131C. As shown in Figure 2, the third program 131C consists of a training data input program 1311, an evaluation model generation program 1312, and an evaluation result output program 1313. The training data input program 1311 is a program for inputting training data to the control unit 120 in order to generate an evaluation model using machine learning. The evaluation model generation program 1312 uses the training data input by the training data input program 1311 to generate each evaluation model using machine learning, and outputs the generated trained evaluation models to the central processing unit 121. The evaluation result output program 1313 outputs the evaluation results using the trained evaluation model generated by the evaluation model generation program 1312.

[0026] The fourth program 131D has the same configuration as the third program 131C. Figure 3 is a conceptual diagram illustrating the function of the third program 131C. As shown in Figure 3, the training data 200 is sent to the evaluation model generation program 1312 by the training data input program 1311, and the evaluation model generation program 1312 generates an evaluation model 210 based on the training data 200. This evaluation model 210 then continuously performs machine learning using the subsequent input as training data 200, becoming a fully trained evaluation model 210. The first evaluation model 210A of the evaluation models 210 is generated by machine learning to evaluate whether the user's language skills are appropriate, using exemplary language skills (such as accurate pronunciation and accent, exemplary fluency, and appropriate vocabulary) stored in the first data storage unit 132D as judgment criteria. The trained first evaluation model 210A receives the user's voice during the conversation as audio data (input 220).

[0027] The trained first evaluation model 210A performs predetermined calculations and outputs 230 an evaluation of whether the user's language skills are appropriate, using exemplary language skills as the criterion. The evaluation result (output 230) generated by the trained first evaluation model 210A is sent to the control unit 120 by the evaluation result output program 1313, and then displayed on the display 142 by the fifth program 131E. The trained second evaluation model 210B is generated by machine learning to evaluate whether the user's nonverbal communication skills during a conversation are appropriate, using exemplary nonverbal communication skills during a conversation stored in the first data storage unit 132D as judgment criteria. The trained second evaluation model 210B receives image data captured by camera 145, which shows the user's nonverbal communication skills during conversation, as input 220.

[0028] The trained second evaluation model 210B performs predetermined calculations and outputs 230 an evaluation of whether the user's nonverbal communication skills during conversation are appropriate, using generally considered exemplary nonverbal communication skills in conversation as the judgment criterion. The evaluation result (output 230) generated by the trained second evaluation model 210B is sent to the control unit 120 by the evaluation result output program 1313, and then displayed on the display 142 by the fifth program 131E. Figure 4 is a flowchart showing the operation of the portable wireless communication device 100. The operation of the portable wireless communication device 100 will be explained below with reference to Figure 4. First, the user selects a video from among the many videos stored in the video storage unit 132A that they want to use for learning (for example, shadowing) (step S100).

[0029] The video storage unit 132A stores various videos. For example, users can select a video that suits their needs from videos categorized by genre (business scenes, shopping scenes, airplane scenes, etc.), language (major languages ​​such as English and French, as well as less commonly used minor languages), region (for example, even the same English can have different pronunciations in the UK, the US, and Australia), and length (from over an hour to just a few minutes). Next, the user specifies their assessment level of their language skills and nonverbal communication skills in conversation before starting the learning process (step S110). For example, the evaluation levels are divided into several stages, from "lenient evaluation" to "strict evaluation," and users can specify an evaluation level that matches their learning progress.

[0030] Subsequently, the user activates the recording device (not shown) and camera 145 to start recording and video recording (step S120). Next, the user begins learning. Specifically, the audio from the video is emitted from speaker 143, and the user imitates the audio from the video, either immediately after the audio or at predetermined intervals. The voice emitted by the user is collected by microphone 144 and recorded in user voice storage unit 132C of data storage unit 132 via second program 131B (step S130). Along with recording the user's voice, the user's nonverbal communication skills during learning are captured by camera 145 (step S140) and recorded in the user image data storage unit 132F of the data storage unit 132 via the second program 131B. The audio data, which is a recording of the user's voice, and the image data, which is a recording of the user's nonverbal communication skills, are sent to the control unit 120. Simultaneously with the recording of the user's voice and the recording of the user's nonverbal communication skills, the audio of the video is also recorded in the video audio storage unit 132B via the first program 131A (step S150).

[0031] The user's voice recorded in the user voice storage unit 132C is converted into text by a program (not shown) having a voice recognition function (step S160). Once learning is complete (step S170), the central processing unit 121 starts the third program 131C and uses the learned first evaluation model stored in the second data storage unit 132E to compare the exemplary language skills stored in the first data storage unit 132D with the actual user voice recorded in the user voice storage unit 132C, and evaluates the user's pronunciation (language skills) (step S180).

[0032] Language skills will be assessed based on at least the following points: (A) Evaluation of pronunciation Specifically, the clarity and accuracy of each sound's pronunciation, syllable pronunciation, distinction between vowels and consonants, and accent placement are evaluated. (B) Evaluation of intonation Specifically, the evaluation will cover aspects such as intonation, pitch changes, rhythm, appropriateness of emphasis, and expression of emotion in sentences and phrases. (C) Fluency assessment Specifically, the fluency, naturalness, consistency, pace, and smoothness of the flow of words are evaluated. (D) Correction of errors The system detects slips of the tongue, pronunciation errors, grammatical errors, and poor vocabulary choices. After this, the central processing unit 121 displays the above evaluation results on the display 142 (step S190).

[0033] Table 1 shows an example of the evaluation results displayed on display 142. The evaluation results consist of a score (points) and a brief comment (message) for each evaluation item. (Table 1) TIFF0007939718000001.tif67138

[0034] For example, pronunciation errors can be displayed on display 142 not only after the learning session is complete, but also during the session. In this case, the user's attention can be drawn to the incorrect pronunciation by highlighting it with a colored highlight or icon. Next, if necessary (as requested by the user), the correct pronunciation is output for any incorrect pronunciations made by the user (step S200). Specifically, the central processing unit 121 activates a speech generation program (not shown) to generate the correct pronunciation of the word that the user mispronounced using synthesized speech, and outputs it from the speaker 143. Furthermore, in addition to the evaluation of the user's language skills as described above, the foreign language acquisition support system 500 according to this embodiment also evaluates the user's nonverbal communication skills during conversation, as follows.

[0035] When the user begins learning, camera 145 captures images of the user during the learning process (specifically, the user's nonverbal communication skills). The captured image data is recorded in the user image data storage unit 132F via the second program 131B. Once the learning is complete (step S170), the central processing unit 121 starts the fourth program 131D. This allows the trained second evaluation model stored in the second data storage unit 132E to compare the exemplary nonverbal communication skills during a conversation stored in the first data storage unit 132D with the user's nonverbal communication skills during the conversation in the image data captured by the camera 145 and sent to the control unit 120 (image data capturing the user's nonverbal communication skills during the conversation) (step S210). The central processing unit 121 generates an evaluation result that assesses the user's nonverbal communication skills during conversation as a result of this comparison.

[0036] Table 2 shows an example of an evaluation (score and message) for facial expressions and gestures, which are part of nonverbal communication skills. (Table 2) TIFF0007939718000002.tif17119 After the central processing unit 121 creates the evaluation results (Table 2), it displays the evaluation results, along with the language skills evaluation results (Table 1), on the display 142 (step S220). The assessment of the nonverbal communication skills mentioned above will be described in detail below.

[0037] The evaluation of the aforementioned nonverbal communication skills (1) to (6) is performed by extracting feature quantities for each of the user's nonverbal communication skills from the user's image data captured by the camera 145, and comparing these feature quantities with exemplary nonverbal communication skills during conversation stored in the first data storage unit 132D via a trained second evaluation model stored in the second data storage unit 132E. Feature quantities are numerical representations of the characteristics of each nonverbal communication skill, used to quantitatively evaluate nonverbal communication skills. The fourth program 131D (evaluation means) compares these feature quantities with exemplary nonverbal communication skills stored in the first data storage unit 132D. The features of each nonverbal communication skill are also used as training data 200 for the trained evaluation model 210B (see Figure 3) (this will be discussed later).

[0038] (1)Facial expression Facial expressions are a primary nonverbal communication skill that conveys emotions, intentions, attitudes, and other information. In the foreign language acquisition support system 500 according to this embodiment, feature quantities for capturing various facial expression elements, particularly smiles which are especially important among facial expressions, are extracted from image data captured by the camera 145. (1A) Smile (1A-1) The angle of the corners of the mouth, the distance of the rise, and the bilateral symmetry are extracted as features to represent the degree of upward movement of the corners of the mouth. (1A-2) As for mouth opening, the degree of opening, opening area, and opening speed are extracted as features. (1A-3) As for the appearance of the teeth, the degree of exposure of the upper teeth only, the upper and lower teeth, and the gums will be extracted as feature quantities. (1A-4) As features of cheek prominence, the contraction strength of the buccinator muscle, the height of the cheek prominence, and the bilateral symmetry are extracted. (1A-5) For wrinkles at the corners of the eyes, the number of wrinkles, depth, length, and bilateral symmetry are extracted as features.

[0039] (1B) Eyebrow movements Since eyebrow movements play an important role in expressing emotions and attitudes, the following features are extracted from the image data captured by camera 145. (1B-1) As for the vertical movement of the eyebrows, the height of the eyebrows, the range of vertical movement, and the left-right symmetry are extracted as features. (1B-2) For frown lines, the contraction strength of the corrugator supercilii muscle, the distance between the eyebrows, and the depth of the wrinkles are extracted as feature quantities. (1B-3) As features of the eyebrow inclination, the angle of inclination of the eyebrow arch and bilateral symmetry are extracted. (1C) Eye opening degree The degree to which the eyes are open indicates the degree of surprise or interest, and the following features are extracted from the image data captured by camera 145. (1C-1) The vertical distance of the palpebral fissure and the degree of eye opening are extracted as features to represent the distance between the upper and lower eyelids. (1C-2) To determine the degree of eye opening, the exposed area of ​​the iris and the exposed area of ​​the sclera are extracted as feature quantities.

[0040] (1D) Blink Blinking indicates tension and level of concentration, and the following features are extracted from the image data captured by camera 145. (1D-1) As the blinking frequency, the number of blinks per unit time is extracted as a feature. (1D-2) As the duration of blinking, the average duration of each blink is extracted as a feature. (1D-3) As a feature of the timing of left and right blinking, the time difference and asynchronous nature of left and right blinking are extracted. (1E) Lip movement Lip movements are deeply involved in speech and emotional expression, and the following features are extracted from the image data captured by camera 145. (1E-1) As the degree of opening and closing of the mouth, the opening width, opening area, and opening speed are extracted as feature quantities. (1E-2) As a measure of lip protrusion, the distance between the protrusion of the upper and lower lips and bilateral symmetry are extracted as features.

[0041] (1E-3) To determine the degree of corner of the mouth's retraction, the amount of displacement of the corner of the mouth in the left-right direction and the curvature of the corner of the mouth are extracted as features. (1E-4) As a measure of lip tension, the degree of contraction of the muscles surrounding the lips, such as the orbicularis oris muscle, levator anguli oris muscle, and depressor anguli oris muscle, is extracted as a characteristic quantity. (1F) Movement of the nose The movement of the nose expresses emotions such as disgust and anger, and the following features are extracted from the image data captured by camera 145. (1F-1) As features of the alae, the width of the alae, the area of ​​the nostrils, and the bilateral symmetry are extracted. (1F-2) As features of nasal wrinkles, the contraction strength of the procerus muscle and the depth of wrinkles on the upper part of the nose are extracted.

[0042] (1G) Cheek movement The movement of the cheeks represents changes in emotion and facial expression, and the following features are extracted from the image data captured by camera 145. (1G-1) The contraction strength of the buccinator muscle, the height of the cheek prominence, and the bilateral symmetry are extracted as features of the cheek fullness. (1G-2) As for cheek tension, the degree of tension in the buccinator muscle and the firmness of the skin are extracted as characteristic quantities. (1H) Jaw movement Jaw movements express confidence and tension, and the following features are extracted from the image data captured by camera 145. (1H-1) As a feature of jaw protrusion, the horizontal movement distance of the chin tip and the angle of the temporomandibular joint are extracted. (1H-2) As a characteristic of chin tuck, the tension of the muscles under the chin (such as the mentalis muscle) and the firmness of the skin are extracted as feature quantities.

[0043] (2) Gaze Eye movements are important nonverbal information that indicates interest, attention, and thought processes in communication, and the following features related to eye movements are extracted from the image data captured by camera 145. (2A) Eye contact Eye contact evaluates the degree of interest and attention towards the other person, and the following features are extracted from the image data captured by camera 145. (2A-1) As the eye contact frequency, the number of eye contacts per unit time is extracted as a feature from the image data captured by camera 145. (2A-2) As the duration of eye contact, the average duration of a single eye contact is extracted as a feature from the image data captured by camera 145. (2A-3) Types of eye contact are extracted as features from the image data captured by camera 145, such as fixation on the entire face of the other person, fixation on a specific part (eyes, mouth, nose, etc.), and avoidance of gaze.

[0044] (2B) Direction of line of sight The direction of the gaze is extracted from the image data captured by camera 145. (2B-1) The angle of the left and right eyeballs is measured as the horizontal direction of the line of sight, and the deviation from the center of the face is calculated as a feature from the image data. (2B-2) The vertical direction of the line of sight is measured, and the deviation from the center of the face is calculated as a feature from the image data. (2C) Gaze duration The stability of the gaze and the degree of concentration are extracted from the image data captured by camera 145. (2C-1) As a gaze time to a specific point, the gaze time to a particular point is measured from image data as a feature. (2C-2) As the eye movement speed, the angle of eye movement per unit time is calculated from the image data as a feature to quantify the movement of the eyes.

[0045] (2D) Pupil diameter Pupil size is related to the degree of emotion and interest, and pupil size is extracted from image data captured by camera 145. (2D-1) The pupil diameter is extracted as a feature from the image data. (2D-2) As pupil contraction and dilation, the amount and rate of change in pupil diameter in response to changes in lighting conditions and emotional changes are extracted as features from image data. (2E) Eye movements Eye movements are related to thought processes and emotions, and eye movements are extracted from image data captured by camera 145. (2E-1) Eye movements are classified into types such as saccadic movements (rapid jumping movements), smooth pursuit movements (smooth tracking movements), and convergence (movements in which both eyes turn inward simultaneously).

[0046] (2E-2) As eye movement velocity, the velocity of each eye movement is extracted from the image data as a feature. (2E-3) As the direction of eye movement, the horizontal and vertical components of each eye movement are extracted as features from the image data. (2E-4) As the eye movement frequency, the number of eye movements per unit time is extracted as a feature from the image data. (3) Gestures Gestures are body movements used to convey information without using words, and are an important nonverbal communication skill in language learning. The following gesture-related features are extracted from image data. (3A) Hand movements Hand movements play an important role in expressing emotions and conveying information, and these hand movements are extracted as features from the image data captured by camera 145.

[0047] (3A-1) The position of the hand is measured in 3D coordinates and relative to body parts (e.g., head, shoulders, torso) for each frame, and the trajectory of the hand movement is extracted as time-series data (features). (3A-2) As the speed of hand movement, the distance the hand moves and acceleration per unit time are measured, and the speed of hand movement is extracted and quantified as a feature. (3A-3) As a measure of the magnitude of hand movement, the range of hand movement and the complexity of the trajectory are analyzed, and the dynamism of the movement is extracted as a feature. (3B) Pointing Pointing is a type of gesture used to indicate a specific object or direction, and pointing is extracted from image data captured by camera 145. (3B-1) The direction vector of the fingertip and the horizontal and vertical angles are extracted as features from the image data to represent the direction of pointing (this makes it possible to identify the object being pointed at). (3B-2) As the frequency of pointing, the number of pointing actions per unit time is extracted as a feature from the image data. (3B-3) As the duration of pointing, the average duration of each pointing motion is extracted as a feature from the image data.

[0048] (3C) The orientation of the palm The orientation of the palms expresses emotions and attitudes, and the orientation of the palms is extracted as a feature from the image data captured by camera 145. (3C-1) To determine whether the palm is facing upwards or downwards, the angle between the palm's normal vector and the vertical axis is extracted as a feature from the image data. (3C-2) To determine whether the palm is facing forward or backward, the angle between the palm's normal vector and the line of sight is extracted as a feature from the image data. (3C-3) To determine whether the palm is facing inward or outward, the angle between the palm's normal vector and the left-right axis is extracted as a feature from the image data.

[0049] (3D)Applause Applause expresses joy or agreement, and the following items are extracted from the image data captured by camera 145. (3D-1) As the speed of clapping, the number of claps per unit time is extracted as a feature from the image data captured by camera 145. (3D-2) As a measure of the strength of the applause, the acceleration of the hands during applause and the sound pressure level are extracted as features from the image data captured by the camera 145 or the audio data acquired by the microphone 144. (3D-3) As the duration of clapping, the average duration of each clap is extracted as a feature from the image data captured by camera 145. (3E) Hand shape Hand gestures express emotions and attitudes, and the following items are extracted as features from the image data captured by camera 145. (3E-1) As the degree of hand opening and closing, the degree to which the fingers are spread and the distance between the fingers are extracted as features from the image data captured by camera 145, and the degree to which the hand is open is quantified.

[0050] (3E-2) As for the way the hand is held, the shape of the hand, such as a clenched fist, a light grip, or fingers extended, is extracted as a feature from the image data captured by camera 145. (3E-3) As for hand positions, hand positions such as interlocking fingers or overlapping hands are extracted as features from the image data captured by camera 145. (3F) Arm movements Arm movements complement explanations and emotional expressions, and the following items are extracted as features from the image data captured by camera 145. (3F-1) The position of the arm is measured in 3D coordinates and relative to body parts (e.g., head, shoulders, torso) for each frame, and the trajectory of the arm's movement is extracted as time-series data (features). (3F-2) As the speed of arm movement, the average distance traveled by the arm per unit time and the average acceleration are extracted as features from the image data captured by camera 145, and the speed of arm movement is quantified. (3F-3) To assess the magnitude of arm movement, the range of arm movement and the complexity of the trajectory are extracted as features from image data, and the dynamism of the movement is evaluated.

[0051] (3G) Elbow movements form gestures in conjunction with arm movements, and the following items are extracted as features from the image data captured by camera 145. (3G-1) To measure elbow flexion and extension, the angle of the elbow joint is extracted as a feature from the image data captured by camera 145, and the degree of elbow flexion and extension is quantified. (3G-2) As elbow rotation, the internal and external rotation angles of the elbow joint are extracted as features from image data captured by camera 145, and the rotational movement of the elbow is quantified. (3H) Shoulder movements express emotions and attitudes, and the following items are extracted as features from the image data captured by camera 145. (3H-1) As for the vertical movement of the shoulder, the vertical movement distance of the scapula is extracted as a feature from the image data captured by camera 145. (3H-2) As an anterior-posterior measure of the shoulder, the anterior-posterior displacement distance of the scapula is extracted as a feature from the image data captured by camera 145. (3H-3) As shoulder rotation, the internal and external rotation angles of the shoulder joint are extracted as features from the image data captured by camera 145.

[0052] (4) Body language Body language is nonverbal information that conveys emotions, attitudes, confidence, and other information through posture, movements, and bodily actions. The following body language features are extracted from image data. (4A) Posture (standing position) (4A-1) As foot width, the distance between both feet and the positional relationship between the left and right feet (parallel, V-shape, inverted V-shape, etc.) are extracted as features from the image data captured by camera 145. (4A-2) As the center of gravity, the bias in the left-right and front-back center of gravity positions is extracted as features from the image data captured by camera 145. (4A-3) As a measure of spinal curvature, the degree of the S-curve of the spine, kyphosis, lordosis, etc., are extracted as features from the image data captured by camera 145. (4A-4) As for the position of the scapula, the degree of adduction / abduction and upward / downward rotation of the scapula is extracted as a feature from the image data captured by camera 145. (4A-5) As head tilt, the forward, backward, and left-right tilt angles of the head are extracted as features from the image data captured by camera 145.

[0053] (4B) Posture (Sitting position) (4B-1) To determine how much the person is leaning against the backrest, the contact area with the backrest and the angle of inclination of the body are extracted as features from the image data captured by camera 145. (4B-2) Regarding the way the legs are crossed, whether or not the legs are crossed and which leg is on top are extracted as features from the image data captured by camera 145. (4B-3) Regarding the position of the arms, whether they are crossed, open, placed at the sides of the body, or placed on the table is extracted as a feature from the image data captured by camera 145. (4B-4) Regarding hand placement, whether the hands are placed on the knees, clasped, or clenched is extracted as a feature from the image data captured by camera 145.

[0054] (4C) Head movements (nodding) (4C-1) As the nodding speed, the average number of nods per unit time is extracted as a feature from the image data captured by camera 145. (4C-2) As nodding angles, the maximum and minimum nodding angles are extracted as features from the image data captured by camera 145. (4C-3) As nodding frequency, the number of nods is extracted as a feature from the image data captured by camera 145. (4D) Head movement (head shaking) (4D-1) As the pan / swing speed, the average number of pan / swing cycles per unit time is extracted as a feature from the image data captured by camera 145. (4D-2) As the panning angle, the maximum and minimum panning angles are extracted as features from the image data captured by camera 145. (4D-3) As the head-shaking frequency, the number of head shakes is extracted as a feature from the image data captured by camera 145.

[0055] (4E) Body orientation (4E-1) As a measure of the degree of facing the opponent, the angle between the front of the body and the opponent is extracted as a feature from the image data captured by camera 145. (4E-2) As a measure of the degree of body opening and closing, the degree to which the arms and legs are opened and the proportion of time the body is facing forward are extracted as features from the image data captured by camera 145. (4F) How to walk (4F-1) As stride length, the average walking distance per step is extracted as a feature from the image data captured by camera 145. (4F-2) As walking speed, the average walking distance per unit time is extracted as a feature from the image data captured by camera 145. (4F-3) As for arm swing, the magnitude of the arm swing, left-right symmetry, and the degree of agreement between the arm swing and the pace are extracted as features from the image data captured by camera 145.

[0056] (4G) Body tilt (4G-1) The degree of forward and backward leaning is determined by extracting features from the image data captured by camera 145, such as the relative position of the body's center of gravity and the center of the soles of the feet, and the angle of inclination of the trunk. (4G-2) As a measure of lateral tilt, the lateral displacement of the body's center of gravity and the lateral tilt angle of the trunk are extracted as features from the image data captured by camera 145. (5) Proxemics Proxemics is nonverbal information that indicates spatial relationships in interpersonal spaces, such as distance, angle, and spatial occupation. The following proxemics features are extracted from image data. (5A) Personal space The relative sense of distance from a fixed camera position is extracted as a feature from the image data captured by camera 145. Specifically, the change in distance is analyzed as a change in the sense of distance within the frame.

[0057] (5B) Angle of contact with another person The angle of the body relative to camera 145 is extracted as a feature from the image data captured by camera 145. Specifically, the body angle is observed in terms of frontality to the camera and changes in angle. (5C)Territoriality The awareness of spatial occupancy and the strength of self-assertion are evaluated based on movements and posture. The following items are extracted as features from the image data captured by camera 145. (5C-1) As spatial occupation, the method of occupying a fixed space is extracted as a feature from the image data captured by camera 145. (5C-2) As physical barriers, the presence and degree of physical barriers such as crossed arms or crossed legs are extracted as features from the image data captured by camera 145.

[0058] (6) Physical appearance Physical appearance (including grooming) is nonverbal information that significantly influences first impressions and self-expression. In particular, in business settings such as interviews and presentations, physical appearance greatly impacts the effectiveness of communication. The following physical appearance features are extracted from image data and scored based on evaluation criteria. (6A)Clothing (6A-1) As for color and design, the selection of colors and design are evaluated for their appropriateness to individual impressions and TPO (Time, Place, Occasion), and are extracted as features from the image data captured by camera 145. (6A-2) As for cleanliness and appropriateness, the cleanliness of the clothing and its appropriateness for the time, place, and occasion are extracted as features from the image data captured by camera 145.

[0059] (6B) Hairstyle (6B-1) Hairstyle and hair color: Features are extracted from image data captured by camera 145 that show the influence of hairstyle and hair color choices on an individual's impression and the occasion. (6B-2) As a measure of cleanliness, the cleanliness and maintenance status of the hair are extracted as features from the image data captured by camera 145. (6C) Accessories The appropriateness of the accessory type and the time, place, and occasion (TPO) is extracted as features from the image data captured by camera 145. (6D) Personal grooming (6D-1) As a measure of skin cleanliness and care, the condition of skin, nails, and beard care is extracted as features from image data captured by camera 145. (6D-2) As a way of grooming, the degree of cleanliness and well-maintained appearance is extracted as features from the image data captured by camera 145.

[0060] (6E) Posture The appropriateness of posture is evaluated by assessing the correctness of standing and sitting postures. Specifically, features such as whether the spine is straight and whether a natural posture is maintained are extracted from the image data captured by camera 145. As described above, the features extracted in this way can be used as input 220 for building and updating the trained evaluation model 210B. Below, an example of building and training (updating) the evaluation model 210B is described. For example, evaluation model 210B is constructed using supervised machine learning as a multimodal deep learning model and a large-scale language model (LLM). Supervised machine learning is a method of training an evaluation model using pairs of input data and correct labels. In this example, video data in a wide variety of languages ​​is used as input, and the evaluation scores for each evaluation item of nonverbal communication skills are used as the correct labels.

[0061] For example, the following features can be used as input data. (a) Facial expression features For example, facial expression features may include the intensity of Action Units (AUs) based on the Facial Action Coding System (FACS), the coordinates of facial landmarks (eyes, nose, mouth, etc.), and facial shape features. (b) Head posture For example, head posture features may include the three-dimensional rotation angles of the head (pitch angle, yaw angle, roll angle, etc.), the speed of head movement, and the trajectory of head movement. (c) Speech features For example, Mel-frequency cepstrum coefficients (MFCCs), linear predictive coding (LPC) coefficients, formant frequencies, and fundamental frequencies (F0) are used as features for speech characteristics.

[0062] (d) Prosodic features For example, prosodic features may include pitch, volume, speech rate, rhythm, and time-series data of intonation. The output data includes, for example, scores for each evaluation item of nonverbal communication skills (e.g., on a 5-point scale from 1 to 5). The number of nodes in the output layer corresponds to the number of types of nonverbal communication skills, and each node represents the evaluation score for the corresponding nonverbal communication skill's evaluation item. The learning algorithms used include deep learning architectures such as convolutional neural networks (CNNs), LSTMs (Long Short-Term Memory), and Transformers.

[0063] Convolutional neural networks are well-suited for extracting features from images and time-series data, and are particularly effective for facial expression recognition and gesture recognition. LSTM is a type of recurrent neural network well-suited for handling time-series data, and is particularly effective for recognizing time-series patterns in speech and prosody. Transformers utilize a self-aware mechanism and are well-suited for parallel processing. They are particularly effective when integrating multiple pieces of non-verbal information. As loss functions, we use the mean squared error (MSE) to handle regression problems and the cross-entropy error to handle classification problems. Mean squared error (MSE) is used to predict the score (continuous value) for each assessment item of nonverbal communication skills, while cross-entropy error is used to determine whether a particular nonverbal communication skill belongs to a particular category (for example, whether the user's facial expression is a smile or not).

[0064] The optimization algorithm used is Adam (Adaptive Moment Estimation). Generally, Adam is adopted as the default because it converges quickly during training and its hyperparameters are relatively easy to adjust. If Adam does not perform sufficiently, SGD (Stochastic Gradient Descent) or RMSprop (Root Mean Square Propagation) can also be used. As mentioned above, nonverbal communication skills encompass multiple modalities (types of information), such as facial expressions, eye contact, gestures, body language, and hand movements. To comprehensively analyze and evaluate these diverse nonverbal communication skills, it is effective to construct the evaluation model 210B as a multimodal deep learning model.

[0065] The following provides an overview of multimodal deep learning models. In the multimodal deep learning model, the following 220 features are used as input (see Figure 3). (1) Facial information Algorithms such as Haar Cascades, HOG+SVM, and MTCNN are used to detect the user's face region from the user's video recorded in the user image data storage unit 132F. Next, libraries such as Dlib, OpenFace, and MediaPipe Face Mesh are used to detect landmarks on the user's face, such as 68, 128, or 468 points. These detected landmarks are then used as features to represent changes in facial shape and expression. Using facial landmark information, texture information, or pre-trained models (e.g., VGGFace, FaceNet), it recognizes basic emotions (joy, sadness, anger, fear, surprise, disgust) and facial expressions (confusion, contempt, interest, etc.). To capture subtle changes in emotions and facial expressions, the movement of individual facial muscles is represented as an Action Unit (AU) based on the FACS (Facial Action Coding System), and its intensity is analyzed.

[0066] (2) Gaze information The direction of gaze is estimated based on facial landmark information and pupil position information. Furthermore, eye movements (saccades, smooth tracking, etc.), gaze duration, and changes in pupil diameter are recorded as time-series data and extracted as features. (3) Audio information Using the VAD (Voice Activity Detection) algorithm, silent sections are removed from the user's voice signal recorded in the user voice storage unit 132B, and only the speech sections are extracted. Next, acoustic features such as MFCC (Mel-Frequency Cepstral Coefficients), LPC (Linear Predictive Coding), formant frequencies, and fundamental frequencies (F0) are extracted. Furthermore, tools such as prosodylab-aligner are used to extract prosodic features such as pitch contour, intensity, and duration.

[0067] (4) Physical information Using posture estimation algorithms such as OpenPose, AlphaPose, and PoseNet, the system extracts the user's skeletal information and obtains time-series data of the position and angle of the body's joints. Next, feature quantities such as the magnitude, speed, direction, and joint angles of the user's body movements are calculated from the skeletal information. (5) Linguistic information To perform grammatical, vocabulary, and semantic analysis of spoken content, we will use a newly machine-learned large-scale language model (LLM). Based on the user's utterances, we analyze their emotions and intentions, as well as the context and intent of their language, and use this analysis to interpret their nonverbal communication skills. As described above, after features are extracted from each modality, preprocessing such as standardization and normalization is performed to unify the scale of the features.

[0068] Specifically, this improves the model's training efficiency and generalization performance by removing redundant or noisy features. Furthermore, we will integrate modalities. For example, an attention mechanism learns the relationships between features of different modalities, dynamically adjusts the importance of each modality according to the situation, and then uses tensor fusion to represent the features of different modalities as tensors, and integrates the modalities using methods such as tensor decomposition. In this case, a self-awareness mechanism and position encoding can be used to efficiently process multimodal data, including time-series and spatial data. Based on these integrated features, the evaluation score for each nonverbal communication skill is calculated (Output 230). Evaluation can be performed using regression models, classification models, or reinforcement learning.

[0069] For example, regression models that predict continuous values ​​can be used, such as linear regression, support vector regression (SVR), and random forest regression. Classification models that perform categorical classification can be used, such as logistic regression, support vector machine (SVM), decision trees, and random forests. According to the foreign language acquisition support system 500 of this embodiment, the following effects (benefits) can be obtained. The Foreign Language Acquisition Support System 500 enables users to improve not only their language skills but also their nonverbal communication skills. Specifically, it allows users to self-study exemplary nonverbal communication skills during conversations, thereby improving their communication abilities. As shown in this embodiment, the foreign language acquisition support system 500 can be built into a portable wireless communication device 100 such as a smartphone, which is currently widely used. Since portable wireless communication devices such as smartphones are usually carried with the user at all times, users can learn a foreign language (especially nonverbal communication skills) at their own convenience, regardless of time or place.

[0070] (Second Embodiment) There is a close relationship between what the interlocutor says and their nonverbal communication skills, and the evaluation of their nonverbal communication skills may change depending on what the interlocutor says. For example, generally speaking, a smile and a cheerful tone of voice are appropriate when expressing joy or surprise, while a calm tone of voice and restrained gestures are required when expressing sadness. Conversely, if you make a sad face or use a somber voice in a joyful situation, or smile or use a calm voice in an angry situation, it will appear extremely unnatural to the other person. Therefore, in the foreign language acquisition support system according to this embodiment, the fourth program 131D (evaluation means) uses the content spoken by the user as auxiliary linguistic data when evaluating the user's nonverbal communication skills.

[0071] The following items will be used as linguistic data. (a) Content of speech: Evaluate the grammar, vocabulary, and pronunciation of what was said. (b) Context: Analyze the context and intent of the utterance to help interpret nonverbal communication skills. (c) Phrase use: Evaluate the frequency and appropriateness of the use of specific phrases and expressions, and examine their relationship with nonverbal communication skills. (d) Speech fluency: Evaluate how smooth the speech is and analyze the consistency of nonverbal communication skills. The fourth program 131D (evaluation means) analyzes each item of this language data (for example, by extracting the aforementioned feature quantities) and reflects the analysis results in the evaluation of the user's nonverbal communication skills.

[0072] For example, in the aforementioned example, if the user shows an irritated expression or turns their face away during a conversation that should be gratifying, the fourth program 131D (evaluation tool) will give a low rating to the user's nonverbal communication skills (for example, in the items of facial expressions and gestures) even though the conversation content indicates that it should be a gratifying situation. In this way, by considering not only the image data captured by the camera 145 but also the user's language data recorded in the user voice storage unit 132C, it becomes possible to evaluate the user's nonverbal communication skills more accurately. Additionally, the following items can be used as supplementary language data.

[0073] (i) Paralanguage Paralinguistics are nonverbal elements contained in a speaker's voice and manner of speaking, playing an important role in conveying emotions, attitudes, intentions, and more. The following paralinguistic features are extracted from audio data and used as supplementary data. (i) Tone of voice Voice tone is an important element in expressing emotions and attitudes, and the following features related to voice tone are extracted from audio data. (I-A-1) As for the pitch of the voice, the average value, standard deviation, and range of variation of the fundamental frequency are extracted as features from the audio data, and finally the pattern of change in voice pitch is extracted. (I A-2) As for the brightness of the voice, the pitch of the spectral centroid and the proportion of high-frequency components are extracted as features from the audio data, and finally the brightness and softness of the voice are extracted.

[0074] (I A-3) In terms of vocal strength, sound pressure level and speech energy are extracted as features from the audio data, and finally, the intensity and impact of the voice are extracted. (I A-4) In terms of voice stability, the stability of the fundamental frequency and volume are extracted as features from the audio data, and finally, voice tremors and instability are extracted. (i) pitch Voice pitch is an important element in conveying emotions and the intent behind speech, and the following features related to voice pitch are extracted from audio data. (I B-1) As fundamental frequencies, the average fundamental frequency, highest fundamental frequency, and lowest fundamental frequency during speech are extracted as features from the speech data, and finally the overall characteristics of the voice pitch are extracted. (I-B-2) As for changes in pitch, the range of pitch variation, the pitch variation pattern (rising / falling), and the frequency of rising / falling are extracted as features from the audio data, and finally, the intonation and emotional expression of the voice are extracted.

[0075] (i) volume Voice volume is an important factor indicating intelligibility and confidence level, and the following features related to voice volume are extracted from audio data. (i C-1) In terms of volume, the average, maximum, and minimum values ​​of the sound pressure level are extracted as features from the audio data, and finally the overall characteristics of the volume of the voice are extracted. (iC-2) As the sound range, the distance the voice reaches is estimated from the sound data as a feature by comparing it with the sound attenuation characteristics and the ambient noise level. (D) Speaking speed Speaking speed is an important factor that reflects intelligibility and the speaker's personality, and the following features related to speaking speed are extracted from audio data. (I-D-1) Speech rate is measured by counting the average number of words spoken per minute and the average number of syllables spoken per minute as features from the speech data.

[0076] (I-D-2) As a way to analyze the interval between speech, the length of the silent interval (pause time) between utterances is measured as a feature from the speech data, and the interval between speech is analyzed. (E) Intonation Intonation plays a crucial role in conveying emotions and nuances in speech, and the following features related to intonation are extracted from audio data. (E-1) Intonation, such as rises and falls in pitch and intonation at the end of sentences, is extracted as features from the audio data, and finally the overall rhythm and emotion of the speech are extracted. (E-2) As accent, the position and degree of stress of words and phrases are extracted as features from the audio data, and finally, clarity and intelligibility of speech are extracted. (E-3) In terms of rhythm, the rhythmic patterns, regularity, and tempo of speech are extracted as features from the audio data, and ultimately the fluency and naturalness of speech are extracted.

[0077] (i) Volume Sound intensity is an important element used for emphasis and emotional expression, and the following features related to sound intensity are extracted from audio data. (I-F-1) In terms of how emphasis is applied, the words or phrases to be emphasized, the degree of change in emphasis, etc., are extracted as features from the audio data, and ultimately it is determined whether effective emphasis has been achieved. (I F-2) In terms of emphasis, differences in volume and richness of intonation are extracted as features from the audio data, and ultimately expressiveness and intelligibility are extracted. (I G) Voice quality Voice quality is an important factor that affects intelligibility and impression, and the following characteristics related to sound quality are extracted from audio data. (I-G-1) In terms of voice clarity, the proportion of high-frequency components and the low level of noise are extracted as features from the audio data, and finally the clarity and intelligibility of the voice are extracted. (I-G-2) As for the resonance of the voice, the richness of vocal cord vibration and the degree of resonance are extracted as features from the audio data, and ultimately the depth and richness of the voice are extracted.

[0078] (I-G-3) Regarding the roughness of the voice, the proportion of low-frequency components and the amount of noise are extracted as features from the audio data, and finally, the roughness and difficulty in understanding the voice are extracted. (I) Clarity of words Speech clarity is a crucial factor influencing the smoothness of communication, and the following features related to speech clarity are extracted from audio data. (I-H-1) As for the accuracy of pronunciation, features such as the accuracy of vowels and consonants and the accuracy of intonation are extracted from the audio data, and the clarity of pronunciation is ultimately determined. (I-H-2) In terms of articulation, clarity and fluency of pronunciation are extracted as features from the audio data. (I) Filler Fillers are words like "uh" and "um" that are unconsciously uttered during speech, and their excessive use can make speech difficult to understand. The following features related to fillers are extracted from speech data.

[0079] (I-1) As a feature of the frequency of use of fillers such as "ah" and "um", the average number of times fillers are used per unit time is extracted from the audio data. (I-2) As for the types, filler words such as "uh," "um," "well," and "um" are classified as features from the speech data. (b) silence Silence refers to a state of no speech. Silence is nonverbal information with diverse meanings, including pauses in conversation, time for thought, and expression of emotion. The following features related to silence are extracted from audio data. (A) Intentional silence (A-1) In terms of timing, the length of silent intervals (pauses) between utterances and the context before and after the pauses are extracted as features from the speech data, and the appropriateness of the timing is ultimately determined. (A-2) The length of silence is extracted as a feature from the audio data, such as less than 1 second, 1-2 seconds, 2-3 seconds, 3 seconds or more, and finally it is determined whether the length of silence is appropriate for the situation. (A-3) As the frequency of silence, the average number of silences per unit time is extracted as a feature from the audio data, and finally the frequency of silence is determined.

[0080] (b) Between the conversations (B-1) As the length of pauses, the length of short pauses during speech is extracted as a feature from the audio data. (B-2) As the frequency of pauses, the number of pauses per unit time is extracted as a feature from the audio data. (B-3) In terms of how a pose is taken, the context before and after the pose is extracted as a feature from the audio data, and finally, it is evaluated whether the pose is natural, unnatural, or drawn out. (C) Nodding in agreement (C-1) As for the timing of interjections, the timing of interjections in response to the other person's speech is extracted as a feature from the audio data. (C-2) Types of interjections such as "yes," "uh," "yes," and "I see" are classified from the audio data. (C-3) As the frequency of interjections, the average number of interjections per unit time is extracted as a feature from the audio data.

[0081] (h) How to use time Chronemics (time management) is nonverbal information that indicates attitudes and behaviors towards time, and it varies greatly depending on culture and situation. The following features related to time management can be extracted from audio data. (H A) Reaction time (H-A-1) The speed of response to a question is extracted as a feature from the audio data, specifically the time from when the question is asked until the answer is started. (H-A-2) As a measure of the speed of response to an action, the time from receiving an instruction to start the action is extracted as a feature from the audio or image data.

[0082] (H B) Pace of conversation (H B-1) As a measure of the speed of speaker changes, the time from when one speaker starts speaking until another speaker takes over is extracted as a feature from the audio data. (H B-2) In terms of the speed of topic development, the amount of time spent talking about one topic and the frequency of topic changes are extracted as features from the audio data. (H C) The timing of the conversation The silent periods between utterances, such as pauses in speech and silences between words, are extracted as features from the audio data, and their naturalness is evaluated. (H D) Auditory information (H-D-1) In terms of voice quality, features such as vocal clarity, breathiness, and the presence or absence of nasal voice are extracted from the audio data. (H-D-2) Laughter is extracted as a feature from the audio data, including its frequency, volume, and type (e.g., loud laughter, muffled laughter, etc.). (H-D-3) As a cough clearing, the frequency and timing of cough clearing are extracted as features from the audio data.

[0083] (Third embodiment) In the foreign language acquisition support system 500 according to the first embodiment, evaluation results for the user's language skills and nonverbal communication skills are generated. Furthermore, it is possible to develop these results to create a user-specific curriculum to supplement the language skills and nonverbal communication skills of users who received low evaluations in the evaluation results. The foreign language acquisition support system according to the third embodiment has a sixth program (not shown) in the application section 131 of the external memory 130, and this sixth program functions as a curriculum creation means that creates a curriculum (learning plan) that indicates future learning policies according to the user's evaluation results. Once the user's evaluation results (steps S180 and S210 in Figure 4) are generated, the central processing unit 121 starts the sixth program.

[0084] A database (not shown) listing countermeasures for each deficiency in language skills and nonverbal communication skills is pre-stored in the data storage unit 132. The sixth program searches this database for countermeasures to address the deficiencies pointed out in the evaluation results, and compiles these countermeasures into user-specific curricula for both language skills and nonverbal communication skills. Table 3 shows an example of a curriculum (learning plan) for language skills. (Table 3) TIFF0007939718000003.tif62139

[0085] Table 4 shows an example of a curriculum (learning plan) for nonverbal communication skills. (Table 4) TIFF0007939718000004.tif62160 These curricula will be displayed on display 142. According to the foreign language acquisition support system of this embodiment, users can obtain a curriculum (learning plan) tailored to their weaknesses in both language skills and non-verbal communication skills. Since this curriculum also provides practice sessions that focus on the user's weaknesses, users can efficiently overcome their weaknesses.

[0086] (Fourth embodiment) The foreign language acquisition support system according to the fourth embodiment stores a seventh program (not shown) in the application unit 131 of the external memory 130, and this seventh program has the function of creating new learning materials that allow the user to learn according to the curriculum created in the third embodiment. In other words, the seventh program involves creating new learning materials tailored to the user's weaknesses identified in the evaluation results. Learning materials are composed, for example, of learning text (subtitles) and images (still images) consisting of audio. The central processing unit 121 starts the seventh program and creates an image using image creation technology. The created image is stored, for example, in the sub-data storage unit 132G. Text generation can be performed, for example, using LLM (Large Language Models) and RAG (Retrieval-Augmented Generation). Thus, according to the foreign language acquisition support system of this embodiment, users can obtain new learning materials that are specifically tailored to their weaknesses, thereby improving their learning efficiency.

[0087] (Fifth embodiment) In the fourth embodiment, images (still images) are created as learning materials, but it is also possible to create a video in which learning text is spoken aloud. Furthermore, in addition to the audio, the text can also be displayed as subtitles in this video. The foreign language acquisition support system according to the fifth embodiment stores an eighth program (not shown) in the application section 131 of the external memory 130, and this eighth program has the function of creating new videos as learning materials that allow the user to learn according to the curriculum created in the third embodiment. The central processing unit 121 activates the eighth program and uses image creation technology and speech synthesis technology to create a video that conforms to the specified conditions. The created video is stored in the video storage unit 132A. For example, this video can be created as a dialogue-style video in which the user interacts with the person (character) in the video.

[0088] Figure 5 is a conceptual diagram of the foreign language acquisition support system according to this embodiment. As shown in Figure 5, the character 310 appears on the screen of display 142, and the user 320 faces the screen of display 142 and interacts with the character 310. The foreign language acquisition support system according to this embodiment stores the ninth, tenth, and eleventh programs (none of which are shown) in the application section 131 of the external memory 130. The ninth program has the function of creating a response to the voice spoken by the user 320 and converting it into speech; the tenth program has the function of selecting an appropriate nonverbal communication skill corresponding to the user 320's nonverbal communication skill; and the eleventh program has the function of creating subtitles in accordance with the instructions of the central processing unit 121 and displaying those subtitles on the screen of the display 142.

[0089] When user 320 begins a conversation with character 310, user 320's voice is collected by microphone 144 and sent as voice data to central processing unit 121. Upon receiving the voice data, central processing unit 121 activates the ninth program to create a response to the voice spoken by user 320 and converts it into voice. The converted response is then output from speaker 143 as voice spoken by character 310. In this way, an audio dialogue is established between character 310 on display 142 and user 320. Furthermore, when user 320 begins a conversation, the user 320's appearance and actions (nonverbal communication skills) are captured by camera 145, and this video is sent to the central processing unit 121 as image data. Upon receiving the image data, the central processing unit 121 activates the eighth program and uses image creation technology to create a video in which the character 310 performs actions that match those of the user 320. The created video is displayed on the display 142. In this way, a visual interaction is established between the character 310 on the display 142 and the user 320.

[0090] Upon receiving the image data, the central processing unit 121 executes the ninth and tenth programs (creating an audible and visual dialogue between the character 310 and the user 320) and simultaneously activates the fourth program 131D. This allows the trained second evaluation model 210B stored in the second data storage unit 132E to compare the exemplary nonverbal communication skills in conversation stored in the first data storage unit 132D with the nonverbal communication skills of the user 320 in the image data of the user 320. As a result of this comparison, the central processing unit 121 generates an evaluation result (see Table 2) of the user 320's nonverbal communication skills during conversation. Next, the central processing unit 121 starts the eleventh program, creates subtitles that reflect the evaluation results, and displays these subtitles 330 (see Figure 5) on the screen of the display 142. Thus, in the foreign language acquisition support system according to this embodiment, if a user 320 exhibits a low-rated nonverbal communication skill while interacting with character 310, subtitles 330 of that skill appear on display 142.

[0091] Therefore, users 320 can become aware of their shortcomings in nonverbal communication skills in real time and respond immediately, thereby improving learning efficiency. For example, if user 320's face doesn't move during a conversation, subtitle 330 might comment, "Their face is expressionless." Furthermore, it is also possible to offer advice following such criticism. For example, after pointing out that "your face is expressionless," you can then offer advice such as "try to smile more." This allows users 320 to recognize their shortcomings and understand in real time how to address them, thereby promoting the improvement of their nonverbal communication skills. Furthermore, it is also possible to configure the system so that the character 310 speaks the content of the subtitles 330, either in place of or alongside the subtitles 330 displayed on the screen of the display 142.

[0092] For example, the foreign language acquisition support system further includes a twelfth program (not shown), which functions as a subtitle-to-speech means for converting subtitles 330 into audio. The twelfth program, either in place of or alongside the subtitles 330, plays audio representing the subtitles 330 as the words of character 310 from the screen of display 142. Generally, humans understand information faster through hearing than through sight. Therefore, compared to using only subtitles 330, being directly pointed out by the character 310, the other party in the conversation, allows user 320 to easily and immediately recognize shortcomings in their nonverbal communication skills. This is particularly effective when user 320 does not have the time to read subtitles 330.

[0093] (Sixth Embodiment) Unlike verbal communication, nonverbal communication can vary significantly depending on cultural background (including religious background). This is because the same nonverbal communication skills may have different meanings depending on the cultural context. For example, the appropriate frequency and duration of eye contact vary depending on the cultural context. Furthermore, some gestures are only understood within specific cultural areas. The foreign language acquisition support system according to the sixth embodiment aims to evaluate appropriate nonverbal communication skills for users from diverse cultural backgrounds, taking into account the cultural backgrounds of both the user and the other party. Therefore, the foreign language acquisition support system according to this embodiment includes a cultural background information database (not shown) that stores information on cultural backgrounds collected from each country or region, and a thirteenth program (not shown). The cultural background information database is stored in the data storage unit 132, and the thirteenth program is stored in the application storage unit 131.

[0094] The thirteenth program retrieves the user's cultural background information from a cultural background information database and functions as an adjustment mechanism to apply adjustments to the evaluation of the user's nonverbal communication skills by the fourth program 131D, taking the user's cultural background information into account. A user's cultural background is either specified by the user themselves before starting foreign language learning, or determined by using an algorithm that adapts to the user's cultural background in real time during the assessment. For example, in one cultural sphere A, maintaining a smile is considered a healthy nonverbal communication skill, but in another cultural sphere B, smiling may be seen as mocking the other person. Therefore, when a user from cultural sphere A interacts with someone from cultural sphere B, Program 13 (Adjustment Mechanism) adjusts the evaluation of the user's nonverbal communication skills by Program 4 131D by giving a lower rating to the user's smile. Furthermore, in the interactive video shown in the fifth embodiment, it is also possible to display subtitles 330 advising the user 320 not to smile, or to have the character 310 verbally point this out. [Explanation of Symbols]

[0095] 500 Foreign language acquisition support system according to the first embodiment of the present invention 100 Portable wireless communication devices 120 Control Unit 121 Central Processing Unit 130 External memory 131 Application Storage Unit 132 Data Storage Unit 142 displays 145 Camera 210 Evaluation Models 310 characters 320 users 330 subtitles

Claims

1. A foreign language learning support system that assists users in learning foreign languages, An imaging means for capturing images of the user speaking according to the audio of the video, The primary memory system that remembers the speaker's exemplary nonverbal communication skills during a conversation, A second memory means that stores a trained assessment model that evaluates the nonverbal communication skills of the speaker during the conversation, An evaluation means that uses the learned evaluation model stored in the second memory means to compare the exemplary nonverbal communication skills stored in the first memory means with the nonverbal communication skills of the user captured by the imaging means, and evaluates the user's nonverbal communication skills. A display means for displaying the evaluation created by the evaluation means, Equipped with, The evaluation means evaluates the user's nonverbal communication skills with respect to at least one evaluation item from among facial expressions, gaze, gestures, body language, proxemics, physical appearance, visual focus, auditory information, and cultural background. The evaluation means is a foreign language acquisition support system that, when evaluating the user's nonverbal communication skills, uses the user's conversation content as supplementary linguistic data and determines whether the user's nonverbal communication skills are consistent with the user's conversation content.

2. The foreign language acquisition support system according to claim 1, characterized in that the evaluation means extracts feature quantities that quantitatively represent the characteristics of each of the evaluation items from the data of the user's image captured by the imaging means, and compares these feature quantities with exemplary nonverbal communication skills stored in the first storage means.

3. A cultural background information database that stores cultural background information for each country or region, An adjustment means that reads the user's cultural background information from the cultural background information database and makes adjustments to the evaluation of the user's nonverbal communication skills by the evaluation means, taking the user's cultural background information into consideration. The foreign language acquisition support system according to claim 1, further comprising the features described above.

4. The trained evaluation model is generated by machine learning to evaluate the speaker's nonverbal communication skills using exemplary nonverbal communication skills stored in the first memory means as a criterion. The input to the aforementioned trained evaluation model is image data of the user's nonverbal communication skills during a conversation, captured by the aforementioned imaging means. The foreign language acquisition support system according to claim 1, characterized in that the output from the trained evaluation model is an evaluation of the user's nonverbal communication skills during conversation, using the exemplary nonverbal communication skills as the criterion, by performing a predetermined calculation process.

5. The foreign language acquisition support system according to claim 1, further comprising a curriculum creation means for creating a curriculum specifically tailored to the user in order to compensate for the user's nonverbal communication skills, which were poorly evaluated by the evaluation means.

6. The foreign language acquisition support system according to claim 5, further comprising a learning material creation means for creating learning materials in accordance with the curriculum created by the curriculum creation means.

7. The foreign language acquisition support system according to claim 6, characterized in that the learning material is a video, and the video is an interactive video in which the user and the characters engage in a dialogue.

8. The system further includes a subtitle generation means for displaying subtitles on the screen of the aforementioned video, The foreign language learning support system according to claim 7, characterized in that, when the subtitle generation means displays a subtitle to that effect and advice on the nonverbal communication skills, at least the subtitle to that effect, on the screen when the user's nonverbal communication skills, which were poorly evaluated by the evaluation means, appear during the dialogue.

9. The system further includes a means for converting the aforementioned subtitles into audio, The foreign language learning support system according to claim 8, characterized in that the subtitle audio conversion means plays audio of the subtitles from the screen in place of or together with the subtitles.

10. A portable wireless communication device incorporating a foreign language acquisition support system according to any one of claims 1 to 9.

11. A program that causes a computer to execute a foreign language learning support method that assists users in learning a foreign language, A first process involves capturing images of the user speaking according to the audio in the video, A second process evaluates the user's nonverbal communication skills by comparing exemplary nonverbal communication skills stored in a memory means that has previously stored exemplary facial expressions, gestures, and other exemplary nonverbal communication skills of the speaker during a conversation, using a trained evaluation model that evaluates the speaker's nonverbal communication skills during a conversation, with the user's nonverbal communication skills captured in the first process. A third process that displays the evaluation created in the second process, Equipped with, In the second process described above, the user's nonverbal communication skills are evaluated with respect to at least one evaluation item from among facial expressions, gaze, gestures, body language, proxemics, physical appearance, visual focus, auditory information, and cultural background. In the second process, the program uses the user's conversation content as supplementary language data when evaluating the user's nonverbal communication skills, and determines whether the user's nonverbal communication skills are consistent with the user's conversation content.

12. The program according to claim 11, characterized in that, in the second process, feature quantities that quantitatively represent the characteristics of each of the evaluation items are extracted from the user's image data, and these feature quantities are compared with exemplary nonverbal communication skills stored in the storage means.

13. The program according to claim 11, characterized in that, in the second process, the user's cultural background information is read from a cultural background information database that stores cultural background information for each country or region, and adjustments are made to the evaluation of the user's nonverbal communication skills taking the user's cultural background information into consideration.

14. The aforementioned trained evaluation model is generated by machine learning to evaluate the speaker's nonverbal communication skills using exemplary nonverbal communication skills stored in the memory means as a criterion. The input to the aforementioned trained evaluation model is image data capturing the user's nonverbal communication skills during conversation. The program according to claim 11, characterized in that the output from the trained evaluation model is an evaluation of the user's nonverbal communication skills during conversation, using the exemplary nonverbal communication skills as the criterion, by performing a predetermined calculation process.

15. The program according to claim 11, further comprising a fourth process of creating a curriculum specifically tailored to the user in order to compensate for the user's nonverbal communication skills, which were poorly evaluated in the second process.

16. The program according to claim 15, further comprising a fifth process for creating learning materials in accordance with the curriculum created in the fourth process.

17. The learning materials are videos. The program according to claim 16, characterized in that the video is an interactive video in which the user and the characters engage in a dialogue.

18. The program according to claim 17, further comprising a fifth process in which, if the user's nonverbal communication skills, which were poorly evaluated in the second process, appear during the dialogue, a subtitle indicating that fact and at least a subtitle indicating advice regarding the nonverbal communication skills appear on the screen.

19. A sixth process to convert the aforementioned subtitles into audio, A seventh process involves playing audio from the screen that is an audio representation of the subtitles, either in place of or along with the subtitles. The program according to claim 18, further characterized by having the following features.

20. A portable wireless communication device incorporating the program described in any one of claims 11 to 19.

Images