Information processing device and information processing method
The information processing device enhances language learning by visually and audibly illustrating the differences and relationships between target language and native language sounds, addressing the challenge of distinguishing similar sounds.
Patent Information
- Application Number
- JP2023077516
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2043-05-09
AI Technical Summary
Existing language learning methods do not adequately account for the relationship between the sounds of the target language and the learner's native language, making it difficult for users to distinguish between similar sounds.
An information processing device and method that acquires voice information, estimates the vocalization state of a user, and displays animated utterance state images, superimposing feature points for both the target language and the learner's native language sounds to highlight differences and similarities.
Facilitates effective language learning by visually and audibly demonstrating the distinctions and relationships between sounds, improving pronunciation accuracy and efficiency.
Smart Images

Figure 2024161861000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing method, and the like. [Background technology]
[0002] Conventionally, there are known methods for visualizing a user's vocalization state based on voice information. For example, Patent Document 1 discloses a pronunciation action visualization device that converts voice data into vocal tract parameters, generates a coordinate vector sequence based on the vocal tract parameters and a classifier, and displays an image based on the coordinate vector sequence. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2013 / 031677 Summary of the Invention [Problem to be solved by the invention]
[0004] The method of Patent Document 1 displays the movement of the tongue etc. accompanying speech as an image, but does not take into account the relationship between the sounds of the language being studied (e.g., English) and the sounds of other languages (e.g., the learner's native language).
[0005] According to some aspects of the present disclosure, it is possible to provide an information processing device, an information processing method, and the like that appropriately support a user who is learning a language. [Means for solving the problem]
[0006] One aspect of the present disclosure includes a voice information acquisition unit that acquires voice information of a user learning a first language, an analysis processing unit that estimates a speech state representing the state of the user's jaw and tongue based on analysis processing of the voice information, and a display processing unit that performs processing to display an animation of multiple speech state images based on a time-series change in the estimated speech state, wherein the display processing unit relates to an information processing device that performs processing to superimpose a first feature point identified based on the speech state when pronouncing a first sound in the first language and a second feature point identified based on the speech state when pronouncing a second sound similar to the first sound in a second language different from the first language on the animation display of the multiple speech state images.
[0007] Another aspect of the present disclosure relates to an information processing method that acquires voice information of a user who is learning a first language, estimates a speech state representing the state of the user's jaw and tongue based on an analysis process of the voice information, performs a process of displaying a plurality of speech state images as an animation based on a time-series change in the estimated speech state, and in the display process, performs a process of superimposing and displaying a first feature point identified based on the speech state when pronouncing a first sound in the first language and a second feature point identified based on the speech state when pronouncing a second sound similar to the first sound in a second language different from the first language on the animation display of the plurality of speech state images. [Brief description of the drawings]
[0008] [Figure 1] 1 is a configuration example of an information processing system including an information processing device. [Diagram 2] 1 is a configuration example of a server system. [Diagram 3] 2 is a configuration example of a terminal device. [Figure 4] 4 is a flowchart illustrating a process of the present embodiment. [Diagram 5] FIG. 1 illustrates a process for plotting a given sound in coordinate space based on voicing state. [Figure 6]13 is an example of a display screen in which feature points representing a first sound and a second sound are superimposed on an animation display. [Figure 7] 13 is an example of a display screen in which feature points representing a first sound and a second sound are superimposed on an animation display. [Figure 8A] This is an example of a display screen when learning a given sound (word). [Figure 8B] This is an example of a display screen when learning a given sound (word). [Figure 9] 4 is a flowchart illustrating a process of the present embodiment. [Figure 10A] 13 is an example of a display screen used for learning using a word list. [Figure 10B] 13 is an example of a display screen used for learning using a word list. [Figure 10C] 13 is an example of a display screen used for learning using a word list. [Figure 10D] 13 is an example of a display screen used for learning using a word list. [Figure 11A] 13 is an example of a screen displaying phonetic symbols of character information. [Figure 11B] 13 is an example of a screen displaying phonetic symbols of image information. [Figure 12] 4 is a flowchart illustrating a process of the present embodiment. [Figure 13] This is an example of a screen that displays words in association with daily actions. [Figure 14] 4 is a flowchart illustrating a process of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, the same or equivalent elements are given the same reference numerals, and duplicated explanations are omitted. Note that the present embodiment described below does not unduly limit the contents described in the claims. In addition, all of the configurations described in the present embodiment are not necessarily essential configurations of the present disclosure.
[0010] 1. System configuration example Fig. 1 is a configuration example of an information processing system 10 including an information processing device according to the present embodiment. The information processing system 10 according to the present embodiment includes a server system 100 and a terminal device 200. However, the configuration of the information processing system 10 is not necessarily limited to that shown in Fig. 1, and various modifications are possible, such as omitting some components and adding other components. The same applies to Figs. 2 and 3 described below, in that modifications such as omitting or adding components are possible.
[0011] The information processing device of this embodiment corresponds to, for example, the server system 100. However, the method of this embodiment is not limited to this, and the processing of the information processing device described in this specification may be executed by distributed processing using the server system 100 and other devices. For example, the information processing device of this embodiment may include both the server system 100 and the terminal device 200. The information processing device of this embodiment may also be the terminal device 200. Hereinafter, in this specification, an example in which the information processing device is mainly the server system 100 will be described.
[0012] The server system 100 may be one server or may be configured to include multiple servers. For example, the server system 100 may be configured to include a database server and an application server. The database server stores various data such as information on vowels and consonants of a first language (a language to be studied, for example, English) and information on vowels and consonants of a second language (a language other than the first language, for example, Japanese). The application server executes the process described later with reference to FIG. 4 and the like. The multiple servers here may be physical servers or virtual servers. In addition, when a virtual server is used, the virtual server may be provided in one physical server or may be distributed and arranged in multiple physical servers. As described above, the specific configuration of the server system 100 in this embodiment can be modified in various ways.
[0013] The terminal device 200 is a device used by a user who uses a service (e.g., a language learning service) realized by the information processing system 10. The terminal device 200 may be a PC (Personal Computer), a mobile terminal device such as a smartphone, or another device having the functions described in this specification.
[0014] The server system 100 is communicably connected to the terminal device 200 via, for example, a network. The network here is, for example, a public communication network such as the Internet, but may also be a LAN (Local Area Network) or the like.
[0015] The information processing system 10 of the present embodiment may be a system that provides learning support to a user who is learning a first language, for example. The user here may be a user whose native language is a second language different from the first language. Below, an example in which the first language is English and the second language is Japanese will be described. However, the first language and the second language may be different languages, and the combination is not limited to this. For example, when the first language is English, the second language may be Thai, Indonesian, or another language. The first language may also be a language other than English.
[0016] 2 is a functional block diagram showing a detailed configuration example of the server system 100. The server system 100 includes a processing unit 110, a storage unit 120, and a communication unit 130, for example, as shown in FIG.
[0017] The processing unit 110 of this embodiment is configured by the following hardware. The hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware may be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC) or a field-programmable gate array (FPGA). The one or more circuit elements are, for example, a resistor, a capacitor, etc.
[0018] The processing unit 110 may be realized by the following processor. The server system 100 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information is, for example, a program and various data. The program may include a program that causes the server system 100 to execute the processing described in this specification. The processor includes hardware. The processor can be various processors such as a central processing unit (CPU), a graphics processing unit (GPU), and a digital signal processor (DSP). The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the function of the processing unit 110 as processing. The instruction here may be an instruction from an instruction set that constitutes a program, or an instruction that instructs the hardware circuitry of the processor to operate.
[0019] The processing unit 110 includes, for example, a voice information acquisition unit 111, an analysis processing unit 112, and a display processing unit 113. As described above, the information processing device according to the present embodiment may be realized by distributed processing of the server system 100 and the terminal device 200, or may be the terminal device 200. Therefore, at least a part of the voice information acquisition unit 111, the analysis processing unit 112, and the display processing unit 113 may be provided in the terminal device 200.
[0020] The voice information acquisition unit 111 acquires voice information of a user who is learning a first language. The voice information here is specifically data related to the voice of the user speaking words and sentences in the first language. For example, the terminal device 200 includes a microphone that records the user's voice, and the voice information acquisition unit 111 acquires the voice information from the terminal device 200. However, a device that acquires the voice information may be provided separately from the terminal device 200.
[0021] The analysis processing unit 112 performs an analysis process on the voice information to estimate the vocalization state representing the state of the jaw and the state of the tongue of the user. For example, in this embodiment, machine learning may be performed on the learning voice information based on training data to which the vocalization state corresponding to the voice information is assigned as correct answer data. The analysis processing unit 112 obtains a trained model that is a result of the machine learning, and performs the analysis process by inputting the voice information to be analyzed into the trained model.
[0022] The display processing unit 113 performs a process of displaying a plurality of utterance state images as animations based on the time-series change of the estimated utterance state. The utterance state image is, for example, an image in which the position of the tongue and the jaw can be changed, and is, for example, an image showing a profile of a person on a screen described later with reference to Figs. 6-7. That is, the display processing unit 113 may perform a process of displaying a screen described later with reference to Figs. 6-7 and the like. For example, the display processing unit 113 generates a plurality of utterance state images based on the time-series change of the utterance state, and performs a process of displaying the plurality of utterance state images sequentially (animation display) on the display unit 240 of the terminal device 200. Note that the display processing unit 113 may obtain information for generating (specifying) the utterance state image, instead of generating the utterance state image itself. For example, candidates of a plurality of utterance state images to be the object of animation display are stored in the terminal device 200, and the display processing unit 113 may transmit information to the terminal device 200 for specifying the utterance state image to be the object of display from the plurality of candidates. In this case as well, the animation display of the plurality of utterance state images is realized by the processing of the display processing unit 113.
[0023] The storage unit 120 is a work area for the processing unit 110, and stores various information. The storage unit 120 can be realized by various memories, and the memory may be a semiconductor memory such as an SRAM, a DRAM, a ROM, or a flash memory, a register, a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device.
[0024] The storage unit 120 stores, for example, a trained model used in processing in the analysis processing unit 112. The storage unit 120 may also store information (described later with reference to FIG. 5) that associates sounds in the first language with positions in a coordinate space (for example, a two-dimensional coordinate plane). In addition, the storage unit 120 can store various other information related to the processing of this embodiment.
[0025] The communication unit 130 is an interface for performing communication via a network, and includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. The communication unit 130 may operate under the control of the processing unit 110, or may include a processor for communication control different from the processing unit 110. The communication unit 130 is an interface for performing communication according to, for example, TCP / IP (Transmission Control Protocol / Internet Protocol). However, the specific communication method can be modified in various ways.
[0026] 3 is a block diagram showing a detailed configuration example of the terminal device 200. The terminal device 200 includes a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, and an operation unit 250.
[0027] The processing unit 210 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processing unit 210 may also be realized by a processor. The processor may be of various types, such as a CPU, a GPU, or a DSP. The processor executes instructions stored in the memory of the terminal device 200, thereby realizing the function of the processing unit 210 as processing.
[0028] The storage unit 220 is a work area for the processing unit 210, and is realized by various types of memory such as an SRAM, a DRAM, or a ROM.
[0029] The communication unit 230 is an interface for performing communication via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 performs communication with the server system 100 via, for example, the network.
[0030] The display unit 240 is an interface that displays various information, and may be a liquid crystal display, an organic EL display, or other display type. The display unit 240 displays various screens, which will be described later, using, for example, Figures 6 to 8B, 10A to 11B, and 13.
[0031] The operation unit 250 is an interface that accepts operation input by a user. The operation unit 250 may be, for example, buttons or the like provided on the terminal device 200. The display unit 240 and the operation unit 250 may be a touch panel that is integrally configured.
[0032] A display processing unit (for example, the display processing unit 113 of the server system 100) of the information processing device according to this embodiment performs a process of displaying a first feature point specified based on a vocalization state when a first sound in a first language is pronounced, and a second feature point specified based on a vocalization state when a second sound similar to the first sound in a second language different from the first language, superimposed on an animation display of a plurality of vocalization state images. For example, when the English words hat, hut, and hot are pronounced, ae, α, and Λ are included as vowels. Note that ae, α, and Λ here are substituted for phonetic symbols by other characters for convenience of notation, and are replaced with phonetic symbols according to Table 1 below. The fact that each character of ae, α, and Λ is replaced with a phonetic symbol according to Table 1 is the same in other parts of this embodiment. For example, English corresponds to the first language, and at least one of ae, α, and Λ corresponds to the first sound. These are vowels that should be clearly distinguished in English, but to a user who speaks Japanese, they are all recognized as sounds close to "a". For example, Japanese corresponds to a second language, and "a" corresponds to the second sound. In the method of this embodiment, when there are sounds that are easily misunderstood or confused by speakers of the second language, it is possible to arrange these at different positions on the animation display. As a result, the difference between "ae" and "a", the difference between "α" and "a", the difference between "Λ" and "a", etc. can be clearly presented to the user, so that it is possible to appropriately support the user in learning the first language. A specific method will be described later with reference to FIG. 6, etc.
[0033] [Table 1]
[0034] In addition, a part or all of the processing performed by the information processing device (information processing system 10) of the present embodiment may be realized by a program. The processing performed by the information processing device may be processing performed by the processing unit 110 of the server system 100, processing performed by the processing unit 210 of the terminal device 200, or both.
[0035] The program according to this embodiment can be stored in a non-transitory information storage medium, which is a medium readable by a computer, for example. The information storage medium can be realized by, for example, an optical disk, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on the program stored in the information storage medium. That is, the information storage medium stores a program for making a computer function as the processing unit 110 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for making a computer execute each step described later with reference to FIG. 4 and the like.
[0036] The technique of the present embodiment can be applied to an information processing method including the following steps. The information processing method includes a step of acquiring voice information of a user who is learning a first language, estimating a speech state representing the state of the user's jaw and tongue based on an analysis process of the voice information, and performing a display process of displaying a plurality of speech state images as an animation based on a time-series change in the estimated speech state. In the step of performing the display process, the information processing method performs a process of displaying a first feature point identified based on the speech state when pronouncing a first sound in the first language and a second feature point identified based on the speech state when pronouncing a second sound similar to the first sound in a second language different from the first language, superimposed on the animation display of the plurality of speech state images.
[0037] 2. Processing Details The processing of this embodiment will be described in detail below.
[0038] 2.1 Basic flow of pronunciation practice Fig. 4 is a flowchart for explaining the processing of the information processing device of this embodiment. Fig. 4 shows the processing executed when, for example, a user of the terminal device 200 studies a first language. First, in step S101, the operation unit 250 of the terminal device 200 accepts an input operation of a study item by the user. For example, the information processing system 10 of this embodiment may perform a process for supporting the pronunciation study of a predetermined sound in the first language. In this case, in step S101, the operation unit 250 may accept the selection of a specific sound to be studied.
[0039] For example, as described above, for Japanese users, the English sounds ae, α, and Λ are all close to the Japanese sound "a," so they may not be easy to learn. Therefore, the information processing system 10 may perform a process of supporting learning about the pronunciation of the English sounds ae, α, and Λ by accepting a selection input of these sounds. The terminal device 200 transmits information specifying the English sound (first sound of the first language) determined by the operation of the operation unit 250 to the server system 100. Note that the operation unit 250 may accept a selection operation of a word (e.g., hat) containing the sound, rather than a selection operation of a specific sound such as ae, α, or Λ.
[0040] In step S102, the processing unit 110 performs a process of identifying a Japanese sound (a second sound of a second language) similar to an English sound. For example, the storage unit 120 of the server system 100 may store table data in which sounds in English are associated with sounds in Japanese similar to the sounds. The processing unit 110 identifies the similar Japanese sound based on the input result of the English sound and the table data. Note that the specific order of the processes in steps S101 and S102 is not limited to this as long as the association between the English sound and the Japanese sound is identified. For example, a Japanese sound (e.g., "a") may be selected in step S101, and an English sound (e.g., ae, α, Λ) corresponding to the sound may be identified as the sound to be learned in step S102. Various modifications of the specific process are possible.
[0041] In step S103, the processing unit 110 performs a process of identifying the position of the first sound (ae, α, Λ) and the position of the second sound ("a") in a given coordinate space. Fig. 5 is a diagram showing an example of a coordinate space (coordinate plane) and an example of the positions corresponding to each sound.
[0042] As shown in FIG. 5, the coordinate space here may have a first axis representing the position of the tongue in the up-down direction (vertical direction), and a second axis representing the position of the tongue in the front-back direction. Here, front-back means the front direction (the direction the face is facing) of a person, and the opposite direction means back. Furthermore, up-down means the direction from a reference point (e.g., the center) of the head to the top of the head is up, and the opposite direction (e.g., the direction from the reference point to the neck) is down. Here, the position of the tongue may represent the position of a specific representative point in the tongue, or the position of a part that moves the most when a target sound is produced may be selected for each sound. In the example of FIG. 5, the feature point of each sound is determined based on the position of the tongue in a plane (i.e., a cross section of a person's face when viewed from the side) defined by the front-back and up-down directions of a person. In the example of FIG. 5, the vertical axis (first axis) may represent the degree of jaw opening, with the upward direction of the vertical axis representing a narrower jaw opening and the downward direction representing a wider jaw opening.
[0043] For example, the storage unit 120 of the server system 100 may store table data in which sounds in each language are associated with values on each axis of the sounds. In the example shown in FIG. 5, the storage unit 120 of the server system 100 may store table data in which sounds in each language are associated with the tongue position in an ideal pronunciation state of the sounds. The ideal pronunciation state here may be determined, for example, from the pronunciation result of a user whose native language is the target language. The processing unit 110 determines a first feature point representing a position corresponding to the first sound based on the first sound identified in steps S101-S102 and the table data. Similarly, the processing unit 110 determines a second feature point representing a position corresponding to the second sound based on the second sound and the table data. Note that the first feature point and the second feature point are not limited to a point in the coordinate space, and may be an area having a certain size.
[0044] Note that examples of the coordinate space are not limited to the above. For example, the axis of the coordinate space may correspond to information on the state of the oral cavity other than the tongue. The state of the oral cavity other than the tongue may include information representing the state of the upper jaw, lower jaw, palate, lips, etc. In other words, the first feature point and the second feature point may be points or areas corresponding to the positions of the jaw, palate, lips, etc. during speech.
[0045] The above process makes it possible to plot the English sound to be studied (first sound) and the similar Japanese sound (second sound) on the coordinate space. Furthermore, the information processing system 10 supports learning by having the user actually speak and clearly indicating the relationship between the result and the first sound and the second sound.
[0046] FIG. 6 is an example of a screen displayed by the processing of this embodiment, for example, a screen displayed on the display unit 240 of the terminal device 200 according to the processing of the display processing unit 113. For example, an English learning application may be installed in the storage unit 220 of the terminal device 200, and the processing unit 210 may operate according to the English learning application to display the screen shown in FIG. 6. The English learning application here may be an application that operates by communicating with the server system 100. For example, the user of the terminal device 200 selects the first sound by performing a predetermined operation from the home screen of the English learning application (step S101). The display processing unit 113 may instruct the English learning application based on the selection input and the information acquired by the processing of steps S102-S103, and the English learning application may display the screen shown in FIG. 6 based on the instruction. Although not described below, the operation of the terminal device 200 in the following description may also be realized by the English learning application.
[0047] The screen shown in FIG. 6 displays, for example, a first sound (ae, α, Λ) and a second sound ("a"). In addition, an image of a person representing the user's vocalization state is displayed. FIG. 6 shows an example in which a simplified profile of a person is displayed to show the movement of the tongue and jaw, but other images that can illustrate the user's vocalization state may be used. The image of the person may be an image that changes according to the user's vocalization state in real time, as will be described later using steps S104 and onwards.
[0048] Furthermore, as shown in FIG. 6, in this embodiment, a first feature point P1 representing a first sound and a second feature point P2 representing a second sound may be displayed superimposed on an image of a person representing a vocalization state. In FIG. 6, circular regions representing ae, α, and Λ are illustrated as the first feature point P1. Also, in FIG. 6, only the character "a" is displayed as the second feature point P2. However, the second feature point P2 may also be displayed as a circular region or the like. For example, as shown in FIG. 5, when each feature point is arranged using a coordinate system representing the position of the tongue, by displaying the first feature point and the second feature point in association with the movable range of the tongue in the profile of the person in FIG. 6, it is possible to present the user with the difference in the state of the tongue when vocalizing each sound in an easy-to-understand manner.
[0049] For example, as shown in Fig. 6, the display unit 240 may display text such as "Compare the tongue position for the Japanese "a" and the English vowels." This makes it possible to display the difference in pronunciation between the first sound and the second sound (the difference in tongue position in the example of Fig. 6) in an easy-to-understand manner.
[0050] In FIG. 6, when a "Try" button at the bottom left of the screen is selected, the information processing system 10 starts acquiring voice information based on the user's voice and processing the voice information.
[0051] Specifically, in step S104, the information processing system 10 acquires voice information generated by the user's voice. For example, in step S104, the display unit 240 of the terminal device 200 performs a display process to prompt the user to speak a predetermined word including a first sound. Then, a microphone included in the terminal device 200 acquires the voice information by recording the user's voice, and the voice information acquisition unit 111 receives the voice information via a network.
[0052] In step S105, the analysis processing unit 112 performs an analysis process of the voice information. For example, the analysis processing unit 112 may obtain formants of the voice information by performing a frequency analysis of the voice information. A formant is a frequency band in the frequency spectrum of a voice that has a stronger intensity than surrounding frequency bands. Then, in step S106, the analysis processing unit 112 estimates the user's real-time speech state based on the formant detection result. The speech state here includes information representing the positions of the tongue and jaw (upper jaw, lower jaw). The speech state may also include information representing the positions of the palate and lips.
[0053] For example, in this embodiment, a learning process may be performed based on training data in which a utterance state is associated as correct answer data with a feature amount including formant information. The learning process generates a trained model that outputs a corresponding utterance state when a feature amount including formants is received as an input. The analysis processing unit 112 may read the trained model from the storage unit 120 and input the feature amount obtained in step S105 to the trained model to obtain the utterance state (step S106). Note that the input data for the model is not limited to the feature amount obtained from the speech information, and the speech information itself may be used as the input data.
[0054] Alternatively, the analysis processing unit 112 may determine the degree of similarity between the formant of the sound uttered by the user and the formant of the first sound or the second sound, and identify the utterance state based on the degree of similarity. For example, since the ideal utterance state of the first sound is considered to be known, the analysis processing unit 112 may determine the utterance state in real time such that the closer the utterance of the user is to the first sound, the closer the utterance state of the first sound is to the utterance state of the first sound.
[0055] Furthermore, a method for estimating the state of the tongue and jaw based on voice information is known, such as in Patent Document 1, and the same method as in Patent Document 1 may be used for the processes of steps S105 and S106 according to the present embodiment. Since the method in Patent Document 1 is publicly known, a detailed description thereof will be omitted.
[0056] In step S107, the analysis processing unit 112 identifies a point in the coordinate space representing the utterance state based on the utterance state obtained in step S106. As described above with reference to Fig. 5, it is possible to identify values (coordinate values) on each axis of the coordinate space based on the utterance state.
[0057] In step S108, the display processing unit 113 performs animation display of a person showing a vocalization state. FIG. 7 is an example of a screen displayed (updated) in step S108. As described above, the animation display may be a screen displaying a person's head including the jaw and tongue. For example, the analysis processing unit 112 generates one vocalization state image (a profile image of a person) based on a vocalization state estimated from one piece of voice information. The display processing unit 113 performs animation display by displaying multiple vocalization state images generated from multiple pieces of voice information in chronological order.
[0058] For example, suppose that the tongue position has been identified as the utterance state based on the voice information, and as a result, the tongue position has moved downward compared to the normal state (for example, the state in FIG. 6). In this case, the display processing unit 113 performs animation display by displaying an utterance state image in which the tongue position has changed downward compared to FIG. 6, as shown in FIG. 7. For convenience of explanation, FIG. 7 illustrates an example of an image in which the jaw opening has not changed from FIG. 6, but the positions of both the tongue and the jaw may be variable.
[0059] In step S109, the display processing unit 113 performs a process of displaying the real-time feature point P, which is specified based on the utterance state estimated by the analysis processing unit 112, superimposed on the animation display. That is, the point P in FIG. 7 is a point whose position on the image changes in real time according to the change in the utterance state of the user. In this way, it is possible to present to the user in an easy-to-understand manner whether the utterance state of the user is closer to the first sound or the second sound (or far from either). In particular, when an axis representing the tongue position is set as the coordinate space, it is possible to allow the user to understand in which direction and to what extent the tongue position has shifted when comparing the current utterance with the utterance of the ideal first sound. In the method of this embodiment, the matters to be improved in order to achieve ideal pronunciation are easy to understand, so that the user's learning can be appropriately supported. Furthermore, since the relationship with the second sound is also illustrated, it is possible to support the user from the viewpoint of, for example, how to change the tongue position to change the second sound familiar to the user to the first sound.
[0060] Furthermore, the display processing unit 113 may perform a process of changing the display mode when the similarity between the utterance state estimated by the analysis processing unit and the utterance state corresponding to the first feature point P1 is equal to or greater than a predetermined threshold, and when the similarity is less than the predetermined threshold. For example, in step S110, the analysis processing unit 112 may calculate the similarity based on the distance between the real-time feature point P and the first feature point P1. When the distance is equal to or less than the threshold (for example, when the real-time feature point P is included in the first feature point P1 in a circular region), the analysis processing unit 112 determines that the similarity is equal to or greater than the threshold. Alternatively, the analysis processing unit 112 may calculate the similarity between the user's utterance and the first sound based on information before plotting in the coordinate space, such as the voice information itself or feature quantities such as formants.
[0061] If the similarity is equal to or greater than the threshold, the display processing unit 113 performs display processing in a first manner (OK display) in step S111, and if the similarity is less than the threshold, performs display processing in a second manner different from the first manner (NG display) in step S112.
[0062] In the example of FIG. 7, the feature point P representing the user's vocal state is close to the first feature point P1 corresponding to "ae", and the similarity is determined to be equal to or greater than a threshold. Therefore, the display processing unit 113 performs processing to display the display mode of the first feature point P1 representing "ae" in a mode different from the state in which the similarity is equal to or less than a threshold (for example, the state of FIG. 6). This makes it possible to clearly present whether the user's vocal state is appropriate or not. Note that FIG. 7 shows an example in which the color of the feature point representing "ae" is changed, but the display mode is not limited to this, and various modes that can identify whether the user's vocal state is close to the first sound or not can be used. For example, the display processing unit 113 may change the display size of the first feature point P1, or may blink the first feature point P1. Alternatively, the display processing unit 113 may display text, images, objects, etc. representing feedback on the user's vocalization in a display area different from the first feature point P1. The text representing the feedback may be, for example, "OK", "NG", etc., or may be information presenting specific correction points when NG. Similarly, various modifications can be implemented when displaying images and objects.
[0063] Alternatively, the display processing unit 113 may perform processing to change the facial expression of the person displayed as an animation depending on whether the similarity is equal to or greater than a predetermined threshold or less than the predetermined threshold. For example, as described below with reference to Figures 8A and 8B, the person may be displayed with a positive facial expression (e.g., a smiling face) in an OK display in which the similarity is equal to or greater than the threshold, and may be displayed with a neutral or negative facial expression (e.g., a crying face) in an NG display in which the similarity is less than the threshold. In this way, the quality of the user's vocalization state can be presented in an easy-to-understand manner by displaying the person's profile.
[0064] After the processing of step S111 or S112, in step S113, the processing unit 110 determines whether or not to end the learning process of the sound selected by the user in step S101. For example, the processing unit 110 may determine whether an end button not shown in Fig. 6 or 7 has been operated. If the learning process is not to be ended (step S113: No), the processing unit 110 returns to step S104 and continues the process. That is, the voice information acquisition unit 111 newly acquires voice information of the user, the analysis processing unit 112 performs analysis processing of the voice information, and the display processing unit 113 updates the image in real time based on the analysis result.
[0065] The first sound in this embodiment may be one sound. For example, the display processing unit 113 may display only one of ae, α, and Λ as an English sound similar to the Japanese "a". Even in such a case, since it is possible to perform learning while comparing the sound of the user's native language with the sound of the language to be studied, the user can be appropriately supported. However, the processing of this embodiment is not limited to this. For example, when there are a first sound and a third sound in the first language as sounds similar to the second sound of the second language, the display processing unit 113 may perform processing to superimpose the third feature point, the first feature point, and the second feature point, which are identified based on the vocalization state when the third sound is pronounced, on the animation display in a mutually distinguishable manner. The first sound is, for example, one of ae, α, and Λ, and the third sound is, for example, another one of ae, α, and Λ. In this way, as described above with reference to Figs. 6 and 7, it becomes possible to display a plurality of sounds in the first language together on one screen. When a third sound exists in addition to a first sound, the first language contains a plurality of sounds similar to the second sound that the user frequently uses. As a result, the first language may be difficult to distinguish between the first sound and the third sound, which may impede learning. In this respect, the method of the present embodiment can clearly present the difference between the first sound and the third sound on one screen, thereby improving learning efficiency.
[0066] 2.2 Flashcard learning As described above, when a second language has a first sound and a third sound (for example, English has sounds such as ae, α, and Λ that are similar to the Japanese "a"), it is important for a user who is a speaker of the second language to distinguish between the first sound and the third sound when learning the first language.
[0067] For example, the display processing unit 113 may perform a process of displaying an instruction screen for prompting the user to pronounce the first sound a predetermined number of times or more, and then perform a process of displaying an instruction screen for prompting the user to pronounce the third sound a predetermined number of times or more. This makes it possible to encourage the user to learn by taking into account the differences in sounds by repeatedly uttering similar sounds.
[0068] 8A and 8B are examples of screens displayed by the display processing unit 113, for example, on the display unit 240 of the terminal device 200. For example, in the first stage of learning, the display processing unit 113 may display an image for repeatedly uttering a word including either the first sound or the third sound while superimposing feature points corresponding to the first sound, the third sound, and the second sound on an animation display of a person's profile. In the example of FIG. 8A and FIG. 8B, the first sound is the English α, the third sound is the English ae, and the second sound is the Japanese "o". In FIG. 8A and FIG. 8B, the first feature point P1, the second feature point P2, and the third feature point P3 corresponding to these are illustrated. Since the English α is similar to the Japanese "o", such a display is also useful for supporting English learning by Japanese speakers. However, the combination of the first sound to the third sound is not limited to this, and as described above with reference to FIG. 7, other combinations of sounds may be used, such as the combination of "a", "ae", "α", and "Λ".
[0069] 8A and 8B also display text instructing the user to say the word "hot" three times. Because the word "hot" contains the "alpha" sound, learning using the screens shown in Figures 8A and 8B allows the user to learn how to pronounce the "alpha" sound, which is easily confused with other sounds.
[0070] As described above, in the screens of Fig. 8A and Fig. 8B, when the similarity between the user's vocalization state and any of the sounds (first sound to third sound) is equal to or greater than a threshold, the display mode is changed from that when the similarity is less than the threshold. For example, Fig. 8A and Fig. 8B may include an area RE1 in which phonetic symbols of multiple sounds that are easily confused are displayed side by side. When the analysis processing unit 112 determines that the user's pronunciation is similar to any of the sounds, the display processing unit 113 changes the display mode of the object representing the sound.
[0071] Furthermore, similar to the examples of Figures 6 and 7, in area RE2 which displays an animation corresponding to the user's profile, the image of the person is updated according to the state of the tongue and jaw, and when a feature point in real time is close to the first feature point P1 or the third feature point P3, the display mode of the corresponding feature point is changed.
[0072] 8B, when the user's vocalization state is judged to be close to the vocalization state of the learning target sound (here, the sound "α") (hereinafter also referred to as "OK judgment"), the display processing unit 113 may change the facial expression of the person to a positive facial expression. This makes it possible to present in an easy-to-understand manner whether the user is able to properly pronounce the learning target sound.
[0073] The accuracy judgment as to whether the user's vocalization state is close to the vocalization state of the learning target sound is executed as a process different from the process using the frequency analysis described above (for example, the judgment using the positional relationship between the first feature point and the feature point in real time). The accuracy judgment here is performed by outputting two types of results, OK and NG. However, the accuracy judgment is not limited to this, and may be a process of outputting numerical data (for example, 0 to 100 points) that increases as the user's vocalization state approaches the target. For example, the analysis processing unit 112 may perform a process of comparing the user's voice information with voice information representing the ideal pronunciation of "hot" and calculating the similarity as a score. Methods of outputting the similarity between two pieces of voice information as a score are widely known, and these conventional methods may be applied in this embodiment. For example, the comparison process of the voice information may be performed in a cloud server different from the server system 100. In this case, the analysis processing unit 112 requests the cloud server to perform a process of comparing the user's voice information with the voice information representing the ideal pronunciation of "hot" and obtains the processing result. However, the judgment of correctness in this embodiment may be performed using the result of frequency analysis. For example, the analysis processing unit 112 may obtain OK / NG or a score according to the distance between a first feature point corresponding to the target sound and a real-time feature point representing the user's vocalization state.
[0074] The processing unit 110 may perform a process of shifting to learning a different word when the screens of FIG. 8A and FIG. 8B are judged to be OK a given number of times (for example, three times). For example, the display processing unit 113 shifts to learning the word "hut" when the user can properly pronounce "hot" three times in a row. Since the word "hut" includes the sound "Λ", the user can learn the pronunciation of "Λ" that is easily confused with other sounds by learning using a screen that allows "hut" to be repeatedly pronounced. Similarly, the processing unit 110 shifts to learning the word "hat" when the user successfully pronounces "hut" a given number of times in a row. Since the word "hat" includes the sound "ae", the user can learn the pronunciation of "ae" that is easily confused with other sounds by learning using a screen that allows "hat" to be repeatedly pronounced. The order of learning is not limited to the above example, and "hat", "hut", and "hot" may be learned in a different order.
[0075] When the first stage of learning, which is learning for each word, is completed, the processing unit 110 may perform a process of learning to quickly pronounce multiple words including similar sounds as a second stage of learning. For example, the display processing unit 113 may perform a process of displaying an instruction screen that causes the user to continuously pronounce multiple words including a word including a first sound and a word including a third sound. In this way, it is possible to cause the user to continuously pronounce a group of words including sounds that should be pronounced distinctly, so that the user can acquire pronunciation appropriate for each of the first sound and the third sound.
[0076] For example, the processing unit 110 creates a word list by selecting and arranging words a predetermined number of times (e.g., 10 times) from three words, "hat," "hut," and "hot," while allowing the same word to be selected more than once. The display processing unit 113 sequentially displays the words included in the word list, and also displays a message prompting the user to pronounce the displayed words. In this way, the user can quickly utter multiple words that contain confusable sounds, making it possible for the user to learn the proper pronunciation of the confusable sounds.
[0077] 9 is a flowchart explaining the processing in this case. First, in step S201, the operation unit 250 of the terminal device 200 accepts a selection input of the English sounds to be studied. For example, the operation unit 250 may accept a selection input of three sounds, "ae", "α", and "Λ". For example, the display unit 240 may perform a process of displaying a list of sounds that are likely to be confused by speakers of the second language, and the operation unit 250 may accept a selection operation of any one of the sounds in the list.
[0078] In step S202, the processing unit 110 creates a word list to be spoken based on the user's selection input. As described above, the word list here is a list in which a group of words including any of a plurality of sounds that are easily confused are arranged in a given order. Note that the order here may be fixed, or may be determined randomly for each learning (each time the process of FIG. 9 is performed).
[0079] In step S203, the display processing unit 113 displays the first word in the word list on the display unit 240 of the terminal device 200. Fig. 10A is an example of a screen displayed at the start of learning (for example, after step S202 and before S203). The screen shown in Fig. 10A includes text indicating the learning content such as "Let's pronounce the words displayed one after another!", a group of words to be pronounced (Hat, Hut, Hot), a start button, etc. For example, when the user performs a selection operation on the start button, the process of step S203 is executed.
[0080] Fig. 10B is an example of a screen displayed in step S203. For example, Fig. 10B shows a state in which the number of unlearned (unpronounced) words remaining in the word list is six, and "Hat" is displayed as the word in this case. The microphone of the terminal device 200 is operating in a recording state, and when a user viewing the screen of Fig. 10B pronounces "hat," voice information of the user is recorded in step S204. The voice information acquisition unit 111 of the server system 100 acquires the voice information via the network.
[0081] In step S205, the analysis processing unit 112 performs an analysis process of the voice information. For example, the analysis processing unit 112 acquires features such as formants, and inputs the features to the trained model to identify the user's vocal state, as in step S105. Furthermore, the analysis processing unit 112 determines the similarity between the user's vocal state and the vocal state of the sound that is the learning target. As described above, the similarity determination (correct / incorrect determination) may be performed as a process different from the process of identifying the vocal state. In the example of FIG. 10B, the analysis processing unit 112 determines OK if the similarity between the user's vocal state and the vocal state of "Hat" (or, in a narrow sense, the vocal state of "ae") is equal to or greater than a threshold, and determines NG if it is not.
[0082] In step S206, the display processing unit 113 performs a process of displaying the analysis results from the analysis processing unit 112. Here, for example, as in Figs. 6-8B, animation of the person's profile, display of feature points in real time, change of display mode of each feature point according to similarity, etc. may be performed. However, as described later with reference to Fig. 10D, the results of the analysis process may be displayed together after pronunciation of all words in the word list is completed, and the process of step S206 may be omitted. Also, in the example of Fig. 10B, animation display according to the state of the tongue or jaw is not performed for the person, so the analysis processing unit 112 may omit the process of identifying the vocalization state and perform only similarity judgment (correct / incorrect judgment).
[0083] In step S207, the display processing unit 113 determines whether all processing of the word list has been completed. The processing here refers to processing of displaying the target word and acquiring the user's speech in response to the display as voice information. If there are unprocessed words remaining (step S207: No), the display processing unit 113 returns to step S203 and continues processing.
[0084] For example, when performing the process of step S203 again, the display processing unit 113 performs a display process of the earliest word among the unprocessed words in the word list. Fig. 10C is an example of a screen displayed on the display unit 240 after Fig. 10B. For example, Fig. 10C shows a state in which the number of unlearned (unpronounced) words remaining in the word list is three, and "Hot" is displayed as the word at that time. The process in this case is the same as steps S204-S207 described above, and each process of obtaining audio information, analyzing, displaying (optional), and determining completion is performed.
[0085] If the processing of all words included in the word list has been completed (step S207: Yes), in step S208, the display processing unit 113 performs processing to display the learning result. Fig. 10D is an example of a screen displayed in step S208.
[0086] In the example of FIG. 10D, the display processing unit 113 displays the accuracy rates of three words, "hat," "hut," and "hot." The accuracy rate here is information determined from the result of the similarity judgment (correct / incorrect judgment). For example, the accuracy rate represents the ratio of the number of times each word is judged to be OK to the number of times each word is displayed. In this way, by displaying the accuracy rate for each word, it is possible to clearly present whether the user can distinguish between the pronunciations of similar sounds, whether there are any sounds that the user has difficulty pronouncing, and the like. The display processing unit 113 may also display the overall accuracy rate of multiple words. In this case, in the example of FIG. 10D, it is possible to present to the user how accurately the user can pronounce English sounds similar to the Japanese "a."
[0087] As shown in FIG. 10D, an object (button) for reviewing each word may be displayed in association with the accuracy rate of the word. For example, when an object corresponding to "hat" is selected, the display processing unit 113 may return to FIG. 8A and FIG. 8B and perform a display process for repeatedly uttering "hat" a predetermined number of times. In this way, the user's pronunciation ability can be appropriately developed by having the user learn by transitioning between the first stage of learning (learning for each word) and the second stage (learning by combining a plurality of words containing similar sounds). In particular, in the second stage, the user learns by using so-called flash cards by pronouncing the displayed words at high speed, which makes it easier to memorize the relationship between words and pronunciation.
[0088] In addition, in the method of the present embodiment, in addition to the speaking learning in which the user utters the words, listening learning may be performed. For example, as shown in FIG. 8A and FIG. 8B, the display processing unit 113 may display an object for outputting the user's own utterance result. In the example of FIG. 8A, the display processing unit 113 displays the reproduction objects OB1 and OB2. When the reproduction object OB1 is selected, the processing unit 110 (or the processing unit 210) outputs audio information (for example, audio information of a speaker of the first language or an educator of the first language) representing the desired utterance of "hot". When the reproduction object OB2 is selected, the processing unit 110 outputs audio information when the user utters "hot". As described above, the user's audio information is recorded for analysis processing, and by outputting the audio information, it is possible to have the user check his / her own utterance.
[0089] In this way, in addition to visual information using animated displays of a person's profile and feature points, auditory information can be used to compare the ideal pronunciation with the user's pronunciation, thereby improving learning efficiency.
[0090] In addition, when the display processing unit 113 determines that the user's proficiency level is low, it may perform processing to display text information as the phonetic symbol of the first sound, and when the user's proficiency level is determined to be high, it may perform processing to display image information representing the shape of the mouth when pronouncing the first sound as the phonetic symbol of the first sound.
[0091] Fig. 11A is an example of a screen where character information is displayed as a phonetic symbol, and Fig. 11B is an example of a screen where image information is displayed as a phonetic symbol. For example, the analysis processing unit 112 determines that the proficiency level is low when the user's accuracy rate is less than a predetermined threshold in Fig. 8A, Fig. 8B, or Fig. 10A to Fig. 10D. In this case, the user is considered to have a low level of understanding of the vocalization state (oral cavity state) when pronouncing the target sound. Therefore, the display processing unit 113 displays the phonetic symbol of the character information that is widely used and easy to understand even for users with a low level of proficiency (Fig. 11A).
[0092] On the other hand, if the user's accuracy rate is above a predetermined threshold, the analysis processing unit 112 determines that the user has a high level of proficiency. In this case, since the user is considered to have a high level of understanding of the vocalization state (oral cavity state), the display processing unit 113 displays phonetic symbols of image information visually expressing the vocalization state (FIG. 11B). In this way, it becomes possible to make the user more aware of the movements of the tongue and jaw, and thus it becomes possible to encourage further improvement in the level of proficiency.
[0093] 2.3 Examples In addition, for a user who is a speaker of the second language, the pronunciation of the first language may be unfamiliar to everyday use and may be difficult to learn. Therefore, in this embodiment, the first language may be learned using everyday pronunciation in the second language as a guide.
[0094] For example, the display processing unit 113 may determine a daily action to be uttered using a pronunciation state close to the pronunciation of the first sound, and perform processing to display the daily action in association with a word including the first sound. In this way, the pronunciation of the first language is associated with the daily action, so that the user's understanding can be improved.
[0095] 12 is a flowchart for explaining the processing of the information processing system 10 in this case. First, in step S301, the operation unit 250 of the terminal device 200 accepts a selection input of an English sound to be studied. For example, the operation unit 250 may accept a selection input of "h", or may accept an input of a word containing "h" (e.g., hat, hut, hot, etc.).
[0096] In step S302, the processing unit 110 determines a daily action having a similar vocalization state based on the selection input of the user. For example, the storage unit 120 may store table data that associates sounds of the first language with daily actions. The processing unit 110 determines the corresponding daily action based on the sound of the first language (first sound) selected in step S301 and the table data. For example, the processing unit 110 determines the action of "blowing on a window glass to fog it up before polishing it" as the daily action corresponding to the sound "h".
[0097] In step S303, the display processing unit 113 performs processing to display a word including the first sound in association with a daily action. Fig. 13 is an example of a screen displaying the word "hat" including the first sound "h" in association with the daily action of blowing on a window glass to fog it up before polishing it. The screen shown in Fig. 13 displays an image representing the daily action, making it possible to suggest to the user a vocalization state suitable for pronouncing the first sound.
[0098] In step S304, the microphone of the terminal device 200 records the user's voice information. The voice information acquisition unit 111 of the server system 100 acquires the voice information via the network.
[0099] In step S305, the analysis processing unit 112 performs a correct / incorrect judgment based on the voice information. For example, as described above, the analysis processing unit 112 performs an OK / NG judgment or calculates a score by a method other than frequency analysis. Also, as described above, these processes may be executed by a server other than the server system 100. Also, in step S305, the analysis processing unit 112 may perform a process including frequency analysis, as in step S105. For example, the analysis processing unit 112 acquires features such as formants, and specifies the user's vocal state by inputting the features to a trained model. Furthermore, the analysis processing unit 112 judges the similarity between the user's vocal state and the vocal state of the sound that is the learning target. In the example of FIG. 13, the analysis processing unit 112 judges the similarity as OK when the similarity between the user's vocal state and the vocal state of "hot" (in a narrow sense, the vocal state of "h" among them) is equal to or greater than a threshold, and judges the similarity as NG when the similarity is less than the threshold.
[0100] In step S306, the display processing unit 113 performs processing to display the processing result in the analysis processing unit 112. For example, the display processing unit 113 displays a judgment result regarding OK / NG, which is a result of the correct / incorrect judgment, and a score indicating the similarity. Furthermore, if frequency analysis is performed in step S305, the display processing unit 114 may display an animation of a person's profile, display feature points in real time, change the display mode according to the similarity, and the like, as in Figs. 6 to 8B.
[0101] In step S307, the display processing unit 113 determines whether to end the learning process for the sound selected in step S301. For example, when the operation unit 250 of the terminal device 200 receives an end operation by the user, the display processing unit 113 determines to end the learning process. If the learning process is not to be ended (step S307: No), the display processing unit 113 returns to step S303 and continues the process. If the learning process is to be ended (step S307: Yes), the process shown in FIG. 12 ends.
[0102] Furthermore, the guide display by the display processing unit 113 is not limited to those using daily actions. For example, the display processing unit 113 may perform a process of determining similar spoken words, which are words in the second language that include a pronunciation state similar to the pronunciation of the first sound, and displaying the similar spoken words in association with words that include the first sound.
[0103] For example, the sound of "sh" is considered to be close to the pronunciation of an onomatopoeia used in Japanese as the sound of a steam locomotive running. Therefore, the display processing unit 113 can support the acquisition of the pronunciation of "sh" by displaying such onomatopoeia together with an English word containing "sh" as a similar vocalization word. For example, the display processing unit 114 may perform a process of displaying a word containing "sh" in association with an image of a steam locomotive and an onomatopoeia in Japanese that matches the image of the steam locomotive. Alternatively, the sound of "s" is considered to be close to the beginning part of "watermelon" in Japanese. Therefore, the display processing unit 113 can support the acquisition of the pronunciation of "s" by displaying "watermelon" as a similar vocalization word in association with an English word containing "s".
[0104] 14 is a flowchart for explaining the processing of the information processing system 10 in this case. First, in step S401, the operation unit 250 of the terminal device 200 accepts a selection input of an English sound to be learned. For example, the operation unit 250 may accept a selection input of "sh" or "s."
[0105] In step S402, the processing unit 110 determines a similar voiced word in the second language that has a similar vocalization state based on a selection input by the user. For example, the storage unit 120 may store table data in which sounds in the first language are associated with similar voiced words. The processing unit 110 determines the corresponding similar voiced word based on the sound of the first language (first sound) selected in step S401 and the table data. Note that the table data may be data in which daily actions or similar voiced words are associated with sounds in the first language. In this case, it is possible to share the process shown in step S302 of FIG. 12 with the process shown in step S402 of FIG. 14.
[0106] In step S403, the display processing unit 113 performs a process of displaying the word including the first sound in association with the similar spoken word. In step S404, the microphone of the terminal device 200 records the user's voice information. The voice information acquisition unit 111 of the server system 100 acquires the voice information via the network.
[0107] In step S405, the analysis processing unit 112 performs a correct / incorrect decision based on the voice information. In step S406, the display processing unit 113 performs a process of displaying the processing result in the analysis processing unit 112.
[0108] In step S407, the display processing unit 113 determines whether to end the learning process of the sound selected in step S401. If the learning process is not to be ended (step S407: No), the display processing unit 113 returns to step S403 and continues the process. If the learning process is to be ended (step S407: Yes), the process shown in FIG. 15 ends.
[0109] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present embodiment. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy can be replaced with that different term anywhere in the specification or drawings. In addition, all combinations of the present embodiment and modifications are also included in the scope of the present disclosure. In addition, the configurations and operations of the information processing system, server system, terminal device, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]
[0110] 10...information processing system, 100...server system, 110...processing unit, 111...audio information acquisition unit, 112...analysis processing unit, 113...display processing unit, 120...storage unit, 130...communication unit, 200...terminal device, 210...processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, OB1, OB2...playback object, P...real-time feature point, P1...first feature point, P2...second feature point, P3...third feature point, RE1, RE2...area
Claims
1. a speech information acquisition unit that acquires speech information of a user who is learning a first language; an analysis processing unit that estimates a speech state representing a state of the jaw and a state of the tongue of the user based on an analysis process of the voice information; a display processing unit that performs processing to display a plurality of utterance state images in animation based on the estimated chronological change in the utterance state; Including, The display processing unit is An information processing device that performs processing to superimpose and display a first feature point identified based on the vocalization state when pronouncing a first sound in the first language, and a second feature point identified based on the vocalization state when pronouncing a second sound similar to the first sound in a second language different from the first language, on an animation display of the multiple vocalization state images.
2. In claim 1, The display processing unit is An information processing device that performs processing to change the display mode when the similarity between the vocalization state estimated by the analysis processing unit and the vocalization state corresponding to the first feature point is greater than or equal to a predetermined threshold and when the similarity is less than the predetermined threshold.
3. In claim 2, the animation display being a screen showing a person's head including their jaw and tongue; The display processing unit is An information processing device that performs processing to change the facial expression of the person displayed as the animation depending on whether the similarity is equal to or greater than the predetermined threshold or less than the predetermined threshold.
4. In claim 1, The display processing unit is The information processing device performs processing for displaying, on the animation display, feature points identified based on the speaking state estimated by the analysis processing unit.
5. In any one of claims 1 to 4, The display processing unit is An information processing device that performs processing to display an instruction screen that prompts the user to successively speak a plurality of words including a word including the first sound and a word including the third sound when the first language has a first sound and a third sound that are similar to the second sound of the second language.
6. In any one of claims 1 to 4, The display processing unit is an information processing device that performs processing for displaying, when the first sound and a third sound exist in the first language as sounds similar to the second sound of the second language, a third feature point, the first feature point, and the second feature point, which are identified based on the vocalization state when pronouncing the third sound, superimposed on the animation display in a mutually distinguishable manner.
7. In claim 6, The display processing unit is an information processing device that performs display processing of an instruction screen for prompting the user to pronounce the first sound a predetermined number of times or more, and then performs display processing of an instruction screen for prompting the user to pronounce the third sound a predetermined number of times or more.
8. In any one of claims 1 to 4, The display processing unit is If it is determined that the user has a low level of proficiency, a process of displaying character information as a phonetic symbol of the first sound is performed; An information processing device that, when it is determined that the user has a high level of proficiency, performs processing to display image information representing the shape of the mouth when pronouncing the first sound as the phonetic symbol of the first sound.
9. In any one of claims 1 to 4, The display processing unit is An information processing device that performs a process of determining a daily activity to be uttered using the utterance state close to the utterance of the first sound, and displaying the daily activity in association with a word including the first sound.
10. In any one of claims 1 to 4, The display processing unit is An information processing device that performs a process of determining similar spoken words, which are words in the second language that include the speaking state that is close to the speaking state of the first sound, and displaying the similar spoken words in association with words that include the first sound.
11. Acquire speech information of a user who is learning a first language; estimating a speech state representing a jaw state and a tongue state of the user based on an analysis process of the voice information; performing a display process for displaying a plurality of utterance state images in animation based on the estimated chronological change in the utterance state; In the display process, performing a process of displaying a first feature point identified based on the utterance state when a first sound in the first language is pronounced, and a second feature point identified based on the utterance state when a second sound similar to the first sound is pronounced in a second language different from the first language, in a superimposed manner on an animation display of the plurality of utterance state images; Information processing methods.
Citation Information
Patent Citations
Foreign language self-study system
JP2001249679A
Language learning apparatus
JP2006162760A
System and program for evaluating pronunciation
JP2011076044A
Language pronunciation learning device with speech analysis function and system thereof
JP2012088675A
Pronunciation movement visualization device and pronunciation learning device
WO2013031677A1
