A digital therapeutic system and method based on cognitive behavioral therapy.

Abstract

A digital therapy system based on cognitive behavioral therapy according to several embodiments of the present invention comprises a user terminal and a digital therapy server that provides learning content based on cognitive behavioral therapy to the user terminal, wherein the digital therapy server includes an evaluation unit that evaluates the user's learning results based on user response information received from the user terminal, and the evaluation unit includes an effectiveness determination unit that evaluates the effectiveness of the response information and a learning evaluation unit that evaluates the user's learning results based on the response information.

Description

Technical Field

[0001] The present invention relates to a digital therapy system and method based on cognitive behavioral therapy. More specifically, when evaluating the learning results of a user with an autism spectrum disorder, the present invention relates to a digital therapy system and method based on cognitive behavioral therapy that can improve the reliability of learning result evaluation by first determining the validity of the response information of the user.

Background Art

[0002] Autism spectrum disorder is a mental disorder characterized by developmental defects in social interaction, restricted interests, repetitive behaviors, and sensory hypersensitivity. It appears from early in life and is a developmental disorder that persists throughout life. People with autism spectrum disorder show developmental delays in various aspects such as cognition, language, social-emotional, physical, and behavior during the developmental process in infancy. Developmental delays due to autism spectrum disorder affect various areas of life, such as school life, occupation, social relationships, daily life, and self-care, and most often require the help of people around them, such as family members, teachers, and activity assistants, due to difficulties in social interaction and safety issues. Therefore, detecting and treating autism spectrum disorder can have a significant impact on the long-term growth and development of children and can have a positive impact on the family and society as a whole.

[0003] However, the development of effective treatments for social interaction deficits in individuals with autism spectrum disorder (ASD) has been slow, and access to these treatments is currently limited due to high costs and limited facilities. While risperidone and aripiprazole have received FDA approval for treating hypersensitivity associated with ASD, there are no FDA-approved standard drugs that target core symptoms such as problems with social contextual perception and interaction (Research and Development Special Zone Promotion Foundation, 2021). According to the American Association of People with Disabilities, the average annual treatment cost for children with ASD in the United States in 2017 was $17,081. Furthermore, if indirect costs (e.g., lost working hours for parents, household labor costs, etc.) are included in addition to the direct treatment costs incurred by parents of children with ASD, the average annual treatment cost for a household is estimated to be over $36,000. In South Korea, Applied Behavior Analysis (ABA) has been introduced for over a decade as a way to enhance social interaction in individuals with autism spectrum disorder, and treatment programs have gradually become more widespread. However, in South Korea, the number of certified therapists is small, and they tend to be concentrated in the Seoul metropolitan area. Furthermore, the cost of each session ranges from 60,000 to nearly 100,000 won, resulting in monthly medical expenses exceeding 3 million won, leading to low utilization rates. In addition, using services such as rehabilitation exercise therapy, speech therapy, art therapy, and music therapy may incur additional costs, and 49.7% of parents of children with developmental disabilities responded that treatment costs are "very burdensome" or "somewhat burdensome" (Childcare Policy Research Institute, 2019).

[0004] On the other hand, digital therapeutic devices help children learn in an engaging way, utilizing games, videos, and animations to deliver content in a way that is easy for children to understand and find interesting. These methods can help children enjoy learning, build confidence, and actively participate. Digital therapeutic devices are highly portable, applicable using smartphones and smartpads, allowing for location-independent and repeated learning, while also being relatively inexpensive compared to in-person therapy. They are particularly useful for resolving everyday problems through real-time feedback.

[0005] In particular, individuals with autism spectrum disorder may exhibit qualitative problems in social interaction, difficulties with language and communication, and mood and emotional instability. They may also use certain language in a stereotyped or repetitive manner.

[0006] Regarding social interaction, individuals with autism spectrum disorder (ASD) lack cognitive empathy, making it difficult for them to read and understand the emotions of others, and simultaneously, they struggle to recognize how to express their own emotions cognitively. In ASD, language therapy is essential to improve communication abilities. Cognitive empathy in individuals with autism spectrum disorder is acquired through learning and is not automatically acquired, unlike in neurotypical individuals.

[0007] Furthermore, in terms of communication, individuals with autism spectrum disorder (ASD) often experience difficulties communicating due to their restrictive, inappropriate, and distinctive language use. Their speech may also exhibit monotonous pitch, unique intonation and rhythm, or extremely high or low voice tones. When speaking, individuals with ASD may exhibit abnormalities such as inaccurate pronunciation, mismatched sentences, or the use of a mixture of informal and formal language. They may find it more difficult to converse with others. For example, they might engage in inappropriate behavior, such as laughing and mimicking the way someone is expressing genuine anger.

[0008] Therefore, in order to solve the aforementioned problems, training programs are needed that cover topics such as the recognition and distinction of emotions via digital therapeutic devices, specific methods for use in social situations, problem-solving techniques that arise in interpersonal relationships, and social communication. In particular, in language therapy via digital therapeutic devices, it is inevitable that the outcome of the treatment will be evaluated based on the responses of individuals with autism spectrum disorder. Consequently, there is a growing need for a digital therapeutic system and method for individuals with autism spectrum disorder that uses digital therapeutic devices to first assess the effectiveness of the responses of individuals with autism spectrum disorder, and only if those responses are deemed effective can the learning outcomes be evaluated based on those responses. [Overview of the project] [Problems that the invention aims to solve]

[0009] The objective of the present invention, in order to solve the above-mentioned problems, is to provide a digital therapy system and method based on cognitive behavioral therapy that can receive responses from individuals with autism spectrum disorder who have received learning and treatment using digital therapy devices, and can determine the effectiveness of the received responses.

[0010] Another objective of the present invention to solve the above-mentioned problems is to provide a digital therapy system and method based on cognitive behavioral therapy that can enhance the daily living-centered executive functions of persons with autism spectrum disorder using digital therapeutic devices. [Means for solving the problem]

[0011] To achieve the above-mentioned objectives, a digital therapy system based on cognitive behavioral therapy according to one embodiment of the present invention comprises a user terminal and a digital therapy server that provides learning content based on cognitive behavioral therapy to the user terminal, wherein the digital therapy server has an evaluation unit that evaluates the user's learning results based on user response information received from the user terminal, and the evaluation unit may include an effectiveness determination unit that evaluates the effectiveness of the response information and a learning evaluation unit that evaluates the user's learning results based on the response information. In this case, the response information may be choice information or utterance information received from the user terminal.

[0012] According to some embodiments, the effectiveness determination unit can evaluate the effectiveness of the response information based on a dialogue response speed evaluation criterion that evaluates whether the user responded at an appropriate time after the other party finished speaking.

[0013] According to some embodiments, when the effectiveness determination unit evaluates the effectiveness of the response information based on the dialogue response speed evaluation criteria, if the response information includes echo words, it can determine that the user responded at the time the echo words ended.

[0014] According to some embodiments, when evaluating the effectiveness of the response information, the effectiveness determination unit can analyze the response information to extract emotions from the user's utterance, and compare the extracted emotions with the emotions included in the model response information to evaluate the effectiveness of the response information.

[0015] According to some embodiments, the learning evaluation unit can evaluate the similarity between the user's response information and a model response when evaluating the user's learning.

[0016] According to some embodiments, the learning evaluation unit can evaluate the user's learning by replacing the mispronounced word with a correctly pronounced word if the response information contains a mispronounced word.

[0017] According to some embodiments, the digital therapy server further comprises a reading unit that receives reading information from the user, the reading unit transmits first data containing text information to be spoken by the user to the user terminal, and the user terminal can generate second data containing reading information generated by reading the text information contained in the received first data and transmit it to the reading unit.

[0018] According to some embodiments, the reading unit analyzes the reading information to extract the user's mispronounced words, maps the mispronounced words to the correct words and stores them, and the learning evaluation unit can evaluate the user's learning by replacing the mispronounced words included in the response information with the correct words based on the mispronounced words extracted from the reading information.

[0019] According to some embodiments, the second data may further include user metadata, which may include information on the user's age, gender, and degree of disability.

[0020] According to some embodiments, the learning evaluation unit can evaluate the similarity between the user's response information and the reading information when evaluating the user's learning.

[0021] The digital therapy method based on cognitive behavioral therapy according to some embodiments of the present invention includes a step of evaluating response information of a user received from a user terminal, and the step of evaluating the response information includes a step of evaluating the validity of the response information and a step of evaluating the learning of the user based on the response information. The step of evaluating the validity can evaluate the validity of the response information based on a dialogue response speed evaluation criterion that evaluates whether the user responds at an appropriate time after the other person's words end.

Advantages of the Invention

[0022] According to the digital therapy system and its method based on cognitive behavioral therapy according to some embodiments of the present invention, it is possible to determine the validity of a response before evaluating the result of learning based on the response information received from an autistic spectrum disorder person, thereby improving the reliability of the learning result evaluation.

[0023] In addition, by providing educational content based on cognitive behavioral therapy to autistic spectrum disorder persons, there is an effect that results such as cognitive improvement, behavior modification, and emotion regulation can be obtained through repeated learning.

[0024] Furthermore, by providing educational content to autistic spectrum disorder persons through digital therapy devices, there is an effect that it is highly portable, not restricted by location, and repeated learning can be performed many times, while being relatively inexpensive compared to face-to-face therapy.

[0025] In addition, by providing educational content to autistic spectrum disorder persons by utilizing games, videos, animations, etc., there is an effect that autistic spectrum disorder persons can more easily understand and be more interested in the educational content.

Brief Description of the Drawings

[0026] [Figure 1] It is a block diagram showing each component of the digital therapy system based on cognitive behavioral therapy according to the present invention. [Figure 2] It is a block diagram showing each component of the digital therapy server according to the present invention. [Figure 3] It is a diagram showing an example of a theme presented by the presentation unit to the user terminal. [Figure 4] It is a flowchart of a digital therapy method based on cognitive behavioral therapy according to the present invention.

Mode for Carrying Out the Invention

[0027] Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. When assigning reference numerals to the components of each drawing, it should be noted that for the same components, as far as possible, the same numerals are used even if they are shown on different drawings. And, when explaining the embodiments of the present invention, if it is determined that a specific explanation of a related known configuration or function hinders the understanding of the embodiments of the present invention, the detailed explanation thereof will be omitted.

[0028] Also, when explaining the components of the embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc. can be used. Such terms are for distinguishing the component from other components, and the essence, order, etc. of the corresponding component are not limited by such terms. In this specification, the singular form includes the plural form unless otherwise specifically mentioned in the text. "Including" and / or "including" used in the specification do not exclude the existence or addition of one or more other components in addition to the mentioned components.

[0029] Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

[0030] FIG. 1 is a block diagram showing each component of a digital therapy system 1000 based on cognitive behavioral therapy according to the present invention.

[0031] A digital therapy system 1000 (hereinafter referred to as "system") based on cognitive behavioral therapy according to the present invention includes a user terminal 10 and a digital therapy server 100.

[0032] The user terminal 10 (digital therapeutic device, hereinafter referred to as "user terminal 10") may be a terminal owned by a user with autism spectrum disorder. The user can send and receive information with the digital therapeutic server 100 via the user terminal 10. An application for treating the user's autism spectrum disorder may be installed on the user terminal 10. The application is a user program for specifically executing the system and method according to the present invention. Such an application may be downloaded and installed from the digital therapeutic server 100 or another app store server. The application installed on the user terminal 10 can communicate with the digital therapeutic server 100 to receive learning content, display learning screens within the learning content on the display device of the user terminal 10, speak audio information, and record the user's speech.

[0033] The digital therapy server 100 can provide learning content based on cognitive behavioral therapy to the user terminal 10. In this specification, the learning content includes basic content provided by the learning unit 130, which will be described later, technical enhancement content provided by the application unit 140, and virtual field content provided by the virtual field unit 150.

[0034] The digital treatment server 100 provides at least one communication protocol for communicating with the user terminal 10 via a communication network, and can provide the user terminal 10 with an interface such as an application programming interface (API) for application integration for service to the application.

[0035] In the following, with reference to Figure 2, each component of the digital treatment server 100 according to the present invention will be described in detail.

[0036] Figure 2 is a block diagram showing the various components of the digital treatment server 100 according to the present invention.

[0037] Referring to Figure 2, the digital treatment server 100 according to the present invention includes a collection unit 110, a presentation unit 120, a learning unit 130, an application unit 140, a virtual field unit 150, and an evaluation unit 160.

[0038] On the other hand, adolescents and adults with autism spectrum disorder, even without intellectual disability, often experience severe difficulties in adapting to daily life due to problems with executive function. In particular, they frequently struggle to complete goal activities due to problems with key executive functions such as cognitive flexibility, self-control, working memory, and planning and initiation.

[0039] Studies have been published demonstrating the effectiveness of identifying individual weaknesses in executive functions and providing customized programs to enhance them in individuals with autism spectrum disorder (Chung, 2016; Oswald, 2017; Yang, 2022; de Vries, 2015; Kenworthy, 2013; Baker-Ericzen, 2017). These studies have shown improvements in working memory, behavioral control, and planning and organizing abilities, as well as a reduction in anxiety.

[0040] However, the above studies are typically conducted under controlled experimental conditions and only report limited, indirect effects on the process of increasing adaptability and social interaction in daily life. Therefore, there are limitations in evaluating whether they actually enhance social behavior based on real-world experiences.

[0041] Accordingly, the system according to the present invention (1) a collection unit 110 grasps the user's current knowledge (background information) via the user terminal 10; (2) a presentation unit 120 presents at least one subject to the user terminal 10 and receives a selection of at least one subject from the user terminal 10; (3) when the user terminal 10 selects at least one subject, the learning unit 130 provides the user terminal 10 with basic content that can be practiced repeatedly along with a number of detailed tasks corresponding to that subject, enabling the user to acquire skills; (4) an application unit 140 provides the user terminal 10 with skill enhancement content so that the skills acquired through the basic content can be applied; (5) a virtual field unit 150 provides virtual field content that reconstructs the content of tasks or skill enhancement content that can be derived from the basic content, thereby promoting repeated practice in a constantly changing field; and (6) an evaluation unit 160 evaluates the user's learning results so that individuals with autism spectrum disorder can be educated and treated in terms of adaptability in daily life and social interaction.

[0042] In particular, the evaluation unit 160 according to the present invention can evaluate the user's learning results based on the user's response information received from the user terminal 10.

[0043] On the other hand, the user response information that the evaluation unit 160 evaluates may include choice information and / or utterance information received from the user terminal 10, and a brief explanation of this is as follows.

[0044] The learning unit 130, application unit 140, and virtual field unit 150 according to the present invention can provide basic content, technology enhancement content, and virtual field content to the user terminal 10, respectively.

[0045] As described above, when the user terminal 10 selects at least one of several themes, the learning unit 130 can provide the user terminal 10 with basic content that can be practiced repeatedly, along with several detailed tasks corresponding to that theme. In this case, the basic content consists of videos, each containing a scene corresponding to a task related to the theme selected by the user. The basic content can consist of, for example, an animation in which multiple speakers converse with each other. The videos of the basic content may also include not only the conversation between multiple speakers, but also separate components that explain each theme and task and help the user understand the current situation.

[0046] As described above, the application unit 140 can provide the user terminal 10 with technology enhancement content that enables the application of the technology acquired through the basic content.

[0047] In this case, the enhanced technical content is structured around a method in which multiple speakers converse with each other, based on the conversation content included in the basic content, so that the user can concentrate solely on conversation training. This includes a first enhanced technical content in which one of the speakers plays the role of the user, and a second enhanced technical content in which only the utterances of the speaker playing the user's role in the first enhanced technical content are muted. The user can repeatedly practice conversation training through the first enhanced technical content. After repeatedly practicing conversation training through the first enhanced technical content, the user can learn linguistic conversational expressions by completing the omitted conversations when the muted speaker utters through the second enhanced technical content.

[0048] In this case, when the conversation of the person acting on behalf of the user is omitted in the second technical enhancement content, the application unit 140 can display multiple options on the user terminal 10 that include the appropriate utterance of that person (i.e., a model response corresponding to the utterance of the speaker that has been repeatedly learned in the first technical enhancement content). The user can select the option containing the appropriate utterance from among the displayed options.

[0049] Alternatively, when the conversation of a person acting on behalf of the user is omitted in the second enhanced content, the application unit 140 can provide an environment in which the user can actually speak the appropriate utterances of that person. When the second enhanced content is played back, if the utterances of a muted speaker are provided, the user can speak a model response corresponding to the utterances of the speaker that have been repeatedly learned in the first enhanced content, and the user terminal 10 can record the user's utterances. The user terminal 10 can transmit the user's selected choice information and / or the recorded user utterance information to the application unit 140, and the application unit 140 can transmit the received choice information and / or recorded utterance information to the evaluation unit 160. The evaluation unit 160 can evaluate the user's learning results based on the received choice information and / or utterance information.

[0050] Furthermore, as mentioned above, the virtual field unit 150 can provide the user terminal 10 with virtual field content that reconstructs the content of tasks or skill enhancement content that may be derived from the subject and / or basic content. The virtual field content is intended to teach and educate individuals with autism spectrum disorder, who may have difficulty adapting and applying flexibly to various changes in the future, how to appropriately cope with diverse changing situations. The virtual field unit 150 provides options so that the user can select appropriate utterances in a virtual field related to the subject presented by the presentation unit 120, and provides an environment in which the user can actually utter, enabling the user to apply the skills they have acquired in the virtual field.

[0051] The virtual field content, like the aforementioned enhanced technology content, may include a first virtual field content containing a speaker performing the user's role and the complete content of that speaker's utterance, as well as a second virtual field content in which the utterance of the speaker performing the user's role is muted.

[0052] Through the first virtual scenario content, users can repeatedly practice conversations that can occur in virtual scenarios simulating various changing situations. After repeatedly practicing conversations through the first virtual scenario content, users can learn linguistic conversational expressions by completing the omitted conversations of muted speakers through the second virtual scenario content.

[0053] At this time, when the conversation of the person acting on behalf of the user is omitted in the second virtual field content, the virtual field unit 150 can display multiple options on the user terminal 10 that include the appropriate utterance of that person (i.e., a model response corresponding to the utterance of the speaker that has been repeatedly learned in the first virtual field content). The user can select the option containing the appropriate utterance from among the displayed options.

[0054] Alternatively, the virtual field unit 150 can provide an environment in which the user can actually speak appropriate utterances of a person acting on behalf of the user when the conversation of that person is omitted in the second virtual field content. When the second virtual field content is played back and the utterances of a muted speaker are provided, the user speaks a model response corresponding to the utterances of the speaker that they have repeatedly learned in the first virtual field content, and the user terminal 10 records the user's utterances. The user terminal 10 transmits the choice information selected by the user and / or the recorded user utterance information to the virtual field unit 150, and the virtual field unit 150 can transmit the received choice information and / or recorded utterance information to the evaluation unit 160. The evaluation unit 160 can evaluate the user's learning results based on the received choice information and / or utterance information.

[0055] The evaluation unit 160 receives user response information from the application unit 140 and the virtual field unit 150, respectively, and can evaluate the user's learning results learned from the technology enhancement content and the virtual field content, respectively, based on the received response information. On the other hand, the digital treatment system 1000 according to the present invention can generate text information by converting choice information presented via text information or speech information via a known method into text, and can evaluate the user's response information by comparing the generated text information with a model response that also includes text information.

[0056] The evaluation unit 160 includes an effectiveness determination unit 161 and a learning evaluation unit 163. The effectiveness determination unit 161 can evaluate the effectiveness of user response information received from the user terminal 10. The learning evaluation unit 163 can evaluate the user's learning results based on that response information.

[0057] When the evaluation unit 160 displays the user's learning results, as evaluated by the learning evaluation unit 163, on the user terminal 10, parent terminal, or administrator terminal, it can also display the effectiveness of the response information evaluated by the effectiveness determination unit 161. In other words, when the evaluation unit 160 displays the user's learning results, as evaluated by the learning evaluation unit 163, it can also display whether the evaluation is effective or not, and to what extent it is effective or not, thereby making it easier to manage the user's learning.

[0058] Alternatively, if the effectiveness determination unit 161 determines that the response information is ineffective, the evaluation unit 160 can improve its processing speed by not evaluating the user's learning results and only displaying information indicating that it is ineffective.

[0059] The effectiveness determination unit 161 can evaluate the effectiveness of response information through evaluation criteria such as dialogue response speed evaluation criteria, speech speed evaluation criteria, and dialogue repetition evaluation criteria.

[0060] The dialogue response speed evaluation criteria refer to evaluation criteria for assessing whether the user responded at an appropriate time after the other person finished speaking. This includes not only cases where a long time has passed since the other person finished speaking, but also cases where a user responds before or immediately after the other person finished speaking, in which case the utterance may be a completely different conversation unrelated to the other person's words.

[0061] Therefore, when presenting options to the user terminal 10, if the user selects a location where the options are presented before the options are presented, or if the user selects an option after a set time has elapsed after the options have been presented, the option information containing the user's option selection information can be evaluated as invalid.

[0062] This can also be applied when the user terminal 10 provides an environment in which the user can actually speak, records the user's speech to generate speech information, and evaluates its effectiveness based on that speech information. In other words, even if the user's speech begins before or immediately after the other party finishes speaking in each piece of content, or begins after the other party has finished speaking and a set amount of time has elapsed, the user's speech information can be evaluated as ineffective.

[0063] In this case, when the effectiveness determination unit 161 evaluates the effectiveness of the utterance information based on the dialogue response speed evaluation criteria, it can also apply the dialogue repetition evaluation criteria. The dialogue repetition evaluation criteria are evaluation criteria for evaluating whether the user repeats the same words or phrases when uttering.

[0064] For example, if a user's utterance includes an echo, the effectiveness determination unit 161 determines that the user responded at the point when the echo ended.

[0065] Echoic words refer to the act of simply mimicking or imitating what someone says without responding directly. While echoic words are normal in infancy (approximately 1-3 years of age), their persistence beyond infancy is a typical symptom of developmental disorders (autism spectrum disorder, language disorders, intellectual disabilities) and hinders smooth communication. In severe cases of developmental disorders, echoic words may persist into adulthood. Echoic words can be divided into immediate echoic words, which appear immediately after hearing what someone says, and delayed echoic words, which appear after a period of time has passed. In this case, echoic words can include types such as "A is A" or "A is A+a".

[0066] The validity determination unit 161, for example, if the other party's question is "What are you doing today?", and the user's utterance in response to that question includes a part that directly imitates or imitates the other party's question, such as "What are you doing today? What are you doing today? What are you doing today? I'm eating strawberries today," then excludes that part and determines that only the subsequent utterance is the user's utterance. Therefore, if the user's utterance contains echolalia, the validity determination unit 161 can determine that the user's response information is invalid if the time interval between the end of the other party's words and the end of the echolalia is longer than the set time interval.

[0067] The effectiveness determination unit 161 can evaluate the effectiveness of spoken information according to the speech rate evaluation criteria. The speech rate evaluation criteria are for evaluating the speaking rate during a conversation. Speaking rate is an important skill for effectively conveying one's words to the other person. If the speaking rate is excessively fast or slow, it can be difficult for the other person to grasp the speaker's words or pay attention to what the speaker is saying, making it difficult to maintain the conversation. Furthermore, the speaker may utter completely unrelated words that are not in response to what the other person said. Therefore, if the user's speaking rate is faster or slower than the set rate, the effectiveness determination unit 161 can determine that the user's response information is ineffective.

[0068] The effectiveness determination unit 161 can recognize the user's emotions from the user's speech information. The effectiveness determination unit 161 compares the emotions recognized at the time of the user's speech with the emotions included in the model response information of the speaker performing the user's role within each piece of content. If the emotions are the same, the effectiveness determination unit 161 can determine that the user's response information is effective; if they are different, it can determine that the response information is ineffective. On the other hand, for this purpose, the speech information of the speaker performing the user's role within the content, i.e., the model response information, includes not only the audio and text corresponding to the model response, but also information about the emotions corresponding to that model response. In this case, the information about emotions can be, for example, one of the following: neutral, joy, sadness, anger, disgust, surprise, or fear. The effectiveness determination unit 161 can also recognize the user's emotions at the time of their speech as one of the aforementioned neutral, joy, sadness, anger, disgust, surprise, or fear by analyzing the user's speech information.

[0069] The effectiveness determination unit 161 analyzes the speech signals within the user's speech information to recognize the user's emotional state. Speech-based emotion recognition technology is one of the speech processing applications that recognizes the voice of a speaker and can recognize the speaker's emotions or physical state from the recognized voice. Methods for recognizing a user's emotions from speech generally use feature extraction methods, and the accuracy of the emotions recognized from the speech varies depending on the extracted features. Currently, commonly used feature extraction methods can recognize emotional states from speech by utilizing speech pitch, speech rate, intensity, MFCC (Mel-Frequency Cepstral Coefficients), and LPCC (Linear Prediction Cepstral Coefficients).

[0070] In one embodiment, the effectiveness determination unit can divide an input data set, which corresponds to an audio signal, into frames according to pre-specified emotional characteristics. For example, the effectiveness determination unit can divide an input data set, which corresponds to an audio signal from various different users that has been collected in advance, into frames corresponding to the user's emotion.

[0071] For example, the effectiveness determination unit can extract features corresponding to pre-specified effective sounds from an input dataset corresponding to an audio signal based on MFCC (Mel Frequency Cepstral Coefficient).

[0072] MFCC is an algorithm devised by Davis and Mermelstein in the 1980s to extract effective features from real sounds as a two-dimensional array. Rather than analyzing the entire sound, it divides the sound into short time intervals, analyzes the spectrum of these intervals, and converts the features into numerical data. MFCC data is designed not to distinguish between pitch differences and is used in fields such as speech recognition, music genre analysis, and emotion recognition.

[0073] In other words, the effectiveness determination unit does not extract features from the entire input dataset, but rather divides it into fixed intervals, i.e., fixed frames, and extracts features frame by frame through spectral analysis. For example, since speech signals change continuously in the time domain, in order to extract features from changing sounds, it can be assumed that speech signals do not change much within a predetermined short time. That is, it can be interpreted that there is actually almost no change in the speech signal within the error range. Then, the effectiveness determination unit can calculate the power spectrum (i.e., frequency) for each frame. Here, the power spectrum calculated for each frame can be extracted as a feature, i.e., a feature vector. Once the neural frequency is calculated in this way, it is possible to know how much energy exists in each interval. The effectiveness determination unit divides the user's speech information and the model response information into intervals in the same ratio, compares the energy of each interval, and can recognize the user's emotion at the time of speech as one of the following: neutral, joy, sadness, anger, disgust, surprise, or fear.

[0074] The learning evaluation unit 163 can evaluate the user's learning results. The learning evaluation unit 163 can convert the speech signals contained in the user's speech information into text using a known method, and compare this with the text contained in the model response information through similarity evaluation and other means to determine how similar the user's speech is to the model response.

[0075] At this time, the learning evaluation unit 163 can evaluate the user's learning results according to the keyword evaluation criteria. The keyword evaluation criteria are used to identify and evaluate whether keywords are included in the user's utterances.

[0076] On the other hand, individuals with autism spectrum disorder may have pronunciation errors (i.e., mispronunciations) when speaking compared to neurotypical individuals. In other words, it is not easy for individuals with autism spectrum disorder to pronounce words accurately (i.e., correctly) as neurotypical individuals. Pronunciation errors are a phenomenon that often occurs in young children whose pronunciation is still developing. In infancy, children cannot pronounce difficult phonemes, so they omit them or change them to simpler phonemes. If pronunciation errors persist into adulthood, it falls under articulation-phonological disorder, a type of language disorder. Articulation-phonological disorder is the most common type of language disorder and often persists into adulthood.

[0077] The learning evaluation unit 163 analyzes the speech signals within the user's utterance information and, if the speech signals contain mispronounced words, can evaluate the user's learning by replacing those mispronounced words with correctly pronounced words. For example, in the case of mispronunciation, examples of substitution, distortion, omission (deletion), addition, contraction, inversion, and repetition can be used.

[0078] JPEG2026509155000002.jpg30150

[0079] JPEG2026509155000003.jpg10170

[0080] JPEG2026509155000004.jpg20170

[0081] JPEG2026509155000005.jpg10170

[0082] JPEG2026509155000006.jpg15170

[0083] JPEG2026509155000007.jpg10170

[0084] JPEG2026509155000008.jpg24170

[0085] On the other hand, the substitution of correct pronunciation words by the learning evaluation unit 163 is performed through a database that has already been learned, and the range of mispronunciation varies widely among users, making it difficult to substitute all mispronunciations with correct pronunciations. Therefore, the digital therapy server 100 can receive and analyze pronunciation information from each user and learn the user's mispronunciation words and the corresponding correct pronunciation words through the audio signals contained in that pronunciation information. Thus, the digital therapy system 1000 according to the present invention can provide customized substitutions of correct pronunciation words for each user.

[0086] More specifically, the digital treatment server 100 according to the present invention may further include a reading unit 111 that receives reading information from a user, analyzes and learns from it, and stores the learning results in a database.

[0087] The reading unit 111 can transmit first data containing text information that the user should respond to to the user terminal 10. The user terminal 10 can display the text information contained in the received first data on its display, and then record the user's speech reading it aloud to generate second data containing the reading information. The user terminal 10 transmits the generated second data to the reading unit 111, which can analyze, learn, and store the received second data.

[0088] In this case, the text information that the user should respond to can mean the technical enhancement content provided to the user terminal 10 by the application unit 140 and the virtual field unit 150, respectively, and the model responses of the speaker who acts on behalf of the user within the virtual field content, which have been transcribed into text.

[0089] In other words, before learning the technology through basic content, technical enhancement content, and virtual field content, users already receive and read aloud the text corresponding to the model responses. At this time, they can read aloud the text corresponding to all of the model responses, or they can receive and read aloud only some of the text corresponding to the model responses.

[0090] The reading unit 111 can analyze second data containing reading information corresponding to only a portion of the model response text to extract the user's mispronunciations, learn the correct pronunciation corresponding to those mispronunciations, and concatenate the extracted and learned mispronunciations and the corresponding correct pronunciations to store them in a database.

[0091] On the other hand, in one embodiment, the reading unit 111 can transmit the reading information received from the user terminal 10 and the mispronounced words extracted from that reading information to a parent terminal (not shown). The parent terminal can use the audio signals contained in the received reading information to determine exactly which correct word the mispronounced word is and transmit this information to the reading unit 111. Therefore, the mispronounced word and the correct word that accurately corresponds to it can be clearly linked. Requesting the correct word corresponding to the mispronounced word from the parent terminal and linking them in this way would be desirable when the system cannot clearly determine the mispronounced word.

[0092] As described above, the learning evaluation unit 163 can evaluate the user's learning by replacing mispronounced words with correctly pronounced words if the response information contains mispronounced words. In this case, if the learning evaluation unit 163 finds that the mispronounced words in the response information match mispronounced words extracted from the pronunciation information received from the user terminal 10, it replaces the mispronounced words in the response information with correctly pronounced words linked to the mispronounced words extracted from the pronunciation information to evaluate the user's learning. Therefore, user-customized evaluation becomes possible.

[0093] The second set of data may include user metadata in addition to pronunciation information. In this case, the metadata may include the user's age, gender, and degree of disability. Therefore, the pronunciation information can also be used when replacing mispronounced words with correctly pronounced words for other users with similar age groups, genders, and degrees of disability.

[0094] On the other hand, pronunciation information is used not only when replacing mispronounced words with correctly pronounced words, but also in evaluating the user's learning. Pronunciation information is generated when the user reads aloud the text that the user should speak in the second technical enhancement content and the second virtual field content. After the user acquires skills through the basic content, technical enhancement content and virtual field content, the user speaks appropriate responses through the second technical enhancement content and the second virtual field content, and the learning evaluation unit 163 evaluates those speeches.

[0095] As mentioned above, the learning evaluation unit 163 converts the speech signals contained in the user's speech information into text using a known method, compares this with the text contained in the model response information through similarity evaluation, and determines how similar the user's speech is to the model response.

[0096] At this time, the learning evaluation unit 163 compares the user's utterances recorded through the second technology enhancement content and the second virtual field content with the already generated reading information through similarity evaluation and other means to determine how similar the user's utterances are to the already read reading information, thereby enabling a more accurate evaluation of the learning results. Alternatively, when evaluating parts of the user's speech that have not been read aloud, the evaluation can also be made based on reading information of other users with similar age groups, same gender, and similar disability levels stored in the database, by referring to metadata.

[0097] The following describes in more detail the other components of the digital treatment server 100 according to the present invention, excluding the reading unit 111 and the evaluation unit 160.

[0098] The data collection unit 110 can collect background information of users with autism spectrum disorder (ASD) via the user terminal 10 to understand the user's current knowledge. For example, the data collection unit 110 collects the user's age, degree of disability, gender, etc. The data collection unit 110 may also include the aforementioned reading unit 111.

[0099] Figure 3 shows an example of a topic that the presentation unit 120 presents to the user terminal 10.

[0100] The presentation unit 120 presents at least one topic to the user terminal 10. The topics presented by the presentation unit 120 can be configured to allow users to practice communication processes with people they may encounter in daily life and social situations, taking into account the shortcomings in social interaction and communication abilities of individuals with autism spectrum disorder. This corresponds to the objective of the present invention to provide a personalized smart healthcare service to individuals with autism spectrum disorder aged 10 to 18, for whom the development of social situational awareness and interaction with peers is more intensively needed than for infants and toddlers. By applying the topics presented by the presentation unit 120 to situations that may occur in daily life, the basic content provided by the learning unit 130, described later, can also include content related to communication processes with people they may encounter in daily life and social situations, thereby comprehensively enhancing planning and organization, cognitive flexibility, and social interaction in the process of adapting to daily life for individuals with autism spectrum disorder.

[0101] As mentioned above, the topics that the presentation unit 120 presents to the user terminal 10 can be topics that may occur in daily life, and may include, for example, as shown in Figure 3, (1) dressing according to the weather, (2) using the toilet, (3) finding directions (using public transport), (4) taking the bus (cash, card), (5) taking the subway (cash, card), (6) going to the bank (using an ATM), (7) doing offline shopping (cash, card), (8) doing online shopping (card), (9) going to the hospital, (10) going to the pharmacy, etc.

[0102] When the presentation unit 120 presents the aforementioned topics to the user terminal 10, the user terminal 10 can select one of the topics presented by the presentation unit 120. Alternatively, in one embodiment, a guardian of a person with autism spectrum disorder, for example, a parent, can also select one of the aforementioned topics. Although not shown in the drawings, if a guardian intervenes in the services provided by the present invention, that guardian can connect to the digital therapy server 100 via a guardian terminal (not shown).

[0103] When the user terminal 10 selects one of the aforementioned topics, the presentation unit 120 receives information regarding that selection and can transmit that information to the learning unit 130.

[0104] The learning unit 130 can provide the user terminal 10 with multiple detailed tasks corresponding to its subject.

[0105] In one embodiment, if the theme "dressing according to the weather" is selected, the learning unit 130 presents detailed tasks such as (1) checking the weather, (2) checking the temperature, (3) determining a general range of clothing, (4) selecting appropriate clothing from the clothing one owns, and (5) identifying inappropriate clothing.

[0106] Alternatively, in one embodiment, if the topic is "Using a bus (card)", the learning unit 130 will present detailed tasks such as (1) searching for the departure point and destination through the Naver Maps or Kakao Maps application, (2) checking the travel time, (3) preparing to depart 20 minutes earlier than the travel time, (4) checking the bus number to take, (5) checking the bus stop, (6) waiting for the bus at the bus stop, (7) having the transportation card in hand beforehand, (8) when the bus arrives, tapping the transportation card after boarding (e.g., dragging and tagging the transportation card), (9) sitting down, (10) checking the bus stop to get off, (11) listening carefully to the bus announcements and pressing the bell, and (12) tapping the card to get off.

[0107] Alternatively, in one embodiment, if the topic is "Using the subway (card)", the learning unit 130 will present detailed tasks such as (1) searching for the starting point and destination through the Naver Maps or Kakao Maps application, (2) checking the number of subway transfers, (3) checking the subway station, (4) checking the travel time, (5) preparing to depart 20 minutes earlier than the travel time, (6) checking the name of the station to get off at, (7) listening carefully to the subway announcements and preparing to get off, (8) checking the map to see which exit to take after getting off, (9) getting off after the announcement, looking at the exit signs to find the ticket gate and exit, (10) tapping the transportation card when exiting, and (11) looking at the exit signs to find the exit and exit.

[0108] Alternatively, in one embodiment, if the topic is "Going to the bank (using an ATM - withdrawing money)", the learning unit 130 will present detailed tasks such as (1) checking the bank mark on the check card (e.g., bank names are displayed for each bank mark), (2) searching for the bank name on Naver Maps or Kakao Maps, (3) finding the nearest bank and going there, (4) opening the door and entering the place where the ATM machine is located, (5) clicking the deposit / withdrawal button, (6) clicking "Card", (7) inserting the card into the flashing area in the direction of the arrow, (8) selecting the amount, (9) if there is no amount, selecting "Other" and then entering a number and pressing the confirmation button, (10) confirming the amount and pressing "Confirm", (11) entering the 4-digit card PIN, (12) printing the receipt, and (13) receiving the card and receipt.

[0109] Alternatively, as one embodiment, if the topic selected is "Online Shopping (Card - App Card Installation Guide)", the learning unit 130 will present detailed tasks such as (1) searching for "masks" on Naver, (2) pressing the shopping tab, (3) viewing and clicking on the desired color and size, (4) confirming the quantity to purchase after selecting the color and size, (5) pressing "Purchase", (6) searching for and entering the home address as the shipping address, (7) entering the mobile phone number, (8) entering a shipping message, (9) clicking "General Payment", (10) selecting "Credit Card", (11) clicking on the name of the company that issued the card, (12) pressing "Pay", (13) pressing "Pay with App Card", (14) pressing the QR code (registered trademark) to complete the payment, and (15) entering the easy payment PIN.

[0110] On the other hand, each task presented by the learning unit 130 can be composed of a single scene using basic content.

[0111] The learning unit 130 can provide the user terminal 10 with basic content corresponding to the subject selected by the user and multiple tasks related to that subject. As mentioned above, the basic content provided by the learning unit 130 to the user terminal 10 can consist of videos, each containing scenes corresponding to the tasks related to the subject selected by the user. The basic content can consist of animations, for example, in which multiple speakers converse with each other. The videos of the basic content can also include not only the conversations between multiple speakers, but also separate components that explain each subject and task and help the user understand the current situation.

[0112] On the other hand, the learning unit 130 can adjust the difficulty level of the basic content according to the user's background information collected by the collection unit 110. For example, the learning unit 130 adjusts the difficulty level of the basic content according to the user's age and degree of disability. In this case, the difficulty level of the basic content is adjusted by adjusting the number of speakers or the number of tasks.

[0113] The more speakers there are, the greater the volume of conversation and the higher the level of conversational ability required. For example, with two speakers, the conversation is exchanged alternately, and one only needs to focus their gaze on the other person. However, with four speakers, one must remember what the other three are saying while responding in accordance with the flow of the conversation, making the timing of the conversation and eye movement more complex.

[0114] Furthermore, the difficulty decreases as the number of tasks increases, and increases as the number of tasks decreases. For example, in the case of the task "Get on the subway and then find the exit at the station," if it can be broken down into more detailed tasks such as "Confirm the station to get off at > Listen carefully to the announcements > Check the exit on the map," the difficulty of each individual task decreases.

[0115] On the other hand, this difficulty level adjustment is not limited to the basic content of the learning unit 130, but can also be applied to the technical enhancement content of the application unit 140 and / or the virtual field content of the virtual field unit 150.

[0116] The application unit 140 can provide enhanced technology content that applies and utilizes technologies corresponding to the basic content to the user terminal 10. As mentioned above, the enhanced technology content can be structured around a method in which multiple speakers converse with each other, based on the conversation content included in the basic content, so that the user can concentrate solely on conversation training. In this case, one of the speakers can play the role of the user. When the speaker playing the role of the user speaks, the animation in the enhanced technology content is partially modified to display the speaker playing the role of the user so that the user can concentrate on that speech. Regarding the display of the speaker, the application unit 140 can assign feature points to the user's face region captured through the camera of the user terminal 10, and calculate and digitize the amount of change in the assigned feature points using an interpolation method, thereby realizing a real-time changing 2D or 3D model of the face and / or facial expression animation, which will be described later.

[0117] When the application unit 140 provides technical enhancement content to the user terminal 10, it can provide the content after muting the speech of the speaker who performed the user's role in the technical enhancement content. That is, the application unit 140 can provide the user terminal 10 with the general technical enhancement content described above (for example, "first technical enhancement content") a set number of times determined according to the user's age and degree of disability, and then provide the user terminal 10 with technical enhancement content (for example, "second technical enhancement content") in which only the speech of the speaker who performed the user's role has been muted. Through the second technical enhancement content, the application unit 140 presents video in which the conversation of the person acting on behalf of the user is omitted during video playback, and the user learns linguistic conversational expressions by completing the omitted conversation.

[0118] The application unit 140 can display multiple options on the user terminal 10 to select the appropriate utterance from the person acting on behalf of the user when the conversation of that person is omitted in the second enhanced technical content. The user can select the option containing the appropriate utterance from among the displayed options.

[0119] Alternatively, the application unit 140 can provide an environment in which the user can actually speak appropriate utterances of a person acting on behalf of the user when the conversation of that person is omitted in the second enhanced technical content. When the second enhanced technical content is played, the user terminal 10 can record the user's utterances when the utterances of a muted speaker are provided. The user terminal 10 transmits the selection information and / or the recorded user utterances selected by the user to the application unit 140, and the application unit 140 transmits the received selection information and / or the recorded user utterances to the evaluation unit 160. The evaluation unit 160 can evaluate the selection information and / or the recorded user utterances selected by the user as described above.

[0120] As described above, the virtual field unit 150 provides the user terminal 10 with virtual field content in which tasks that may be derived from the subject and / or basic content, as well as utterances from the technical enhancement content, have been reconstructed. The virtual field content provides options so that the user can select appropriate utterances in a virtual field related to the subject presented by the presentation unit 120, and provides an environment in which the user can actually utter, enabling the user to apply the skills they have learned in the virtual field. The virtual field content is intended to teach and educate people on how to appropriately cope with diverse changing situations, taking into account the characteristics of individuals with autism spectrum disorder who may have difficulty flexibly adapting and applying to various changes in the future.

[0121] For example, if the user selects the topic "Dressing according to the weather," the associated detailed tasks would be, as mentioned above, (1) checking the weather, (2) checking the temperature, (3) determining a general range of clothing, (4) selecting appropriate clothing from what they own, and (5) checking for inappropriate clothing. The basic content is a video containing scenes corresponding to each detailed task, and the technical enhancement content is composed by extracting only the spoken content from each basic content. In this case, the virtual field content may include content that can be derived from the aforementioned topic and tasks, such as guidance on how to borrow or buy an inexpensive umbrella when the weather forecast differs from the actual weather, or when it suddenly rains. Furthermore, the virtual field content can be a reconstruction of the spoken content of the technical enhancement content, which is composed by extracting only the spoken content from each basic content. This reconstruction could mean changing the speaker from female to male, changing the age range of the speakers, or changing the number of speakers. Therefore, the virtual field content can present a variety of changing situations based on the topic and detailed tasks, and educate and teach the correct spoken language accordingly.

[0122] The virtual field content, like the aforementioned enhanced technology content, may include a first virtual field content containing a speaker performing the user's role and the complete content of that speaker's utterance, as well as a second virtual field content in which the utterance of the speaker performing the user's role is muted.

[0123] In other words, the first virtual field content may include a video containing content derived from the basic content and / or a video in which the content of the technical enhancement content has been reconstructed. The first virtual field content is composed of multiple speakers conversing with each other, and one of the speakers may play the role of the user.

[0124] The second virtual site content can be provided to the user after the speech of the speaker who was performing the user's role in the first virtual site content has been muted.

[0125] The virtual field unit 150 can provide the first virtual field content to the user terminal 10 a set number of times according to the user's age and degree of disability, and then provide the user terminal 10 with second virtual field content in which only the utterances of the speaker who was performing the user's role have been muted. Through the second virtual field content, the virtual field unit 150 can present video in which the utterances of the person acting on behalf of the user have been omitted during video playback, enabling the user to learn linguistic conversational expressions by applying the utterances learned from the basic content and the technical enhancement content to complete the omitted conversation.

[0126] At this time, the virtual field unit 150 can receive selection information and / or speech information selected or recorded from the user terminal 10 through the virtual field content, and the virtual field unit 150 transmits the received selection information and / or recorded speech information to the evaluation unit 160. The evaluation unit 160 can evaluate the selection information and / or recorded user speech selected by the user according to the criteria described later.

[0127] The evaluation unit 160 evaluates the content learned by the user based on the correctness of the choice information received by the application unit 140 and / or the virtual field unit 150, respectively, and / or the recorded user utterances. After analyzing and evaluating the audio of the recorded user utterances, the evaluation unit 160 determines that the learning training was successful if the user's conversation content matches the pre-set evaluation criteria, and determines that the learning training was unsuccessful if the user's conversation content does not match the evaluation criteria. A detailed explanation of the evaluation unit 160 is as described above.

[0128] On the other hand, the services provided to the user terminal 10 by the collection unit 110, presentation unit 120, learning unit 130, application unit 140, virtual field unit 150, and evaluation unit 160 correspond to (1) assessment or psychological assessment, (2) reconceptualization, (3) skills acquisition, (4) skills consolidation and application training, (5) generalization and maintenance, and (6) post-treatment assessment follow-up, respectively, in cognitive behavioral therapy.

[0129] Cognitive behavioral therapy (CBT) is a psychotherapeutic technique that focuses on the relationship between human thought, behavior, and emotion. Known for its ability to improve cognition, modify behavior, and regulate emotions through repetitive learning, CBT is an evidence-based treatment used for a variety of mental illnesses, including ADHD, mood disorders, and addictions.

[0130] For example, the data collection unit 110 grasps the user's current knowledge in a manner corresponding to the "assessment or psychological evaluation" in cognitive behavioral therapy. The data collection unit 110 can grasp the user's current knowledge and collect information such as the degree of disability. In addition to this, the data collection unit 110 can also collect information such as the user's age and gender.

[0131] The presentation unit 120 can present learning topics in a manner corresponding to "reconceptualization" in cognitive behavioral therapy. Reconceptualization, on the other hand, refers to the process of re-evaluating and reconstructing how the user understands and processes the current problem situation. Therefore, the topics presented by the presentation unit 120 can be selected based on situations that the user may encounter in their daily life.

[0132] The learning unit 130, in accordance with the "skill acquisition" aspect of cognitive behavioral therapy, can present detailed tasks corresponding to the user's chosen topic and provide the user with basic content that includes the details of those detailed tasks. The basic content is designed to enable the user to acquire skills through repeated practice of the detailed tasks.

[0133] The application unit 140 can provide the user with skill enhancement content consisting only of speech content included in the basic content, in accordance with the "skill enhancement and application training" of cognitive behavioral therapy. In this case, the skill enhancement content provides the user with choices so that they can select appropriate speech, and provides an environment in which the user can actually speak, so that they can apply the acquired skills to real situations.

[0134] The virtual field unit 150 can provide users with virtual field content that allows them to repeatedly practice tasks derived from basic content and skill-enhancing content, as well as situations that may arise for each topic, in order to correspond to the "generalization and maintenance" of cognitive behavioral therapy. The virtual field content provides options so that users can select appropriate utterances in a virtual field related to the topic presented by the presentation unit 120, and provides an environment in which users can actually utter, enabling them to apply the skills they have acquired in a virtual field.

[0135] The evaluation unit 160 can be configured to evaluate and repeat the overall process, corresponding to the "post-treatment follow-up evaluation" in cognitive behavioral therapy. For example, the evaluation by the evaluation unit 160 can assess the user's education and learning outcomes based on the choices the user made and / or the content actually spoken during the application unit 140's technical enhancement content and / or the virtual field content of the virtual field unit 150.

[0136] Figure 4 is a flowchart of the digital treatment method based on cognitive behavioral therapy according to the present invention.

[0137] The digital treatment method based on cognitive behavioral therapy according to the present invention (hereinafter referred to as "the method"), as shown in Figure 4, may include the steps of: a collection unit 110 collecting background information of the user from a user terminal 10 (S100); a presentation unit 120 presenting at least one topic to the user terminal 10 (S200); a learning unit 130 providing the user terminal 10 with multiple tasks and basic content corresponding to the selected topic once the user terminal 10 has selected one of the topics (S300); an application unit 140 providing the user terminal 10 with technology enhancement content that allows the application of technology corresponding to the basic content (S400); a virtual field unit 150 providing the user terminal 10 with virtual field content derived from the basic content or reconstructed from the content of the technology enhancement content (S500); and an evaluation unit 160 evaluating the user's response information received from the user terminal 10 in order to evaluate the learning results of the user terminal 10 (S600).

[0138] In particular, step S600 may include a step in which the effectiveness determination unit 161 evaluates the effectiveness of the response information (S610) and a step in which the user's learning is evaluated based on the response information (S620). In this case, step S610 can evaluate the effectiveness of the response information based on a dialogue response speed evaluation criterion that evaluates whether the user responded at an appropriate time after the other party finished speaking.

[0139] The details of each step are as described above. For example, step S100 may include a step (not shown) in which the reading unit 111 transmits first data containing text information that the user should speak to the user terminal 10, and a step (not shown) in which the user terminal 10 generates second data containing spoken information generated by reading the text information aloud from the user and transmits it to the reading unit 111.

[0140] On the other hand, this method may further include steps (not shown) that provide a reward to the user after step S600.

[0141] In the step of providing rewards to users, once the evaluation unit 160 completes the evaluation, the first reward can be paid to the user terminal 10.

[0142] In this step, when the evaluation unit 160 determines that the user's conversation content matches the pre-set evaluation criteria and that the learning and training has been successful, the user terminal 10 can receive a second reward along with the first reward.

[0143] For this purpose, the system according to the present invention may further include a reward unit (not shown).

[0144] The reward unit can pay the first reward to the user terminal 10 upon receiving evaluation result information from the evaluation unit 160. At this time, the first reward can be given regardless of the result when the user completes learning a subject and the tasks corresponding to that subject. Therefore, the size of the first reward can be smaller than the other rewards described later.

[0145] On the other hand, the second reward is given when the user achieves a result that meets the evaluation criteria for a subject and the tasks corresponding to that subject, and is deemed to have successfully completed the learning training. Therefore, the second reward can be even greater than the first reward. The reward unit that provides the second reward to the user transmits the result to the collection unit 110, the presentation unit 120, and / or the learning unit 130. The collection unit 110 collects and stores this as background information for the user when the user has been offered the second reward. The presentation unit 120, when the user has been offered the second reward, can either display a learning training success indicator for the subject for a certain period of time, or it can choose not to present the subject to the user terminal 10 for a certain period of time. If the user selects the subject for which they have received the second reward again, the learning unit 130 can increase the difficulty level of the basic content corresponding to that subject. Therefore, as mentioned above, the user terminal 10 can be provided with basic content in which the number of speakers has been increased or the number of tasks has been decreased. On the other hand, for individuals with autism spectrum disorder, repeated learning of the same subject and task is important. Therefore, if the user repeatedly learns the basic content corresponding to the subject and task for which they received a second reward of increased difficulty, and the evaluation unit 160 evaluates this as successful learning again, the evaluation unit 160 provides the user with a third reward. Thus, the reward unit not only encourages participation in learning by providing the user with rewards corresponding to repeated learning, but can also motivate spontaneous repeated participation in the subject and task learned within a certain period.

[0146] On the other hand, the first to third rewards mentioned above can also be provided by the parental device. For example, each reward can be applied as mileage or points that can be exchanged for actual rewards. Alternatively, each reward can take the form of a symbolic icon that can be directly exchanged for actual rewards. For example, in this case, the symbolic icon could represent actual items that the user likes, such as chocolate, strawberries, or snacks, and the parental device can directly provide the reward to the user through the icon provided by the user. This allows for the provision of tangible rewards to the user and provides motivation for learning participation and repeated participation.

[0147] Furthermore, if, for example, each reward is mileage or points, the rewards unit can categorize the topics the user has learned and determine the topics with the highest similarity as similar topics.

[0148] The rewards section can separately calculate the reward amount based on factors such as the difficulty level and average time required for similar subjects. The reward amount calculated in this way is determined as the estimated reward amount and can be displayed on the user terminal 10 along with the subject when the user selects a subject.

[0149] This is to provide users with a larger reward when they repeatedly learn similar subjects. Furthermore, learning similar subjects has the advantage of improving learning efficiency.

[0150] On the other hand, as mentioned above, the application unit 140 and the virtual field unit 150 can provide the user terminal 10 with technology enhancement content and virtual field content, respectively. Meanwhile, the technology enhancement content and virtual field content include a speaker who performs the role of the user. In this case, the application unit 140 and / or the virtual field unit 150 can recognize the user's face or expression and provide the user terminal 10 with an animation in which the user's current face or expression is superimposed in real time onto the face of the speaker performing the user's role.

[0151] The user terminal 10 captures the user's face during learning via its camera and transmits the captured face to the digital treatment system 1000. The application unit 140 and / or virtual field unit 150 synthesize the received user's face and expressions onto the face of the speaker who performs the user's role in the animation.

[0152] This allows users to recognize the synthesized speaker, whose face and expressions are based on their own, as the speaker performing their role, and their gaze is naturally directed towards that speaker, improving learning efficiency. This not only makes learning more enjoyable for users, but also allows for real-time confirmation that users are focusing on the technology-enhanced content and / or virtual field content within the correct area.

[0153] The application unit 140 and the virtual field unit 150 assign feature points to the facial region captured through the camera of the user terminal 10, and calculate and digitize the amount of change in the assigned feature points using interpolation, thereby realizing animations of 2D or 3D model faces and / or expressions that change in real time.

[0154] The application unit 140 and the virtual field unit 150 each search for the user's face region from an image captured through the camera of the user terminal 10, assign feature points to the searched face region, and when a change in the feature points is input, convert that change into data. By applying the converted change to a 2D or 3D model in which control points corresponding to the feature points and their change amounts are set, the application unit 140 and the virtual field unit 150 can realize a real-time changing animation of the face and / or expression of the 2D or 3D model. When applying the converted change, the application unit 140 and the virtual field unit 150 each load a 2D or 3D model in which feature points and control points are set. When the converted change in the feature points is recognized, the application unit 140 controls the change in the feature points using the set control points and outputs it. If no change in the feature points is transmitted, the application unit 140 outputs the initialized face and / or expression of the loaded model. At this time, it is desirable that the converted change is a vector function value with respect to time.

[0155] The following provides a more detailed explanation of how the application unit 140 and the virtual field unit 150 simulate the user's current face or expression in real time by compositing it onto the face of the speaker who is performing the user's role.

[0156] The application unit 140 and the virtual field unit 150 can each capture a face and / or facial expression through a camera to search for a facial region, and assign feature points to the eyes, nose, mouth, eyebrows, and the overall contour of the face within the searched facial region. Then, the amount of change in feature points as the face and / or facial expression changes is converted into data, and by applying this to a 2D or 3D model in which feature points and control points corresponding to the amount of change in feature points are set, it is possible to realize a real-time changing facial and / or facial expression animation of the 2D or 3D model.

[0157] When applying the digitized change amounts of feature points, a 2D or 3D model with feature points and control points that change according to the change amounts of the feature points is loaded. The system checks whether the digitized change amounts of the feature points are received, and if so, the change amounts of the feature points can be controlled by the configured control points and output. If the change amounts of the feature points are not received, the initialized face and / or facial animation of the loaded model is output. Therefore, it is possible to recognize whether the user is viewing the technical enhancement content and virtual field content, or is viewing it outside a certain area.

[0158] On the other hand, the amount of change in feature points within the facial region captured by the camera can be calculated using interpolation.

[0159] Interpolation methods are techniques for generating facial expression animations by approximating intermediate stages from faces and / or expressions represented through multiple feature points. There are various known methods, such as shape interpolation, which approximates geometric structures, and keyframe interpolation, which approximates images over time.

[0160] The following describes in detail one embodiment of the process in which the interpolation method is applied.

[0161] First, still images are extracted from video input in real time via a camera at regular intervals. Feature points are assigned to the facial regions explored in the still images and temporarily stored in the first data memory. Then, after a certain period of time (e.g., 0.1 seconds to 1 second) has elapsed, still images are extracted from the incoming video, feature points are assigned to the explored facial regions, and temporarily stored in the second data memory. The feature points temporarily stored in the first and second data memories are then compared, and the amount of change is calculated and converted into data. At this time, the amount of change appears as an approximate vector function value with respect to time. Subsequently, a 2D or 3D model with the feature points set is loaded, and facial expression animation is realized by applying the above-calculated amount of change in the feature points.

[0162] Furthermore, the effectiveness determination unit 161 can recognize the user's facial expression through the amount of change in feature points within the facial region captured by the camera, and can analyze the recognized user's facial expression to extract the emotions the user felt when speaking. Therefore, the effectiveness determination unit 161 can extract the emotions the user felt when speaking more accurately through the user's facial expression along with the analysis of the response information mentioned above.

[0163] The user terminal 10 and guardian terminal according to the present invention are terminals such as desktop PCs, notebook PCs, tablet PCs, and smartphones equipped with input means such as keyboards, mice, touchpads, and touchscreens, and a display screen. However, they are not limited to these, and include any configuration that can connect to a digital treatment server 100 via a communication network and is capable of digital information processing, including the installation of an application program that can input search information and selected information and display the searched result information. The user terminal 10 according to the present invention is a component that connects to the digital treatment server 100 via a communication network to send and receive information, and may include at least one of the following: a smartphone, a tablet personal computer, a mobile phone, a video phone, a desktop personal computer, a laptop personal computer, a netbook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a wearable device (e.g., smart glasses, a head-mounted device (HMD), a kiosk, or a smart watch).

[0164] The user terminal 10 and the digital treatment server 100 according to the present invention can communicate with each other via their respective communication units and communication networks. The communication network refers to a connected structure that enables information exchange between each node, such as the terminal and the server. Examples of such communication networks include, but are not limited to, the 3GPP (registered trademark) (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, 5G network, WiMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Wi-Fi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network. The communication units of the user terminal 10 and the digital treatment server 100 may include electronic components provided for the communication network so that they can perform wired and wireless data communication via the aforementioned communication network.

[0165] In this specification, "unit" includes units implemented by hardware, units implemented by software, and units implemented using both. Furthermore, one unit may be implemented using two or more pieces of hardware, and two or more units may be implemented by one piece of hardware.

[0166] The scope of protection of the present invention is not limited to the descriptions and expressions of the embodiments explicitly described above. Furthermore, it should be reiterated that the scope of protection of the present invention is not limited by obvious modifications or substitutions in the art to which the present invention pertains.

Claims

1. User terminal and A digital therapy server that provides learning content based on cognitive behavioral therapy to the user terminal, Equipped with, The digital treatment server has an evaluation unit that evaluates the user's learning results based on response information received from the user terminal. The response information includes choice information or utterance information, The evaluation unit described above, A validity determination unit that evaluates the validity of the response information, A learning evaluation unit that evaluates the learning results based on the response information, A digital treatment system, including...

2. A digital treatment system according to claim 1, The aforementioned effectiveness determination unit evaluates the effectiveness of the response information based on a dialogue response speed evaluation criterion that evaluates whether the user responded at an appropriate time after the other party finished speaking, in this digital therapeutic system.

3. A digital treatment system according to claim 2, The aforementioned effectiveness determination unit evaluates the effectiveness of the response information based on the dialogue response speed evaluation criteria, and if the response information includes an echo, it determines that the user responded at the time the echo ended, in this digital therapeutic system.

4. A digital treatment system according to claim 1, The aforementioned effectiveness determination unit, when evaluating the effectiveness of the response information, analyzes the response information to extract emotions from the user's utterance, compares the extracted emotions with the emotions included in the model response information, and evaluates the effectiveness of the response information in this digital therapeutic system.

5. A digital treatment system according to claim 1, The learning evaluation unit evaluates the similarity between the user's response information and a model response when evaluating the user's learning, in this digital therapeutic system.

6. A digital treatment system according to claim 5, The learning evaluation unit is a digital therapy system that evaluates the user's learning by replacing mispronounced words with correctly pronounced words if the response information contains mispronounced words.

7. A digital treatment system according to claim 6, The digital treatment server further comprises a reading unit that receives reading information from the user, The reading unit transmits first data containing text information that the user should speak to the user terminal. A digital treatment system in which the user terminal generates second data including spoken information generated by reading aloud the text information contained in the received first data, and transmits it to the reading unit.

8. A digital treatment system according to claim 7, The reading unit analyzes the reading information to extract the user's mispronounced words, maps the correctly pronounced words to the mispronounced words, and stores them. The learning evaluation unit is a digital therapy system that evaluates the user's learning by replacing mispronounced words in the response information with correctly pronounced words, based on mispronounced words extracted from the pronunciation information.

9. A digital treatment system according to claim 8, The second data further includes user metadata, The aforementioned metadata includes information on the user's age, gender, and degree of disability, in a digital therapeutic system.

10. A digital treatment system according to claim 8, The learning evaluation unit evaluates the similarity between the user's response information and the reading information when evaluating the user's learning, in this digital therapeutic system.

Images