Language rehabilitation training method, system and device based on artificial intelligence and medium
Through an artificial intelligence-based method, using multimodal data and voice and video information to dynamically adjust the content and intensity of language rehabilitation training, the problem of solidification of training content in traditional systems is solved, the efficiency and effect of language rehabilitation training is improved, and the Personalized language rehabilitation training.
Patent Information
- Application Number
- CN202410844792.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-08
AI Technical Summary
The language rehabilitation training in traditional computer systems has solidified training content, and it is impossible to dynamically adjust the training intensity and difficulty according to the patient's real-time psychological state and training feedback, and it is impossible to personalize the training content, resulting in the training effect dependent on the initial evaluation and cannot meet personalized needs.
Using an artificial intelligence-based method, by obtaining initial multimodal data, generating initial training corpus and schemes, performing language rehabilitation pre-training, extracting language barrier information, adjusting training content and intensity, combining voice and video information to identify psychological states, and dynamically adjusting Training intensity to achieve personalized adjustment.
It improves the efficiency and effectiveness of language rehabilitation training, can dynamically adjust the training content and intensity according to the specific situation of the patient, solves the problem of solidification of training content, and realizes personalized language rehabilitation training.
Smart Images

Figure CN118919014A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of language rehabilitation training, and in particular to a language rehabilitation training method, system, device and medium based on artificial intelligence. Background Art
[0002] Artificial language rehabilitation training requires one-on-one consultation with doctors or professionals. Patients need to go to medical institutions in person, and doctors or professionals are scarce. Therefore, there are drawbacks such as high cost and inconvenience. At the same time, the training effect of language rehabilitation training depends on the experience of doctors or professionals.
[0003] Compared with manual methods, traditional computer system rehabilitation training has solved the problem of scarcity of doctors or professionals to a certain extent. Patients with mild symptoms can use software systems for training at home. Language rehabilitation training for stroke patients requires personalized design of training plans and content. At the same time, the patient's emotions and psychological state must be regularly evaluated, and attention must be paid to the patient's psychological stress, anxiety and other negative emotions. Traditional computer system rehabilitation training has fixed training content. Training plans and content can only be selected based on initial evaluations. Training content and training methods cannot be adjusted dynamically in real time based on training results and feedback; intensity and difficulty cannot be adjusted based on new conditions of patients (such as psychological state and emotional changes, physical condition, and the quality of training feedback). Summary of the invention
[0004] The present application aims to propose a language rehabilitation training method, system, device and medium based on artificial intelligence, which can personalize the training content and improve the efficiency and effect of language rehabilitation training.
[0005] In a first aspect, an embodiment of the present application provides a language rehabilitation training method based on artificial intelligence, the method comprising:
[0006] Acquire initial multimodal data;
[0007] Generate initial training corpus and initial training scheme according to the initial multimodal data and historical training scheme;
[0008] Using the initial training corpus and the initial training program to perform language rehabilitation pre-training on the patient to obtain a language rehabilitation training result;
[0009] Extracting language barrier information from the language rehabilitation training results, and adjusting the initial training corpus and the initial training program according to the language barrier information to obtain target training corpus and the target training program;
[0010] Using the target training corpus and the target training program to conduct daily language rehabilitation training for the patient, and obtaining voice information and video information during the daily language rehabilitation training;
[0011] Determine the correct result of the voice information, and feed back the correct result to the terminal device;
[0012] The patient's mental state is identified based on the voice information and the video information to obtain a mental state identification result, and the intensity of daily language rehabilitation training is adjusted based on the mental state identification result.
[0013] Compared with the prior art, the first aspect of the present application has the following beneficial effects:
[0014] This method obtains initial multimodal data, and the use of multimodal data can more comprehensively examine the effect of the patient's language rehabilitation training. According to the initial multimodal data and the historical training plan, the initial training corpus and the initial training plan are generated, and then the initial training corpus and the initial training plan are used to perform language rehabilitation pre-training on the patient to obtain the language rehabilitation training results, and then the information of the language disorder is extracted from the language rehabilitation training results, and the initial training corpus and the initial training plan are adjusted according to the information of the language disorder to obtain the target training corpus and the target training plan. Through multiple trainings, the training corpus and the training plan that are more in line with the patient's situation can be obtained, which solves the problem of solidified training content in rehabilitation training using traditional computer systems. The target training corpus and the target training plan are used to perform daily language rehabilitation training on the patient, and the voice information and video information in the daily language rehabilitation training process are obtained, the correct results of the voice information are judged, and the correct results are fed back to the terminal device, so that the patient can know his training results and maintain a good mentality. Based on voice information and video information, the patient's mental state is identified to obtain the mental state recognition results, and the intensity of daily language rehabilitation training is adjusted according to the mental state recognition results. By adjusting the intensity of daily language rehabilitation training through the recognition results, personalized adjustment of training content is achieved, thereby improving the efficiency and effectiveness of language rehabilitation training.
[0015] In some implementations, generating an initial training corpus and an initial training scheme based on the initial multimodal data and the historical training scheme includes:
[0016] The initial multimodal data and the historical training scheme are input into a generative artificial intelligence system so that the generative artificial intelligence system learns the historical training scheme and generates an initial training corpus and an initial training scheme based on the initial multimodal data.
[0017] In some embodiments, the using the initial training corpus and the initial training program to perform language rehabilitation pre-training on the patient to obtain a language rehabilitation training result includes:
[0018] Inputting the initial training corpus and the initial training scheme into an artificial intelligence system, so that the artificial intelligence system outputs the initial training corpus corresponding to the initial training scheme to a terminal;
[0019] The initial training corpus is displayed and played through the terminal, so that the patient can perform language rehabilitation pre-training according to the displayed and played initial training corpus.
[0020] In some embodiments, the language rehabilitation training results include the entire speech information recorded during the language rehabilitation training process, and the information about the language disorder is extracted from the language rehabilitation training results, and the initial training corpus and the initial training program are adjusted according to the information about the language disorder to obtain the target training corpus and the target training program, including:
[0021] Using a preset speech-to-text model to recognize the entire speech information as first text information;
[0022] Performing a text comparison between the first text information and the initial training corpus, extracting erroneous training corpus as information of language barriers, and extracting correct training corpus;
[0023] The information of the language barrier and the correct training corpus are input into a generative artificial intelligence system, so that the generative artificial intelligence system performs adversarial learning on the information of the language barrier and the correct training corpus to generate target training corpus and a target training plan.
[0024] In some implementations, the determining a correct result of the voice information includes:
[0025] Using a preset speech-to-text model to recognize the speech information as second text information;
[0026] The second text information is compared with the target training corpus to obtain a correct result corresponding to the voice information.
[0027] In some implementations, identifying the patient's mental state based on the voice information and the video information to obtain a mental state recognition result includes:
[0028] Acquire a facial image in the video information;
[0029] The facial image and the voice information are input into a trained psychological state recognition model to obtain a psychological state recognition result, wherein the trained psychological state recognition model includes multiple feature extraction modules, feature fusion modules and state recognition modules, and the multiple feature extraction modules, the feature fusion modules and the state recognition modules are trained by using multiple groups of facial images and voice information for training to obtain a trained psychological state recognition model.
[0030] In some embodiments, inputting the facial image and the voice information into a trained mental state recognition model to obtain a mental state recognition result includes:
[0031] Using a first feature extraction module to extract features from the facial image to obtain facial expression features, and using a second feature extraction module to extract features from the voice information to obtain voice information features;
[0032] Fusing the facial expression feature and the voice information feature to obtain a fused feature;
[0033] The fusion feature is input into a state recognition module to perform psychological state recognition to obtain a psychological state recognition result.
[0034] In a second aspect, the embodiment of the present application further provides a language rehabilitation training system based on artificial intelligence, the system comprising:
[0035] A data acquisition unit, used for acquiring initial multimodal data;
[0036] A data generating unit, configured to generate an initial training corpus and an initial training scheme according to the initial multimodal data and the historical training scheme;
[0037] A pre-training unit, used to perform language rehabilitation pre-training on the patient using the initial training corpus and the initial training program to obtain a language rehabilitation training result;
[0038] A data acquisition unit, configured to extract information about language disorders from the language rehabilitation training results, and adjust the initial training corpus and the initial training program according to the information about the language disorders to obtain target training corpus and the target training program;
[0039] A rehabilitation training unit, used to perform daily language rehabilitation training on the patient using the target training corpus and the target training program, and to obtain voice information and video information during the daily language rehabilitation training;
[0040] A result determination unit, used for determining a correct result of the voice information, and feeding back the correct result to the terminal device;
[0041] The training intensity adjustment unit is used to identify the patient's psychological state according to the voice information and the video information, obtain a psychological state recognition result, and adjust the intensity of daily language rehabilitation training according to the psychological state recognition result.
[0042] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute an artificial intelligence-based language rehabilitation training method as described above.
[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned artificial intelligence-based language rehabilitation training method.
[0044] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0046] Figure 1 It is a flowchart of an embodiment of the language rehabilitation training method based on artificial intelligence provided by the present application;
[0047] Figure 2 is a structural schematic diagram of an embodiment of an artificial intelligence-based language rehabilitation training system provided by the present application;
[0048] Figure 3 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION
[0049] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.
[0050] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0051] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0052] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.
[0053] 1. Artificial-based language rehabilitation training for stroke patients.
[0054] Rehabilitation training for language disorders after stroke can adopt methods such as language perception training, oral muscle movement training, articulation organ movement training, listening comprehension training, and expression ability training. It includes the following types of training:
[0055] 1. Language perception training.
[0056] By letting patients listen to the speech of normal people, analyzing their pronunciation characteristics and rules, and improving their understanding of speech, this measure helps restore the patient's ability to recognize and understand sound signals lost due to brain damage, thereby improving language communication.
[0057] 2. Oral muscle movement training.
[0058] Targeted exercises are performed on the oral muscles involved in swallowing, breathing and vocalization, such as puffing cheeks, blowing balloons, etc. This is to enhance the strength and coordination of these muscles and lay the foundation for the restoration of independent eating and speaking.
[0059] 3. Articulatory organ movement training.
[0060] It includes functional activity exercises for the lips, tongue, soft palate and other structures, such as blowing candles, marbles and other game-based training. The goal is to re-establish the cooperative relationship between the various articulatory organs and restore the normal pronunciation pattern.
[0061] 4. Listening comprehension training.
[0062] Using various scenarios to simulate real conversations, patients can be strengthened to understand spoken information. Since stroke may cause hearing loss or distraction, this training can help overcome these problems and improve the speed of responding to complex language instructions.
[0063] 5. Expressive ability training.
[0064] Gradually transition from single words to short sentences, and encourage patients to try to use the new vocabulary to express themselves orally. This measure focuses on stimulating patients' creative thinking and promoting their active participation in communication and interaction.
[0065] 2. Language rehabilitation training software system for stroke patients based on computer system.
[0066] The language rehabilitation training software system for stroke patients integrates a variety of language rehabilitation training knowledge bases that are commonly used at home and abroad in neurology, psychology, rehabilitation medicine, speech-language pathology, and human development, and carries out classified programming. It uses computer multimedia interactive technology to evaluate data and form a complete evaluation report to assist medical personnel in conducting special training for patients based on the evaluation results.
[0067] This method will input and maintain the training question types and training question library, and has a voice recognition function that can automatically determine whether the patient's pronunciation is correct, and guide the patient to continuously improve training based on programmed judgment.
[0068] By analyzing the above prior art, it is found that the prior art has the following shortcomings:
[0069] 1. Artificial language rehabilitation training requires one-on-one consultation with doctors or professionals. Patients need to go to medical institutions in person, and doctors or professionals are scarce. Therefore, there are defects such as high cost and inconvenience. At the same time, the training effect of language rehabilitation training depends on the experience of doctors or professionals.
[0070] 2. Compared with manual methods, traditional computer system rehabilitation training has solved the problem of scarcity of doctors or professionals to a certain extent. Patients with mild symptoms can use software systems for training at home. However, language rehabilitation training for stroke patients requires personalized design of training plans and content. At the same time, the patient's emotions and psychological state must be regularly evaluated, and attention must be paid to the patient's psychological stress, anxiety and other negative emotions. Traditional computer system rehabilitation training has fixed training content. Training plans and content can only be selected based on initial evaluations. Training content and training methods cannot be adjusted in real time based on training results and feedback; intensity and difficulty cannot be adjusted based on new conditions of patients (such as psychological state and emotional changes, physical condition, and the quality of training feedback).
[0071] In order to solve the above-mentioned problems of rigid training content, inability to dynamically adjust training content and training methods in real time according to training results and feedback, and inability to adjust intensity and difficulty according to new conditions of patients, the present application proposes a language rehabilitation training method, system, device and medium based on artificial intelligence.
[0072] Reference Figure 1 The present application embodiment provides a language rehabilitation training method based on artificial intelligence, and the method comprises the following steps:
[0073] Step S100, obtaining initial multimodal data;
[0074] Step S200: generating an initial training corpus and an initial training plan according to the initial multimodal data and the historical training plan;
[0075] Step S300, using the initial training corpus and the initial training plan to perform language rehabilitation pre-training on the patient to obtain a language rehabilitation training result;
[0076] Step S400, extracting language barrier information from the language rehabilitation training results, and adjusting the initial training corpus and the initial training program according to the language barrier information to obtain the target training corpus and the target training program;
[0077] Step S500: Perform daily language rehabilitation training on the patient using the target training corpus and the target training program, and obtain voice information and video information during the daily language rehabilitation training;
[0078] Step S600: determine the correct result of the voice information, and feed back the correct result to the terminal device;
[0079] Step S700: Identify the patient's mental state based on the voice information and video information, obtain a mental state recognition result, and adjust the intensity of daily language rehabilitation training based on the mental state recognition result.
[0080] In this embodiment, by acquiring initial multimodal data, the multimodal data can be used to more comprehensively examine the effect of the patient's language rehabilitation training. According to the initial multimodal data and the historical training program, the initial training corpus and the initial training program are generated, and then the initial training corpus and the initial training program are used to perform language rehabilitation pre-training on the patient to obtain the language rehabilitation training results, and then the information of the language disorder is extracted from the language rehabilitation training results, and the initial training corpus and the initial training program are adjusted according to the information of the language disorder to obtain the target training corpus and the target training program. Through multiple trainings, the training corpus and the training program that are more in line with the patient's situation can be obtained, which solves the problem of solidification of training content in rehabilitation training using traditional computer systems. The target training corpus and the target training program are used to perform daily language rehabilitation training on the patient, and the voice information and video information in the daily language rehabilitation training process are obtained, the correct result of the voice information is judged, and the correct result is fed back to the terminal device, so that the patient can know his training results and maintain a good mentality. Based on voice information and video information, the patient's mental state is identified to obtain the mental state recognition results, and the intensity of daily language rehabilitation training is adjusted according to the mental state recognition results. By adjusting the intensity of daily language rehabilitation training through the recognition results, personalized adjustment of training content is achieved, thereby improving the efficiency and effectiveness of language rehabilitation training.
[0081] The above-mentioned initial multimodal data includes multimodal data such as the patient's corresponding language disorder symptoms, causes, pronunciation training videos, and various daily text materials. These multimodal data can be pre-processed data. For example, the language disorder symptoms and causes are described in text form, and the pronunciation training video can also be converted into a document form using a preset speech-to-text model.
[0082] The above-mentioned preset speech-to-text model adopts the existing technology of software for converting speech into text, which is not described in detail in this embodiment.
[0083] The above-mentioned historical training scheme may be a method specified for patients with the same case in the past.
[0084] In some implementations, generating an initial training corpus and an initial training scheme based on the initial multimodal data and the historical training scheme includes:
[0085] The initial multimodal data and the historical training plan are input into the generative artificial intelligence system so that the generative artificial intelligence system learns the historical training plan and generates an initial training corpus and an initial training plan based on the initial multimodal data.
[0086] In this embodiment, the initial multimodal data and historical training plans of the patient's corresponding case are obtained and then input into a generative artificial intelligence system, and then the historical training plans are learned through the generative artificial intelligence system. The initial training corpus and initial training plan are generated according to the initial multimodal data. A set of plans and training corpus are pre-customized using the generative artificial intelligence system, which can ensure a certain degree of correctness and lay a good data foundation for the subsequent adjustment of the training content.
[0087] The above-mentioned generative artificial intelligence system (AIGC system) can be an artificial intelligence technology that uses complex algorithms, models and rules to learn from large-scale data sets to create new original content. This technology can create multiple types of content such as text, pictures, sounds, videos and codes, and comprehensively surpasses the data processing and analysis capabilities of traditional software.
[0088] In some embodiments, the patient is pre-trained for language rehabilitation using the initial training corpus and the initial training program to obtain language rehabilitation training results, including:
[0089] Inputting the initial training corpus and the initial training scheme into the artificial intelligence system so that the artificial intelligence system outputs the initial training corpus corresponding to the initial training scheme to the terminal;
[0090] The initial training corpus is displayed and played through the terminal, so that the patient can perform language rehabilitation pre-training according to the displayed and played initial training corpus.
[0091] In this embodiment, the initial training corpus is displayed and played through the terminal, so that the patient can perform language rehabilitation pre-training according to the displayed and played initial training corpus, which is convenient for the patient to perform language rehabilitation pre-training, and by training together by listening and watching, the efficiency of the patient's language rehabilitation training can be improved.
[0092] The artificial intelligence system can be an artificial intelligence system that can output corresponding initial training corpus according to the initial training scheme in the prior art. The artificial intelligence system includes various artificial intelligence algorithms.
[0093] The above-mentioned terminal can be a mobile phone, computer, tablet computer or other device that can play voice and display text and images.
[0094] In some embodiments, the language rehabilitation training results include the entire speech information recorded during the language rehabilitation training process, and the information about the language disorder is extracted from the language rehabilitation training results. The initial training corpus and the initial training program are adjusted according to the information about the language disorder to obtain the target training corpus and the target training program, including:
[0095] Using a preset speech-to-text model to recognize the entire speech information as the first text information;
[0096] Performing a text comparison between the first text information and the initial training corpus, extracting erroneous training corpus as information of language barriers, and extracting correct training corpus;
[0097] The information on language barriers and the correct training corpus are input into the generative artificial intelligence system, so that the generative artificial intelligence system can perform adversarial learning on the information on language barriers and the correct training corpus to generate target training corpus and target training plan.
[0098] In this embodiment, the information on language disorders and correct training corpus are input into a generative artificial intelligence system, so that the generative artificial intelligence system performs adversarial learning on the information on language disorders and the correct training corpus to generate target training corpus and a target training plan. The generative artificial intelligence system is again used to perform adversarial learning on the information on language disorders and the correct training corpus, and the training content is adjusted to obtain training corpus and training plan that meet the patient's condition.
[0099] The text comparison between the first text information and the initial training corpus may be performed by using existing text comparison technology or text comparison tools, which will not be described in detail in this embodiment.
[0100] In some implementations, determining a correct result of the voice information includes:
[0101] Using a preset speech-to-text model to recognize the speech information as second text information;
[0102] The second text information is compared with the target training corpus to obtain the correct result corresponding to the voice information.
[0103] In this embodiment, the second text information is compared with the target training corpus to obtain the correct result corresponding to the voice information. By obtaining the correct result during the training process, the patient can understand his or her own strengths and weaknesses, and then adjust his or her mentality and adjust the daily training time according to his or her demerits.
[0104] The above-mentioned text comparison between the second text information and the target training corpus to obtain the correct result corresponding to the voice information can be performed by comparing the second text information with the target training corpus, saving the correctly compared text as the correct result, and obtaining the correct result corresponding to the voice information.
[0105] In some embodiments, the patient's mental state is identified based on the voice information and the video information to obtain a mental state identification result, including:
[0106] Obtaining facial images from video information;
[0107] The facial image and voice information are input into a trained mental state recognition model to obtain a mental state recognition result, wherein the trained mental state recognition model includes multiple feature extraction modules, feature fusion modules and state recognition modules, and the multiple feature extraction modules, feature fusion modules and state recognition modules are trained by using multiple groups of facial images and voice information for training to obtain a trained mental state recognition model.
[0108] In this embodiment, the psychological state recognition result is obtained based on the facial image and voice information through the trained psychological state recognition model, which can improve the accuracy of psychological state recognition, thereby laying a good data foundation for the subsequent adjustment of training content.
[0109] In some embodiments, the facial image and voice information are input into a trained mental state recognition model to obtain a recognition result, including:
[0110] Using a first feature extraction module to extract features from a facial image to obtain facial expression features, and using a second feature extraction module to extract features from voice information to obtain voice information features;
[0111] Perform feature fusion of facial expression features and voice information features to obtain fusion features;
[0112] The fused features are input into the state recognition module to perform psychological state recognition and obtain the recognition result.
[0113] In this embodiment, by extracting features of facial images and voice information, then fusing facial expression features and voice information features, and then performing psychological state recognition based on the fused features, the accuracy of psychological state recognition can be improved, thereby laying a good data foundation for subsequent adjustment of training content.
[0114] The first feature extraction module and the second feature extraction module may use the same convolutional neural network or different convolutional neural networks.
[0115] The feature fusion module may adopt principal component analysis (PCA), maximum relevance minimum redundancy algorithm (mRMR), autoencoders, etc., which are not specifically limited in this embodiment.
[0116] The state recognition module may be a method in the prior art, such as a support vector machine, which is not specifically limited in this embodiment.
[0117] Reference Figure 2 The embodiment of the present application also provides a language rehabilitation training system based on artificial intelligence, which includes a data acquisition unit 100, a data generation unit 200, a pre-training unit 300, a data acquisition unit 400, a rehabilitation training unit 500, a result determination unit 600 and a training intensity adjustment unit 700, wherein:
[0118] A data acquisition unit 100, used to acquire initial multimodal data;
[0119] A data generating unit 200, configured to generate an initial training corpus and an initial training scheme according to the initial multimodal data and the historical training scheme;
[0120] The pre-training unit 300 is used to perform language rehabilitation pre-training on the patient using the initial training corpus and the initial training plan to obtain a language rehabilitation training result;
[0121] The data acquisition unit 400 is used to extract the information of the language disorder from the language rehabilitation training results, and adjust the initial training corpus and the initial training program according to the information of the language disorder to obtain the target training corpus and the target training program;
[0122] The rehabilitation training unit 500 is used to perform daily language rehabilitation training on the patient using target training corpus and target training program, and to obtain voice information and video information during the daily language rehabilitation training;
[0123] The result determination unit 600 is used to determine the correct result of the voice information and feed back the correct result to the terminal device;
[0124] The training intensity adjustment unit 700 is used to identify the patient's psychological state based on the voice information and video information, obtain the psychological state recognition result, and adjust the intensity of daily language rehabilitation training based on the psychological state recognition result.
[0125] It should be noted that since the artificial intelligence-based language rehabilitation training system in this embodiment and the above-mentioned artificial intelligence-based language rehabilitation training method are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.
[0126] Reference Figure 3 The embodiment of the present application further provides an electronic device, the electronic device comprising:
[0127] at least one memory;
[0128] at least one processor;
[0129] at least one program;
[0130] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned artificial intelligence-based language rehabilitation training method implemented in the present disclosure.
[0131] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.
[0132] The electronic device according to the embodiment of the present application is described in detail below.
[0133] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;
[0134] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the artificial intelligence-based language rehabilitation training method of the embodiment of the present disclosure.
[0135] Input / output interface 1800, used to implement information input and output;
[0136] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0137] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );
[0138] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .
[0139] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned artificial intelligence-based language rehabilitation training method.
[0140] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0141] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.
[0142] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0143] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0144] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0145] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0146] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.
[0151] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.
Claims
1. A language rehabilitation training method based on artificial intelligence, characterized in that: The method comprises: Acquire initial multimodal data; Generate initial training corpus and initial training scheme according to the initial multimodal data and historical training scheme; Using the initial training corpus and the initial training program to perform language rehabilitation pre-training on the patient to obtain a language rehabilitation training result; Extracting language barrier information from the language rehabilitation training results, and adjusting the initial training corpus and the initial training program according to the language barrier information to obtain target training corpus and the target training program; Using the target training corpus and the target training program to conduct daily language rehabilitation training on the patient, and obtaining voice information and video information during the daily language rehabilitation training; Determine the correct result of the voice information, and feed back the correct result to the terminal device; The patient's mental state is identified based on the voice information and the video information to obtain a mental state identification result, and the intensity of daily language rehabilitation training is adjusted based on the mental state identification result.
2. The artificial intelligence-based language rehabilitation training method according to claim 1, characterized in that: Generating initial training corpus and initial training scheme according to the initial multimodal data and historical training scheme includes: The initial multimodal data and the historical training scheme are input into a generative artificial intelligence system so that the generative artificial intelligence system learns the historical training scheme and generates an initial training corpus and an initial training scheme based on the initial multimodal data.
3. The artificial intelligence-based language rehabilitation training method according to claim 1, characterized in that: The method of using the initial training corpus and the initial training program to perform language rehabilitation pre-training on the patient to obtain language rehabilitation training results includes: Inputting the initial training corpus and the initial training scheme into an artificial intelligence system, so that the artificial intelligence system outputs the initial training corpus corresponding to the initial training scheme to a terminal; The initial training corpus is displayed and played through the terminal, so that the patient can perform language rehabilitation pre-training according to the displayed and played initial training corpus.
4. The artificial intelligence-based language rehabilitation training method according to claim 1, characterized in that: The language rehabilitation training result includes the whole speech information recorded during the language rehabilitation training process, and the information of the language disorder is extracted from the language rehabilitation training result, and the initial training corpus and the initial training program are adjusted according to the information of the language disorder to obtain the target training corpus and the target training program, including: Using a preset speech-to-text model to recognize the entire speech information as first text information; Performing a text comparison between the first text information and the initial training corpus, extracting erroneous training corpus as information of language barriers, and extracting correct training corpus; The information of the language barrier and the correct training corpus are input into a generative artificial intelligence system, so that the generative artificial intelligence system performs adversarial learning on the information of the language barrier and the correct training corpus to generate target training corpus and a target training plan.
5. The artificial intelligence-based language rehabilitation training method according to claim 1, characterized in that: The correct result of determining the voice information includes: Using a preset speech-to-text model to recognize the speech information as second text information; The second text information is compared with the target training corpus to obtain a correct result corresponding to the voice information.
6. The artificial intelligence-based language rehabilitation training method according to claim 1, characterized in that: The step of identifying the patient's mental state based on the voice information and the video information to obtain a mental state recognition result includes: Acquire a facial image in the video information; The facial image and the voice information are input into a trained psychological state recognition model to obtain a psychological state recognition result, wherein the trained psychological state recognition model includes multiple feature extraction modules, feature fusion modules and state recognition modules, and the multiple feature extraction modules, the feature fusion modules and the state recognition modules are trained by using multiple groups of facial images and voice information for training to obtain a trained psychological state recognition model.
7. The artificial intelligence-based language rehabilitation training method according to claim 6, characterized in that: The step of inputting the facial image and the voice information into a trained mental state recognition model to obtain a mental state recognition result includes: Using a first feature extraction module to extract features from the facial image to obtain facial expression features, and using a second feature extraction module to extract features from the voice information to obtain voice information features; Fusing the facial expression feature and the voice information feature to obtain a fused feature; The fusion feature is input into a state recognition module to perform psychological state recognition to obtain a psychological state recognition result.
8. A language rehabilitation training system based on artificial intelligence, characterized in that: The system comprises: A data acquisition unit, used for acquiring initial multimodal data; A data generating unit, configured to generate an initial training corpus and an initial training scheme according to the initial multimodal data and the historical training scheme; A pre-training unit, used to perform language rehabilitation pre-training on the patient using the initial training corpus and the initial training program to obtain a language rehabilitation training result; A data acquisition unit, configured to extract information about language disorders from the language rehabilitation training results, and adjust the initial training corpus and the initial training program according to the information about the language disorders to obtain target training corpus and the target training program; A rehabilitation training unit, used to perform daily language rehabilitation training on the patient using the target training corpus and the target training program, and to obtain voice information and video information during the daily language rehabilitation training; A result determination unit, used for determining a correct result of the voice information, and feeding back the correct result to the terminal device; The training intensity adjustment unit is used to identify the patient's psychological state according to the voice information and the video information, obtain a psychological state recognition result, and adjust the intensity of daily language rehabilitation training according to the psychological state recognition result.
9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the artificial intelligence-based language rehabilitation training method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the artificial intelligence-based language rehabilitation training method according to any one of claims 1 to 7.
