Comprehensive rehabilitation training machine language control method and comprehensive rehabilitation training machine

Abstract

The invention discloses a comprehensive rehabilitation training machine language control method and a comprehensive rehabilitation training machine, and the method comprises the steps: obtaining the case and demand information of a patient through a display controller, obtaining the eye expression characteristics of the patient through an image collection device in the vertical direction of the display controller, comparing the eye expression characteristics with the characteristics in an eye expression library, and obtaining the language information of the patient. Obtaining an eye control result; after voice information and mouth shape information of a patient are obtained, a voice control result and a mouth shape control result are determined; determining a control priority according to the case and demand information of the patient, wherein the control priority comprises an eye control result, a voice control result and a mouth shape control result; and executing a corresponding control instruction according to the control priority. Through the method, the control mode most suitable for different patients can be matched by evaluating the disability types and the disability degrees of the different patients, it can be ensured that the patients can smoothly control the equipment to a certain extent, and the patients can control the equipment more reliably and accurately through personalized adjustment of the control mode.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a language control method for a comprehensive rehabilitation training machine and a comprehensive rehabilitation training machine. Background Art

[0002] The comprehensive rehabilitation training machine mainly provides rehabilitation training for patients with limb paralysis, paraplegia, high paraplegia, etc. These patients have impaired muscle and nerve control and cannot complete training movements independently.

[0003] The comprehensive rehabilitation training machines in the prior art are mainly controlled by voice, and some comprehensive rehabilitation training machines are also equipped with lip shape and eye expression control methods for control.

[0004] However, the speech recognition in the prior art is usually based on the speech feature modeling of most ordinary patients, which does not take into account individual differences and cannot be specialized according to the actual needs of different patients. For example, some disabled patients stutter. If the comprehensive rehabilitation training machine also uses speech recognition or lip recognition, the recognition result is largely not what the patient wants. For example, some patients are deaf-mute. In this case, the patient cannot use speech to control the comprehensive rehabilitation training machine, but the prior art lacks alternative control modes. Summary of the invention

[0005] The present application provides a language control method for a comprehensive rehabilitation training machine and a comprehensive rehabilitation training machine, which are used to evaluate the disability type and degree of disability of different patients, match the control mode that best suits them, ensure that they can control the device smoothly, and through personalized adjustment of the control mode, make the patient's control of the device more reliable and accurate.

[0006] In a first aspect, the present application provides a language control method for a comprehensive rehabilitation training machine, the method comprising: obtaining a patient's case and demand information through a display controller, the case including the patient's disability type and degree of disability; obtaining the patient's eye features through an image acquisition device in a vertical direction of the display controller, the eye features including the start position, end position and trajectory of eye movement; comparing the eye features with the features in an eye library to obtain eye control results; after obtaining the patient's voice information and lip shape information, determining the voice control results and lip shape control results; determining a control priority based on the patient's case and demand information, the control priority including the eye control results, voice control results and lip shape control results; and executing corresponding control instructions based on the control priority.

[0007] In the above embodiment, the multi-mode adaptive control in the embodiment of the present application is adopted, by obtaining the patient's case information, eye control, voice control, and lip control, and determining the control priority of each mode according to the case information, and finally executing the corresponding control instructions according to the priority, thereby realizing personalized intelligent control for different patient disabilities, truly making the control of the comprehensive rehabilitation training machine "tailored to the needs of the students", serving the vast number of disabled groups, and bringing them a better user experience and greater convenience.

[0008] In combination with some embodiments of the first aspect, in some embodiments, after the step of obtaining the patient's case and demand information through the display controller, it also includes: determining a corresponding rehabilitation plan based on the patient's medical history and demand information, the rehabilitation plan including a video rehabilitation plan corresponding to the comprehensive rehabilitation training machine; and sending the video rehabilitation plan to the display controller for display.

[0009] In the above embodiment, since a video-assisted rehabilitation program is adopted, the video can be used to guide the patient to exercise, thereby improving the rehabilitation effect, and effectively solving the problem of the lack of customized programs for disabled patients in the existing rehabilitation system.

[0010] In combination with some embodiments of the first aspect, in some embodiments, after the step of sending the video rehabilitation program to the display controller for display, it also includes: collecting eye movement image information of the user watching the video through an image acquisition device; extracting feature data of the eye movement image information; judging the patient's attention distraction to the played video content based on the feature data, and obtaining an attention distraction judgment result; and adjusting the video rehabilitation program based on the attention distraction judgment result.

[0011] In the above embodiment, since the eye movement analysis and attention judgment technology is used, the patient's attention status when watching the video can be monitored in real time, which effectively solves the problem of mismatch between video playback and individual conditions.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the step of determining the voice control result and the lip control result, it also includes: after obtaining the patient's voice information, detecting whether the voice information is dialect voice information; if so, comparing it with the dialect database according to the corresponding dialect features to obtain the dialect recognition result; and obtaining the voice control result according to the dialect recognition result.

[0013] In the above embodiment, since the dialect speech recognition technology is adopted, accurate speech control can be performed even if the patient uses a dialect, which effectively solves the problem of insufficient support for dialects in the prior art.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after the step of executing the corresponding control instruction according to the control priority, it also includes: detecting the patient's heart rate information; associating the heart rate information with the remote monitoring end, and sending the heart rate information to the remote monitoring end; if the heart rate is detected to be greater than the set threshold, an alarm message is sent to the remote monitoring end, and the alarm message includes a text message alarm.

[0015] In the above embodiment, due to the use of physiological monitoring and early warning technology, the patient's heart rate can be monitored, and an early warning can be sent to the doctor when a problem occurs, which effectively solves the risk of recurrence of the patient's condition during use and ensures safe use.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the step of executing the corresponding control instruction according to the control priority specifically includes: detecting the user's operation of clicking the massage control on the display controller, and turning on the massage function.

[0017] In the above embodiment, since the massage auxiliary technology is used, the discomfort of patients sitting still for a long time can be alleviated, and the problem that the existing equipment cannot fully meet the needs of patients is effectively solved.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of executing the corresponding control instruction according to the control priority, it also includes: receiving the ease of use evaluation result of each control result fed back by the patient; updating the credibility weight of each control result according to the ease of use evaluation result; multiplying the credibility weight by the priority weight to obtain an adjusted weight; and determining the control result with the highest adjusted weight as the final control instruction.

[0019] In the above embodiment, since the feedback adjustment personalization technology is adopted, the system can be continuously optimized according to the patient's usage experience, which effectively solves the problem that the existing technology cannot adapt to individual changes.

[0020] In a second aspect, an embodiment of the present application provides a comprehensive rehabilitation training machine, the comprehensive rehabilitation training machine comprising: A medical record acquisition module is used to acquire the patient's case and demand information through a display controller, wherein the case includes the patient's disability type and degree of disability; An eye feature acquisition module, used to acquire the patient's eye features through an image acquisition device in a vertical direction of the display controller, the eye features including the start position, end position and trajectory of the eye movement; An eye control result acquisition module is used to compare the eye features with features in the eye library to obtain eye control results; A voice information and lip shape information acquisition module, used to determine the voice control result and the lip shape control result after acquiring the voice information and lip shape information of the patient; A control priority determination module, used to determine the control priority according to the patient's case and demand information, the control priority including the eye control result, the voice control result and the lip control result; The comprehensive rehabilitation training machine control module is used to execute corresponding control instructions according to the control priority.

[0021] In the third aspect, an embodiment of the present application provides a comprehensive rehabilitation training machine, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the comprehensive rehabilitation training machine to perform the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the above-mentioned computer program product is run on a comprehensive rehabilitation training machine, the above-mentioned comprehensive rehabilitation training machine executes the method described in the first aspect and any possible implementation method of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a comprehensive rehabilitation training machine, the comprehensive rehabilitation training machine executes the method described in the first aspect and any possible implementation method of the first aspect.

[0024] It is understandable that the comprehensive rehabilitation training machine provided in the second and third aspects, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0025] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By obtaining patient case information, eye control, voice control, and lip control, and determining the control priority of each mode based on the case information, the corresponding control instructions are finally executed according to the priority, realizing personalized intelligent control for different patient disabilities. While serving the vast number of disabled people, it also brings them a better user experience and greater convenience.

[0026] 2. Due to the use of video-assisted rehabilitation programs, videos can be used to guide patients to exercise and improve rehabilitation effects, effectively solving the problem of lack of personalized programs for disabled people in the existing rehabilitation system.

[0027] 3. Due to the use of feedback adjustment personalized technology, the system can be continuously optimized according to the patient's usage experience, which effectively solves the problem that the existing technology cannot adapt to individual changes and realizes a system that is continuously iterated and updated according to individual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a language control method of a comprehensive rehabilitation training machine in an embodiment of the present application; Figure 2 is another flow chart of a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application; Figure 3 is another flow chart of a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application; Figure 4 It is a schematic diagram of the functional module structure of the comprehensive rehabilitation training machine in the embodiment of the present application; Figure 5 It is a schematic diagram of the physical device structure of the comprehensive rehabilitation training machine in the embodiment of the present application. DETAILED DESCRIPTION

[0029] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0031] For ease of understanding, a related application of an embodiment of the present application is introduced below.

[0032] At present, comprehensive rehabilitation training machines mainly provide rehabilitation exercises for patients with limb paralysis, paraplegia, high paraplegia, etc. These patients have impaired muscle and nerve control ability and cannot complete training movements independently. Comprehensive rehabilitation training machines in the prior art are mainly controlled by voice or manual joysticks, and some comprehensive rehabilitation training machines are also equipped with lip shape and eye control methods. However, speech recognition in the prior art is usually based on the voice feature modeling of most patients, and individual differences are not taken into consideration enough, and it is impossible to perform specialized control according to the actual needs of different patients.

[0033] The multi-mode control in the embodiment of the present application is used to obtain patient instructions. By obtaining multi-mode control instructions such as eye contact, voice, and lip shape, and determining the control priority of each mode according to the patient's specific disability situation, the highest priority control instruction is finally executed according to the priority, thereby realizing personalized intelligent control for different patient disabilities. This not only greatly enhances the availability and ease of use of the system, but more importantly, improves the intelligence and personalization level of the system, allowing the comprehensive rehabilitation training machine to teach students in accordance with their aptitude and control according to individual needs, bringing better user experience and greater convenience to the majority of disabled people.

[0034] Compared with the single control method based on the universal model in the prior art, this application realizes precise intelligent control based on individual needs by obtaining multi-mode control instructions and performing personalized priority processing, truly solving the individual differences of users with different disabilities, enabling them to control the equipment smoothly and effectively and obtain good use results.

[0035] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , Figure 1 It is a flow chart of a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application.

[0036] S101, obtaining the patient's case and demand information through a display controller, where the case includes the patient's disability type and degree of disability; The medical record acquisition module in the comprehensive rehabilitation training machine is first connected to the display controller. The display controller can be a touch screen display specially set on the comprehensive rehabilitation training machine, or it can be a connected tablet computer or mobile phone. An image acquisition device can be set above the screen of the display controller for subsequent eye and lip recognition. The display controller can also be set with a voice recognition function. After starting the display controller, the comprehensive rehabilitation training machine will actively prompt the patient or doctor to enter the patient's personal information on the display controller, including text information such as name, age, medical information and contact information, and can also prompt to upload the patient's image data, such as X-rays, CT scans, etc. The comprehensive rehabilitation training machine can use image recognition, natural language processing and other technical analysis to determine the patient's main disability type, such as paraplegia, high paraplegia, etc. from these text or image information. For example, if the uploaded CT film shows spinal cord damage, it can be judged as paraplegia.

[0037] The degree of disability can also be analyzed and determined from the medical records, such as the height of the paraplegic segment, whether it is completely paralyzed, etc. Different patients have different muscle and nerve control abilities. These cases and demand information will be recorded by the display controller on the comprehensive rehabilitation training machine for subsequent determination of the control mode.

[0038] S102, acquiring the patient's eye features through an image acquisition device in a vertical direction of the display controller, the eye features including the start position, end position and trajectory of the eye movement; After obtaining the patient's medical history information, the comprehensive rehabilitation training opportunity will actively control the display controller to adjust its position so that the image acquisition device on the display controller, such as the camera, is vertically aligned with the patient's face and eyes, which is necessary to obtain clear eye movement video.

[0039] Then the image acquisition device starts to acquire images of the patient's eye area to obtain an eye movement video.

[0040] The display controller processes the collected eye movement video, mainly extracting features such as pupil edge and corneal reflection point of the captured eye movement through image processing technology. These feature points reflect the starting and ending positions of the eye movement. The trajectory of the eye movement can be calculated through the time series changes of the feature points. These data including the starting position, ending position and trajectory will be recorded and provided as eye features for subsequent eye control.

[0041] S103, comparing the eye gaze feature with features in an eye gaze database to obtain an eye gaze control result; After obtaining the eye feature data, it is matched with the pre-stored eye library. The eye library pre-stores different standard eye movement features corresponding to different control instructions, such as turning the eyes upwards for the chair to move forward, turning the eyes right for the chair to turn right, etc. The display controller on the comprehensive rehabilitation training machine calculates the similarity between the patient's eye movement features and each eye movement feature in the standard library, and finds the most matching one, that is, the standard feature with the highest matching degree. Then the control instruction corresponding to the most matching standard feature is taken out and determined as the current eye control result. This result, together with other control results, will provide a basis for the subsequent control priority determination to determine the final control instruction.

[0042] It should be noted that if the eye movement feature does not match any standard feature in the library, or the matching degree does not exceed the set matching threshold, it is judged that the eye control is invalid.

[0043] S104, after acquiring the patient's voice information and lip shape information, determining a voice control result and a lip shape control result; While obtaining the patient's eye control results, the display controller on the comprehensive rehabilitation training machine will begin to collect the patient's voice input and lip shape change information. For voice input, the display controller first performs pre-processing to remove silence and noise, and then extracts the acoustic features of the voice, including parameters such as intonation, pitch, and form. These parameters are matched with the preset voice pattern library to find the most likely corresponding voice recognition result as the voice control result.

[0044] It should be noted that if the recognition result is empty or lower than the voice recognition confidence threshold, the voice control is determined to be invalid.

[0045] For lip shape, the display controller captures the patient's mouth shape changes, lip movements and other features through image processing, and matches them with the standard lip shape pattern library to obtain lip shape recognition results and determine the lip shape control results. Similarly, if the result is empty or lower than the lip shape recognition confidence threshold, it is determined that the lip shape control is invalid.

[0046] It should be noted that because the mouth shape of the disabled people of the deaf-mute population may be different from that of non-deaf-mute people, the mouth shape of the deaf-mute people is obviously different from that of ordinary people, mainly reflected in: the deaf-mute people may not speak, and do not understand what kind of mouth shape the mouth should form when speaking; the deaf-mute people mainly express their meaning through the degree and direction of the mouth opening, rather than the specific lip shape; for congenital deaf-mute people, they have never learned the mouth shape of speaking, they may not know what the specific mouth shape of speaking is, and the mouth shape is more different from that of ordinary people. In this case, a separate sample library of deaf-mute mouth shape can be established, which includes a variety of mouth shapes corresponding to the deaf-mute people and the specific meanings expressed by the corresponding mouth shapes. When the patient uses the comprehensive rehabilitation training machine, he can input relevant information such as deafness and dumbness, so that the comprehensive rehabilitation training machine can determine that the patient is deaf-mute. Then, when extracting features, not only the lip shape features are extracted, but also the opening size and direction features of the mouth shape are extracted, the mouth shape area is segmented, and the mouth shape boundary information is obtained, and then the mouth shape opening information, including the size and direction of the mouth shape, is determined. Then the mouth shape information is compared with the deaf-mute population shape sample library to obtain the mouth shape recognition result. For example, when the mouth of Mr. Li, a deaf-mute patient, tilts to the left, after lip shape recognition, it is determined that the mouth direction is to turn left, so a control command to turn left is sent to the comprehensive rehabilitation training machine, and the comprehensive rehabilitation training machine rotates to the left after receiving the command. When Mr. Li's mouth opens into an "O" shape, the comprehensive rehabilitation training machine measures and determines that the mouth opening is large, and the corresponding control command is "acceleration", so the comprehensive rehabilitation training machine receives the command to increase the speed. When Mr. Li closes his mouth until only a thin gap is left, the lip shape recognition determines that the mouth opening is very small, corresponding to the "deceleration" command, and the comprehensive rehabilitation training machine decreases the speed after receiving it.

[0047] In some embodiments, some disabled people cannot speak Mandarin. For this situation, after obtaining the patient's voice information, the display controller on the comprehensive rehabilitation training machine detects whether the voice information is dialect voice information; if so, it compares the corresponding dialect features with the dialect database to obtain the dialect recognition result; and obtains the voice control result based on the dialect recognition result. Specifically, the display controller first collects the patient's voice input information through a microphone. After collecting the voice, the system will determine whether it is standard Mandarin or what dialect it is based on the acoustic characteristics of the voice. This usually requires distinguishing through voice spectrum features, intonation and tone features, etc. If it is detected as a dialect, the system will call the corresponding dialect speech recognition model for recognition. The dialect database will pre-store dialect feature models of different regions, such as Cantonese model, Hunan dialect model, etc. The voice command is recognized by the dialect speech recognizer and mapped to the control operation of the comprehensive rehabilitation training machine to generate a voice control result. In addition, we can also combine multiple languages ​​from other countries, such as English, French, Japanese, etc., especially to obtain language corpora for people with disabilities. We can use the collected multilingual corpora to train speech recognition models in different languages ​​through deep learning and other methods, and make language judgments on the input speech to distinguish the language, such as judging whether it is English or French through speech spectrum features. According to the judged language type, we call the corresponding speech recognition model for recognition to obtain speech recognition results in different languages.

[0048] S105, determining a control priority according to the patient's case and demand information, the control priority including eye gaze control results, voice control results, and lip shape control results; After obtaining the control results of multiple modes, the comprehensive rehabilitation training machine will determine the weights and priorities for different control modes based on the previously recorded patient cases and demand information. For example, if the patient's disability type is high paraplegia, the comprehensive rehabilitation training machine can automatically preset the eye control result with the highest weight, followed by the voice control result, and then the lip shape result. The weight of each mode can also be fine-tuned according to the patient's degree of disability, usage habits, etc. The mode with a high weight will have a higher priority when determining the final control instruction.

[0049] In some embodiments, in addition to the preset weights, the comprehensive rehabilitation training machine also adjusts the priority of each mode based on its recognition confidence.

[0050] S106: Execute corresponding control instructions according to the control priority.

[0051] After determining the priority of the control mode, the comprehensive rehabilitation training machine will select the one with the highest priority from each control result, determine it as the priority control instruction, and send the control instruction to the executive body of the comprehensive rehabilitation training machine for execution. For example, if the patient is a deaf-mute with high paraplegia, after the patient enters his case information on the display controller, the comprehensive rehabilitation training machine analyzes the case and identifies the patient as a high paraplegic with severe hearing and speech disorders. Because the patient is identified as a deaf-mute, the weight and priority of the speech recognition mode are automatically lowered. For the deaf-mute, speech recognition is meaningless, but more attention is paid to eye contact and lip shape control, and the priority of eye contact and lip shape control is correspondingly increased. More specifically, if the current eye contact result of the deaf-mute patient is "forward" and the lip shape result is "turn left", if the preset priority is that the eye contact is the highest, the eye contact result "forward" is selected as the final control instruction.

[0052] After the comprehensive rehabilitation training machine determines the final control instruction, it will call the motor, servo and other motion devices in the actuator to control the seat movement, change posture, etc. according to the instruction. At the same time, it will also record the final executed control instruction for subsequent control optimization and adjustment.

[0053] In the above embodiment, the multi-mode adaptive control in the embodiment of the present application is adopted. By obtaining the patient's case information, eye control, voice control, and lip control, and determining the control priority of each mode according to the case information, the corresponding control instructions are finally executed according to the priority, thereby realizing personalized intelligent control for different patient disabilities. Not only does it greatly enhance the availability and ease of use of the system, this application not only realizes intelligent language control, but more importantly, it improves the intelligence and personalization level of the system, truly making the control of the comprehensive rehabilitation training machine "tailored to the needs of the students", serving the vast number of disabled people, and bringing them a better user experience and greater convenience.

[0054] In some embodiments, the comprehensive rehabilitation training machine can detect the patient's heart rate information; associate the heart rate information with the remote monitoring terminal, and send the heart rate information to the remote monitoring terminal; if the heart rate is detected to be greater than the set threshold, an alarm message is sent to the remote monitoring terminal, and the alarm message includes a text message alarm. Specifically, a heart rate sensor is integrated in the comprehensive rehabilitation training machine, which can detect the heart rate value of the patient in real time during use, and send the detected heart rate data to the remote monitoring terminal via Bluetooth or WiFi. The monitoring terminal can be operated by a doctor or nurse or the patient's family. If the patient's heart rate exceeds the normal threshold, that is, an abnormality occurs, an alarm will be triggered. The normal heart rate threshold can be preset according to personal circumstances. At this time, an alarm message is sent to the remote monitoring terminal. The alarm message contains heart rate data, alarm time, etc. At the same time, a prompt is sent to the mobile phone terminal in the form of a text message, which is convenient for nurses or family members to be informed of the situation in time, monitor the patient's physiological state in real time, and provide quick feedback once an abnormality occurs, playing a role in remote care.

[0055] In some embodiments, a control button or touch area for a massage function may be set on the interface of the display controller, and the position is set in an area that the patient can easily touch. The comprehensive rehabilitation training machine has a built-in massage device, including a massage module, a motor, and a transmission device. The massage module can be set in both the backrest and the seat to provide full-body massage. The comprehensive rehabilitation training machine detects that the user clicks on the massage control area through the touch sensor of the display controller, and then turns on the corresponding massage function, which can press, knead or tap the patient's muscle tissue to complete the massage service. This provides a convenient massage control method that can relieve fatigue caused by long-term sitting.

[0056] In some embodiments, the comprehensive rehabilitation training machine can obtain the patient's usage evaluation after the patient finishes using it, so as to optimize the control actions later.

[0057] The following is a supplement to this embodiment. Figure 2 , Figure 2 It is another flow chart of a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application.

[0058] S201, receiving the usability evaluation result of each control result fed back by the patient; After the comprehensive rehabilitation training machine executes the action corresponding to the control instruction, the patient will be prompted on the display controller interface to evaluate the ease of use of each control mode used, such as eye control, voice control, and lip control. The evaluation can be carried out by clicking to select. For example, if a 5-star rating is set, the patient can click to select the ease of use rating for each control mode. You can also set a voice prompt to require the patient to voice feedback on the ease of use of each control mode, such as saying "very good", "good", "average", "difficult", or "very difficult". Voice evaluation does not require hand operation, which is more in line with the usage habits of some patients with limb disabilities; patients can also choose to evaluate different modes by rotating their eyeballs, such as rotating left and right to select the rating level, which is suitable for some patients who cannot operate with their hands.

[0059] The display controller will receive and record the ease-of-use evaluation results submitted by the patient. These evaluation results can more intuitively reflect the actual use experience of different modes for the patient. Collecting these feedbacks will help to adjust and optimize the control mode in the future.

[0060] S202, updating the credibility weight of each control result according to the usability evaluation result; After collecting the ease of use evaluation from patients, the display controller will apply these evaluation results to update the credibility weight of each control mode. For example, if the patient feedback that the ease of use of eye control is very high, the credibility weight of the eye control result in determining the final control instruction will be increased. Conversely, if a certain mode receives a very low ease of use score, the credibility weight of the mode will be reduced.

[0061] Through such a mechanism, the credibility weights of different modes can be automatically adjusted according to patient feedback to make them consistent with the patient's actual usage experience. From a statistical point of view, if a mode receives highly positive feedback from patients for a long time, it means that it is suitable for the patient and its credibility should gradually increase.

[0062] S203, multiplying the credibility weight by the priority weight to obtain an adjusted weight; After updating the credibility weight of each mode, the comprehensive rehabilitation training machine will multiply it with the corresponding preset priority weight to obtain the adjusted weight. For example, assuming that the credibility weight of eye control is adjusted to 0.8 and the priority weight is 0.9, then the two weights are multiplied and the adjusted weight of eye control is 0.72.

[0063] In this way, when determining the final control instructions, both the credibility of the mode and the preset priority are considered. This combines the learning of the patient's usage habits with the preset priority based on the disability situation to make a smarter and more personalized control mode fusion.

[0064] S204: Determine the control result with the highest adjusted weight as the final control instruction.

[0065] The comprehensive rehabilitation training opportunity selects the one with the largest weight from the adjusted weights of all control modes and determines it as the final control instruction. For example, assuming that the current set of control results is: the eye control result is "forward", with an adjusted weight of 0.72; the voice control result is empty, with an adjusted weight of 0.1; the lip control result is "turn left", with an adjusted weight of 0.6, then the eye control "forward" is selected as the final control instruction because its adjusted weight is the largest.

[0066] Compared with selecting control results based solely on preset priorities, this weight-based selection method is smarter and more reliable. The reason is that the preset priorities are based on the overall situation of the disabled patient group, but there are still deviations when it comes to individuals. The weights take into account the feedback from the individual's long-term usage habits and can adapt to each patient more accurately. In addition, this adjusted weight also incorporates the confidence of the recognition results, which can eliminate some recognition errors with low credibility. Therefore, selecting the final control instructions based on the weights can make the comprehensive rehabilitation training machine more "intelligent" for individual users and establish a "privately customized" interaction mode with patients.

[0067] After combining the above embodiments, the method provided by this embodiment is further described in more detail below. Figure 3 , Figure 3 It is another flow chart of a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application.

[0068] S301, determining a corresponding rehabilitation program according to the patient's medical history and demand information, the rehabilitation program including a video rehabilitation program corresponding to the comprehensive rehabilitation training machine; The comprehensive rehabilitation training machine matches an exercise rehabilitation program suitable for the patient's condition from the original rehabilitation program library based on previously recorded patient case information, such as the patient's disability type and degree of disability. The program will include physical rehabilitation movements performed using the comprehensive rehabilitation training machine, as well as video content displayed on the display controller for guidance. For example, for a paraplegic patient, if the patient clicks on the rehabilitation program on the display screen, the comprehensive rehabilitation training machine will recommend a corresponding video rehabilitation program to the patient. The video will introduce what kind of exercise movements are suitable for paraplegic patients, as well as teaching specific methods to complete the movements. The determined rehabilitation program will be sent to the display controller for use in exercise.

[0069] S302, sending the video rehabilitation program to the display controller for display; After the display controller receives the video rehabilitation program sent by the comprehensive rehabilitation training machine, it will play these videos on the screen for the patient to watch. The video will give specific exercise steps and method prompts, such as using virtual characters displayed on the screen to simulate movements, deepen the patient's impression and improve the training effect. The actuator of the comprehensive rehabilitation training machine will cooperate with the video content to guide the patient to complete the exercise movements.

[0070] S303, collecting eye movement image information of the user watching the video through an image acquisition device; In order to monitor the patient's concentration level when watching videos, the camera on the display controller will collect the patient's eye movement video within a set time. The eye movement video will contain information such as the patient's pupil position, the starting and ending positions of the eye movement. The display controller on the comprehensive rehabilitation training machine can determine whether the patient is watching the video carefully by analyzing the changes in the eye's focus point and gaze time.

[0071] S304, extracting feature data of the eye movement image information; The display controller of the comprehensive rehabilitation training machine has collected the eye movement image information of the patient when watching the video. In order to analyze whether the patient's attention is focused, it is necessary to first extract the feature data of these eye movement images, which may include: the change of pupil position coordinates, the coordinates of the eye gaze point, the gaze duration, the number of frequent changes of the gaze point, etc. These feature data can reflect whether the patient's eyes are focused on the video content.

[0072] For example, if the gaze point frequently falls outside the video frame, it means that the eye tracking is not focused; if the gaze time is too short, it also means that the attention is distracted.

[0073] S305, judging the patient's distraction status to the played video content according to the characteristic data, and obtaining a distraction judgment result; The display controller of the comprehensive rehabilitation training machine will compare the eye movement image features collected during video playback with the characteristic data of the standard concentration state to determine the difference between the two. For example, if the average coordinates of the eye gaze point deviate from the center area of ​​the video by more than a certain distance, or the coordinate variation range of the gaze point exceeds a certain threshold, it can be judged as distraction. By combining multiple characteristic data, a distraction degree score or probability value is obtained. If the score or probability exceeds a certain threshold, it is judged as obvious distraction, and finally the distraction judgment result is obtained.

[0074] S306: Adjust the video rehabilitation program according to the attention distraction judgment result.

[0075] After obtaining the distraction results, the comprehensive rehabilitation training machine can adjust the video program according to the results to improve the patient's concentration. For example, if it is judged that the attention is significantly distracted, a reminder voice can be inserted to prompt the patient to concentrate; or a slight audio-visual stimulation element can be added to the video to attract attention. If the distraction is serious, the length of a single video can be reduced, and more small videos can be used to combine and increase changes to avoid the loss of attention due to too long a duration. The degree of distraction of different patients can also be recorded, and a combination of audio-visual content with a faster rhythm and stronger stimulation can be used for patients who are easily distracted to form a personalized video rehabilitation program.

[0076] In the embodiments of the present application, due to the use of video-assisted rehabilitation and attention monitoring and program adjustment technology, the video-assisted rehabilitation program can be determined according to the patient's disability, the patient's attention can be monitored through eye movement analysis, and the video program can be adjusted intelligently, which effectively solves the problem of patient inattention caused by ignoring individual differences in existing rehabilitation systems, thereby achieving accurate video-assisted rehabilitation based on individual needs and situations, and bringing a better rehabilitation experience to disabled people.

[0077] The following is an introduction to a comprehensive rehabilitation training machine in the embodiment of the present application from the perspective of modules. Figure 4 , Figure 4 It is a schematic diagram of the functional module structure of the comprehensive rehabilitation training machine in the embodiment of the present application.

[0078] The medical record acquisition module 401 is used to acquire the patient's case and demand information through the display controller, and the case includes the patient's disability type and degree of disability; An eye feature acquisition module 402 is used to acquire the patient's eye features through an image acquisition device in a vertical direction of the display controller, wherein the eye features include a start position, an end position, and a trajectory of eye movement; The eye gaze control result acquisition module 403 is used to compare the eye gaze feature with the features in the eye gaze library to obtain the eye gaze control result; The voice information and lip shape information acquisition module 404 is used to determine the voice control result and the lip shape control result after acquiring the voice information and lip shape information of the patient; A control priority determination module 405 is used to determine the control priority according to the patient's case and demand information, and the control priority includes eye control results, voice control results and lip control results; The comprehensive rehabilitation training machine control module 406 is used to execute corresponding control instructions according to the control priority.

[0079] In some embodiments, the medical record acquisition module 401 specifically includes: A rehabilitation program determination unit, used to determine a corresponding rehabilitation program according to the patient's medical history and demand information, the rehabilitation program including a video rehabilitation program corresponding to the comprehensive rehabilitation training machine; The rehabilitation program display unit is used to send the video rehabilitation program to the display controller for display.

[0080] In some embodiments, the medical record acquisition module 401 further includes: An eyeball movement image information acquisition unit, used to acquire eyeball movement image information of a user watching a video through an image acquisition device; A feature data extraction unit, used to extract feature data of the eye movement image information; an attention distraction judgment result obtaining unit, used to judge the patient's attention distraction to the played video content according to the characteristic data, and obtain an attention distraction judgment result; The program adjustment unit is used to adjust the video rehabilitation program according to the attention distraction judgment result.

[0081] In some embodiments, the voice information and lip shape information acquisition module 404 specifically includes: A dialect voice information detection unit, used to detect whether the voice information is dialect voice information after acquiring the voice information of the patient; A dialect recognition result obtaining unit is used to compare the dialect features with the dialect database to obtain the dialect recognition result; The voice control result obtaining unit is used to obtain the voice control result according to the dialect recognition result.

[0082] In some embodiments, the comprehensive rehabilitation training machine control module 406 specifically includes: A heart rate information detection unit, used to detect the patient's heart rate information; A heart rate information sending unit, used to associate the heart rate information with a remote monitoring terminal and send the heart rate information to the remote monitoring terminal; The alarm information sending unit is used to send an alarm message to the remote monitoring terminal if the heart rate is detected to be greater than a set threshold, and the alarm message includes a text message alarm.

[0083] In some embodiments, the comprehensive rehabilitation training machine control module 406 further includes: The massage function activation unit is used to detect the user's operation of clicking the massage control on the display controller and activate the massage function.

[0084] In some embodiments, the comprehensive rehabilitation training machine control module 406 further includes: An ease of use evaluation result receiving unit, used for receiving an ease of use evaluation result of each control result fed back by a patient; A credibility weight obtaining unit, used for updating the credibility weight of each control result according to the usability evaluation result; An adjusted weight obtaining unit, used for multiplying the credibility weight by the priority weight to obtain an adjusted weight; The final control instruction determining unit is used to determine the control result with the highest adjusted weight as the final control instruction.

[0085] The above describes a comprehensive rehabilitation training machine in the embodiment of the present application from the perspective of modular functional entities. The following describes a comprehensive rehabilitation training machine in the embodiment of the present application from the perspective of hardware processing. Figure 5 , Figure 5 It is a schematic diagram of the physical device structure of the comprehensive rehabilitation training machine in the embodiment of the present application.

[0086] The comprehensive rehabilitation training machine 500 includes: one or more processors 501 ( Figure 5 Taking a processor 501 as an example), a memory 502, an input device 503 and an output device 504. In some embodiments of the present invention, the processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means, wherein: Figure 5 The example of connecting through bus is taken in the following.

[0087] The processor 501 calls the computer instruction so that the computer program is executed by the processor to implement a language control method for a comprehensive rehabilitation training machine in an embodiment of the present application.

[0088] The memory 502 is used to store computer program codes, where the computer program codes include computer instructions.

[0089] The input device 503 is used to input the patient's case information.

[0090] The output device 504 is used to execute the action corresponding to the control instruction.

[0091] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0092] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0093] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0094] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A language control method for a comprehensive rehabilitation training machine, applied to a comprehensive rehabilitation training machine, wherein the comprehensive rehabilitation training machine is provided with a display controller, characterized in that: The method comprises: Obtaining the patient's case and demand information through the display controller, wherein the case includes the patient's disability type and degree of disability; Acquiring the patient's eye features through an image acquisition device in a vertical direction of the display controller, wherein the eye features include a starting position, an ending position, and a trajectory of eye movement; Comparing the eye gaze feature with features in an eye gaze library to obtain an eye gaze control result; After obtaining the patient's voice information and lip shape information, the decibel determines the corresponding voice control result and lip shape control result; Determine the control priority according to the patient's case and demand information, wherein the control priority includes eye control results, voice control results, and lip control results; Execute corresponding control instructions according to the control priority.

2. The method according to claim 1, characterized in that After the step of obtaining the patient's case and demand information through the display controller, the method further includes: Determine a corresponding rehabilitation program according to the patient's medical history and demand information, wherein the rehabilitation program includes a video rehabilitation program corresponding to the comprehensive rehabilitation training machine; The video rehabilitation program is sent to the display controller for display.

3. The method according to claim 2, characterized in that After the step of sending the video rehabilitation program to the display controller for display, the method further includes: Collecting eye movement image information of a user watching a video through an image acquisition device; Extracting feature data of the eye movement image information; Determine the patient's attention distraction to the played video content according to the characteristic data, and obtain an attention distraction determination result; The video rehabilitation program is adjusted according to the attention distraction judgment result.

4. The method according to claim 1, characterized in that: After the step of determining the voice control result and the lip shape control result, the method further includes: After acquiring the patient's voice information, detecting whether the voice information is dialect voice information; If yes, then compare the corresponding dialect features with the dialect database to obtain the dialect recognition result; A voice control result is obtained according to the dialect recognition result.

5. The method according to claim 1, characterized in that After the step of executing the corresponding control instruction according to the control priority, the method further includes: Detect the patient's heart rate information; Associating the heart rate information with a remote monitoring terminal, and sending the heart rate information to the remote monitoring terminal; If it is detected that the heart rate information is greater than a set threshold, an alarm message is sent to the remote monitoring terminal, and the alarm message includes a text message alarm.

6. The method according to claim 1, characterized in that After the step of executing the corresponding control instruction according to the control priority, the method further includes: The massage function is turned on when a user clicks a massage control on the display controller.

7. The method according to claim 1, characterized in that After the step of executing the corresponding control instruction according to the control priority, the method further includes: Receive patient feedback on the ease of use evaluation results for each control result; updating the credibility weight of each control result according to the usability evaluation result; Multiplying the credibility weight by the priority weight to obtain an adjusted weight; The control result with the highest adjusted weight is determined as the final control instruction.

8. A comprehensive rehabilitation training machine, characterized in that: include: A medical record acquisition module, used to acquire the patient's case and demand information through the display controller, wherein the case includes the patient's disability type and degree of disability; An eye feature acquisition module, used to acquire the patient's eye features through an image acquisition device in a vertical direction of the display controller, wherein the eye features include a starting position, an ending position, and a trajectory of eye movement; An eye control result acquisition module, used for comparing the eye features with features in an eye library to obtain an eye control result; A voice information and lip shape information acquisition module, used to determine the voice control result and the lip shape control result after acquiring the voice information and lip shape information of the patient; A control priority determination module, used to determine the control priority according to the patient's case and demand information, wherein the control priority includes eye control results, voice control results and lip control results; The comprehensive rehabilitation training machine control module is used to execute corresponding control instructions according to the control priority.

9. A comprehensive rehabilitation training machine, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the comprehensive rehabilitation training machine to execute the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the comprehensive rehabilitation training machine, the comprehensive rehabilitation training machine executes the method as claimed in any one of claims 1 to 7.

Images