Training method, system and equipment based on multi-modal man-machine interaction cooperative control

Abstract

The invention discloses a training method, system and equipment based on multi-modal man-machine interaction cooperative control. Firstly, a user information management module determines a target training item based on basic information of a user, and a customized training plan can be provided, so that the accuracy and pertinence of training are improved. Then, the rehabilitation training selection module determines a target training demand according to the target training item and the basic information of the user; and then, the signal acquisition and decoding module determines a mapping relation between an interaction object control command and a user control action based on the target training requirement, so that interaction between the target interaction object and the user is realized. And finally, the man-machine interaction cooperative control module determines a target interaction control mode based on the target training demand, determines an interaction object control command corresponding to the received user control action according to the mapping relation through the target interaction control mode under the condition of receiving the user control action, and sends the interaction object control command to the user. And controlling the target interaction object based on the interaction object control command to realize training.

Description

Technical Field

[0001] The present invention relates to the field of human-computer collaborative technology, and in particular to a training method, system and equipment based on multi-modal human-computer interactive collaborative control. Background Art

[0002] Current rehabilitation training usually relies on a single biological signal (such as electromyography, electroencephalography, etc.) for identification and control. Although it can help patients with rehabilitation training to a certain extent, it fails to fully utilize the advantages of human-computer interactive collaborative control. Therefore, it is necessary to propose a training system based on multimodal human-computer interactive collaborative control. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the present invention proposes a training method, system and device based on multi-modal human-computer interactive collaborative control.

[0004] The present invention provides a training system based on multi-modal human-computer interactive collaborative control, the system comprising:

[0005] A user information management module, used to determine target training items based on user basic information;

[0006] A rehabilitation training selection module, used to determine target training requirements based on the target training items and basic user information;

[0007] A signal acquisition and decoding module, used to determine the mapping relationship between the interactive object control command and the user control action based on the target training requirement;

[0008] The human-computer interaction collaborative control module is used to determine the target interaction control mode based on the target training requirements. When a user control action is received, the interaction object control command corresponding to the received user control action is determined according to the mapping relationship through the target interaction control mode, and the target interaction object is controlled based on the interaction object control command to achieve training.

[0009] In one embodiment, the signal acquisition and decoding module includes a signal selection module:

[0010] The signal selection module is used to determine the target signal according to the target training requirement, determine the corresponding target sensor based on the target signal, and calibrate the target sensor.

[0011] In one embodiment, the signal acquisition and decoding module includes a model selection module and a decoding control module, and the determining of the mapping relationship between the interactive object control command and the user control action includes:

[0012] The model selection module is used to output interactive object control commands according to the target training requirements;

[0013] The decoding control module is used to set a user control action for the interactive object control command and determine a mapping relationship between the interactive object control command and the user control action.

[0014] In one embodiment, the signal acquisition and decoding module further includes a model training module:

[0015] The model training module is used to train the custom model and obtain interactive object control commands.

[0016] In one of the embodiments, determining a target interaction control mode based on the target training requirement, determining an interaction object control command corresponding to the received user control action according to the mapping relationship through the target interaction control mode, and controlling the target interaction object based on the interaction object control command to achieve training, includes:

[0017] Determining a target interaction object based on the target training requirement, and determining a target interaction control mode based on a target signal determined by the target training requirement;

[0018] Sending an interactive object control command to the target interactive object;

[0019] Based on the target interaction control mode and according to the user control action, when preset conditions are met, the target interaction object is controlled based on the interaction object control command to achieve training.

[0020] In one of the embodiments, the human-computer interaction collaborative control module includes at least one of the following: a visual positioning interaction control module, a tactile perception interaction control module, a speech recognition interaction control module, a limb trajectory perception interaction control module, a brain-computer interaction control module, and an electromyography interaction control module.

[0021] In one of the embodiments, the user information management module is further used to manage historical feedback reports based on the user basic information.

[0022] In one embodiment, the system further includes a rehabilitation training and feedback module, wherein the rehabilitation training and feedback module is configured to output at least one of the following:

[0023] A visual training interface, used to output at least one of a real-time rehabilitation training signal graph and a control interaction object signal graph;

[0024] A visual training report, including at least one of a signal graph indicator and a signal change trend;

[0025] Visual training feedback interface, used to provide feedback on training effects based on preset target results;

[0026] Visual feedback report, including visualization content obtained by analyzing the visualization training report;

[0027] The historical feedback report includes a historical visualization training report and a historical visualization feedback report of the target training requirement corresponding to the basic information of the user.

[0028] The present invention provides a training method based on multimodal human-computer interactive collaborative control, the method comprising:

[0029] Determine target training items based on user basic information;

[0030] Determine target training requirements based on the target training items and basic user information;

[0031] Determine a mapping relationship between interactive object control commands and user control actions based on the target training requirements;

[0032] The target interaction control method is determined based on the target training requirements. When a user control action is received, the interaction object control command corresponding to the received user control action is determined according to the mapping relationship through the target interaction control method, and the target interaction object is controlled based on the interaction object control command to achieve training.

[0033] The present invention provides an electronic device, comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the steps of the method described in any one of the above-mentioned embodiments.

[0034] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0035] The present invention provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to execute the steps of the method described in any one of the above embodiments.

[0036] In the above embodiment, first, the user information management module determines the target training items based on the user's basic information, and can provide the user with a customized training plan, thereby improving the accuracy and pertinence of the training. Next, the rehabilitation training selection module determines the target training requirements based on the target training items and the user's basic information. Then, the signal acquisition and decoding module determines the mapping relationship between the interactive object control command and the user control action based on the target training requirements, thereby realizing the interaction between the target interactive object and the user. Finally, the human-computer interaction collaborative control module determines the target interactive control mode based on the target training requirements. When the user control action is received, the interactive object control command corresponding to the received user control action is determined according to the mapping relationship through the target interactive control mode, and the target interactive object is controlled based on the interactive object control command to realize training. The above system integrates multimodal human-computer interaction methods to make the training process more intelligent and personalized, thereby realizing different target training requirements.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a training system based on multimodal human-computer interactive collaborative control provided according to an embodiment of this specification;

[0039] Figure 2 A schematic diagram of a signal acquisition and decoding module provided according to an embodiment of this specification;

[0040] Figure 3 A schematic diagram of a signal acquisition and decoding module provided according to an embodiment of this specification;

[0041] Figure 4 A schematic diagram of a flow chart of controlling a target interactive object to implement training provided in an embodiment of this specification;

[0042] Figure 5 A flowchart of a training method based on multimodal human-computer interactive collaborative control provided in an embodiment of this specification;

[0043] Figure 6 A schematic diagram of a training system based on multimodal human-computer interactive collaborative control provided in an embodiment of this specification;

[0044] Figure 7 This is a diagram of the internal structure of an electronic device provided in accordance with an embodiment of this specification. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0046] In the related technology, the multi-muscle collaborative myoelectric feedback rehabilitation training system may include: a human-computer interaction module, which is used to guide users to perform human-computer interaction tasks; a signal acquisition module, which is used to collect mechanical signals and electromyographic signals generated by the target muscle during the contraction action; a signal analysis and processing module, which extracts the characteristic parameters of the signal collected by the signal acquisition module, obtains and transmits the control signal to the human-computer interaction module for corresponding training control; a trajectory and parameter setting module, which sets various parameters of the human-computer interaction task; a muscle abnormal contraction pattern recognition module, which identifies the abnormal muscle contraction pattern during rehabilitation training and gives an alarm; a signal storage module is used to store mechanical signals, electromyographic signals and the characteristic parameters of both. In addition, a multimodal human-computer interaction collaborative control method based on electromyographic signals and visual sensing is also provided. This method helps to improve rehabilitation efficiency by training the coordination of multiple muscles in real time and continuously, and combining biofeedback.

[0047] In the related technology, first, the surface electromyography signal data of the human hand movement and the image data of the human arm movement are collected by the surface electromyography sensor and the visual sensor respectively. Then, the surface electromyography signal data is filtered and preprocessed, and the missing data of the key point data of the arm bone obtained by the recognition is filled and the sliding window analysis is performed. On this basis, the fusion feature is extracted to obtain the recognition result of the arm movement. Finally, the hand movement recognition result and the arm movement recognition result are fused as the control instruction of the real-time movement of the robot, effectively realizing the human-machine gesture interaction control and coordinated movement. This method can meet the real-time processing of signals from two sensors and realize the effective recognition of complex gestures, which helps to improve the accuracy of human-machine interaction control and coordinated movement.

[0048] In the related technology, a mathematical model of a multi-agent system and a virtual leader is established, and the agents are divided into human-controlled agents and autonomous agents. Then, the error system is obtained based on the model, and a state feedback controller is designed to make the error of the multi-agent system tend to zero and achieve hysteresis consistency. In addition, by constructing the Lyapunov functional, a criterion for judging hysteresis consistency is established. This method reduces system errors by introducing human intervention, and improves the reliability and consistency of the multi-agent system in a complex environment.

[0049] In the related technology, the manipulator human-machine safety collaborative control system based on tactile sensing may include: a human-machine communication module, a command parsing module, a visual positioning module, a tactile perception module and a manipulator control module. Among them, the manipulator control module controls the manipulator to grab the target object according to the instructions sent by the user through the man-machine communication module and the position coordinate information of the target object sent by the visual positioning module. When the manipulator encounters an obstacle, the tactile perception module collects the tactile data of the obstacle and generates a tactile information image, which is then sent to the manipulator control module for obstacle avoidance. By collecting tactile information of the manipulator in contact with the external environment, the defect that machine vision is easily affected by environmental conditions in actual application can be compensated, and the efficiency of the manipulator in performing complex tasks can be improved.

[0050] However, related technologies all rely on a single signal for recognition or training, failing to fully utilize the advantages of human-computer interactive collaborative control and its role in training.

[0051] Therefore, it is necessary to propose a training system based on multimodal human-computer interactive collaborative control that can meet the needs of different trainings, effectively realize human-computer interactive control, and improve the training effect.

[0052] This specification provides a training system based on multi-modal human-computer interactive collaborative control. Figure 1 The training system 100 based on multi-modal human-computer interaction collaborative control includes a user information management module 110, a rehabilitation training selection module 120, a signal acquisition and decoding module 130 and a human-computer interaction collaborative control module 140. Among them:

[0053] The user information management module 110 is used to determine the target training item based on the user's basic information.

[0054] The rehabilitation training selection module 120 is used to determine the target training requirements according to the target training items and the basic information of the user.

[0055] The signal acquisition and decoding module 130 is used to determine the mapping relationship between the interactive object control command and the user control action based on the target training requirement.

[0056] The human-computer interaction collaborative control module 140 is used to determine the target interaction control method based on the target training requirements. When a user control action is received, the interaction object control command corresponding to the received user control action is determined according to the mapping relationship through the target interaction control method, and the target interaction object is controlled based on the interaction object control command to achieve training.

[0057] The interactive object control command may be used to control the behavior of the target interactive object. The user control action may be an action that the user needs to perform.

[0058] In this embodiment, in the user information management module 110, the user's basic information is determined by the user input or obtaining the user's detailed information from the system. Then, the user information management module 110 analyzes the user's basic information to determine the items that the user needs to train, thereby determining the target training items. Among them, the user's basic information may include personal identity information (such as name, gender, age, etc.), health information (such as past medical history, current symptoms, physical condition, etc.), and psychological state (such as stress level, mood swings, etc.).

[0059] For example, by analyzing the basic information of the user and determining that the user has inattention, it is determined that the target training item can be an attention ability training item.

[0060] In this embodiment, the rehabilitation training selection module 120 further analyzes the target training item according to the basic information of the user and selects the most appropriate detailed training requirement from the multiple sub-training items included in the target training item, and determines the selected sub-training item as the target training requirement. Exemplarily, the rehabilitation training selection module 120 may include at least one training item of prosthetic multi-finger coordination training item, arm muscle activity training item, attention ability training item, and cognitive ability training item.

[0061] In this embodiment, the signal acquisition and decoding module 130 uses a machine learning algorithm to determine the interactive object control command in combination with the target training requirements. Then, the signal acquisition and decoding module 130 constructs a mapping relationship between the interactive object control command and the user control action to subsequently realize the control of the target interactive object.

[0062] In this embodiment, the human-computer interaction collaborative control module 140 includes multiple interactive control modes, such as voice recognition, gesture recognition, touch input, eye tracking, etc. According to different target training requirements, the default interactive control mode will be automatically selected, and it can also be customized according to the requirements.

[0063] One or more default interactive control objects are automatically selected for different target training requirements. For example, for training content (such as images or objects for guidance) presented on the screen, interactive control objects related to the target training requirements are preset.

[0064] The human-computer interaction collaborative control module 140 determines the target interaction control mode by analyzing the target training requirements. When the user performs an operation according to the user control action, it is determined whether the data obtained through the target interaction control mode meets the execution conditions of the control target. When the execution conditions are met, the target interaction object controls the target interaction object according to the control command of the interaction object that has a mapping relationship with the user control action, so as to achieve training.

[0065] In the above implementation, first, the user information management module determines the target training items based on the user's basic information, and can provide the user with a customized training plan, thereby improving the accuracy and pertinence of the training. Next, the rehabilitation training selection module determines the target training requirements based on the target training items and the user's basic information. Then, the signal acquisition and decoding module determines the mapping relationship between the interactive object control command and the user control action based on the target training requirements, thereby realizing the interaction between the target interactive object and the user. Finally, the human-computer interaction collaborative control module determines the target interactive control method based on the target training requirements. When the user control action is received, the interactive object control command corresponding to the received user control action is determined according to the mapping relationship through the target interactive control method, and the target interactive object is controlled based on the interactive object control command to realize training. The above system integrates multimodal human-computer interaction methods to make the training process more intelligent and personalized, thereby realizing different target training requirements.

[0066] In some embodiments, see Figure 2 The signal acquisition and decoding module 130 includes a signal selection module 210. Among them:

[0067] The signal selection module 210 is used to determine a target signal according to a target training requirement, determine a corresponding target sensor based on the target signal, and calibrate the target sensor.

[0068] In this embodiment, the signal selection module 210 determines the required signal and determines it as the target signal according to the target training requirements, so as to perform human-machine collaborative control and provide a basis for the subsequent target sensor selection. After determining the target signal, the signal selection module 210 further determines the target sensor that matches it to ensure that the target sensor can capture the target signal stably and accurately. After selecting the target sensor, the signal selection module 210 will also perform necessary calibration on the target sensor. For example, if the target sensor is a wearable device, if the target sensor is not worn accurately, it will detect whether the sensor is in good contact with the body to ensure that the signal collection is not interfered with. If the sensor is not worn correctly, it may cause the signal to fail to capture normally or capture an unreliable signal, and the user will be prompted to readjust the position to achieve calibration so that the target sensor is worn in the correct position.

[0069] Exemplarily, different types of signals can be selected as target signals according to target training requirements. For example, a unimodal signal can be selected as the target signal, such as an electromyographic signal or an electroencephalographic signal. A multimodal signal can also be selected as the target signal, such as an electromyographic signal and an electroencephalographic signal. The default target signal will be matched according to different target training requirements. Each signal has its specific characteristics and application scenarios, which can better support different target training requirements. In addition to the default settings, users can also flexibly select and customize signal combinations according to their personal target training requirements to achieve the best training effect.

[0070] In the above implementation, the signal selection module determines the target signal according to the target training requirements, determines the corresponding target sensor based on the target signal, and calibrates the target sensor to ensure the reliability of subsequent training.

[0071] It should be noted that each target training requirement has a set of default configuration schemes, which can also be customized as needed. During the customization process, due to different requirements for signals for different training requirements, not all signals are applicable to the selected training requirements, and unmatched signals cannot be selected. Specifically, there is a close interactive relationship between the signal selection module 210 and the rehabilitation training selection module 120, and the user selects each other through these two modules to complete the personalized customization of the training requirements and signal configuration, avoiding the mismatch problem between the signal and the training project. In some embodiments, the target training requirement can be selected in the rehabilitation training selection module 120, and then the available signals matching the target training requirement will be displayed in the signal selection module 210 according to the target training requirement, and the user can select a suitable signal from it as the target signal. In other embodiments, the desired target signal can be selected in the signal selection module 210, and then the training requirements applicable to the signal will be displayed in the rehabilitation training selection module 120 according to the target signal, so that the user can select a suitable training project as the target training requirement. In addition, the decoding control module 230 also needs to be configured and adjusted accordingly according to the user's custom settings in the signal selection module and the rehabilitation training selection module. In this process, the model training module 310 needs to be used for model training so that the output of the model can be mapped in the decoding control module 230.

[0072] In some embodiments, see Figure 2 The signal acquisition and decoding module 130 includes a model selection module 220 and a decoding control module 230, and determining the mapping relationship between the interactive object control command and the user control action may include:

[0073] The model selection module 220 is used to output interactive object control commands according to target training requirements.

[0074] The decoding control module 230 is used to set a user control action for the interactive object control command and determine a mapping relationship between the interactive object control command and the user control action.

[0075] In this embodiment, the model selection module 220 selects a model corresponding to the target training requirement from a pre-trained model library according to the target training requirement. Then the model corresponding to the target training requirement outputs the interactive object control command according to the target signal. In addition, the model selection module 220 can also select a custom training model according to the target training requirement.

[0076] In this embodiment, the decoding control module 230 is preset with a default corresponding relationship, so the decoding control module 230 is responsible for setting a corresponding user control action for each control command and determining the mapping relationship between the interactive object control command and the user control action. In addition, the decoding control module 230 can also customize the mapping relationship between the interactive object control command and the user control action.

[0077] Exemplarily, the model selection module outputs interactive object control commands of clenching a fist and opening a palm according to the prosthetic training requirements. The decoding control module sets a user control action corresponding to the muscle group activity for the interactive object control command of clenching a fist, and sets a user control action corresponding to the muscle group activity for the interactive object control command of opening a palm.

[0078] In the above implementation, the model selection module outputs the interactive object control command according to the target training requirements, the decoding control module sets the user control action for the interactive object control command, and determines the mapping relationship between the interactive object control command and the user control action, providing a basis for the subsequent training of the target interactive object.

[0079] In some embodiments, see Figure 3 The signal acquisition and decoding module 130 further includes a model training module 310, wherein:

[0080] The model training module 310 is used to train the custom model and obtain interactive object control commands.

[0081] In this embodiment, after selecting a custom training model, the model training module 310 is responsible for training a custom model by selecting sample data, annotated data, and selecting an appropriate algorithm, thereby generating a new model for outputting interactive object control commands. Specifically, first, an algorithm for training needs to be selected. The choice of algorithm should be determined according to the data type and task objectives. Secondly, select a suitable training set, validation set, and test set. Usually, the data set is divided according to a certain ratio (such as 70% of the data is used for training, 15% for validation, and 15% for testing). When selecting sample data, it must be ensured that the data record contains fragment data of the target signal, wherein the sample data can be raw data or preprocessed (such as denoising, standardization, or normalization) data. Each sample data is labeled with a corresponding category label. Then the custom model is trained, and the algorithm selected for training will learn based on the labels corresponding to the training set and the sample data in the training set to optimize the model parameters. At this time, the model effect can be improved by hyperparameter tuning (such as learning rate, batch size, network structure, etc.). Then use the validation set to evaluate the effect of the model. After training is completed, the model is finally evaluated using the test set to obtain indicators such as model accuracy, F1 score, precision, and recall. If the evaluation results do not meet the expected effect, further optimization can be performed. Based on the evaluation results, the model with the best performance after training is selected for deployment, and the interactive object control commands corresponding to the target training requirements are output.

[0082] In the above implementation, the model training module is used to train the custom model to obtain interactive object control commands. Through customized training, the model can intelligently generate control commands, thereby enhancing the adaptability and scalability of the system.

[0083] In some embodiments, see Figure 4 , determining a target interaction control mode based on a target training requirement, determining an interaction object control command corresponding to the received user control action according to a mapping relationship through the target interaction control mode, and controlling the target interaction object based on the interaction object control command to implement training, including:

[0084] S410: determining a target interaction object based on a target training requirement, and determining a target interaction control mode based on a target signal determined based on the target training requirement.

[0085] S420: Send an interactive object control command to the target interactive object.

[0086] S430: Based on the target interaction control mode, according to the user control action, when preset conditions are met, the target interaction object is controlled based on the interaction object control command to achieve training.

[0087] Specifically, the human-computer interaction collaborative control module 140 first needs to select a suitable target interaction object according to the target training requirements. Then, the human-computer interaction collaborative control module 140 will further clarify the target interaction control method of the target interaction object according to the target signal determined by the target training requirements. After the target interaction object and the target interaction control method are determined, the human-computer interaction collaborative control module 140 will issue the interaction object control command to the target interaction object in sequence according to the mapping relationship between the interaction object control command and the user control action determined by the signal acquisition and decoding module 130, so that the target system performs the corresponding action. Prompt according to the user control action to enable the user to perform the user control action. In the process of executing the user control action, based on the target interaction control method, the sensor data is acquired and analyzed in real time through the target sensor. When the data meets the preset conditions, the target interaction object will implement the corresponding action according to the received interaction object control command, realizing real-time feedback and control of human-computer interaction, so as to realize training and ensure the smooth progress of the target training requirements.

[0088] Exemplarily, the human-computer interaction collaborative control module determines the electromyographic interaction control module according to the determined electromyographic signal. The target interaction object is determined to be a prosthesis according to the prosthesis training requirements. Then, the human-computer interaction collaborative control module sends a fist-clenching interactive object control command to the prosthesis, and the prosthesis determines its corresponding preset information based on the received fist-clenching interactive object control command, such as performing a fist-clenching operation under electromyographic threshold conditions. The user control action that the user needs to perform is displayed on the display or a voice prompt is given to the user, and the user performs the action according to the prompt. In the process of the user performing the action, the electromyographic information is obtained and analyzed in real time through the electromyographic sensor. When the electromyographic information reaches the electromyographic threshold condition, the prosthesis is controlled to perform the fist-clenching operation based on the fist-clenching interactive object control command to achieve training.

[0089] In the above implementation, the target interactive object is determined based on the target training requirements, and the target interactive control method is determined based on the target signal determined based on the target training requirements, and an interactive object control command is sent to the target interactive object. Based on the target interactive control method, according to the user control action pair, when the preset conditions are met, the target interactive object is controlled based on the interactive object control command to achieve training, so that the user can achieve the purpose of training according to the target training requirements.

[0090] In some embodiments, the human-computer interaction collaborative control module includes at least one of the following: a visual positioning interaction control module, a tactile perception interaction control module, a speech recognition interaction control module, a limb trajectory perception interaction control module, a brain-computer interaction control module, and an electromyography interaction control module.

[0091] In this embodiment, the human-computer interaction collaborative control module integrates a variety of interaction modes and perception technologies, so that the system can realize the selection of more intelligent and personalized interaction control modules according to different target training requirements or target signals. The human-computer interaction collaborative control module includes at least one of the following:

[0092] The visual positioning interactive control module can capture the user's movements in real time through a camera or other visual sensors, and accurately perceive the user's posture and movement intentions.

[0093] The tactile perception interaction control module can allow users to perceive the actual touch, pressure and texture changes of virtual objects or interfaces through tactile feedback devices.

[0094] The voice recognition interaction control module can use voice recognition technology to convert the user's voice instructions into commands that the machine can understand, realizing two-way interaction between voice and the system.

[0095] The limb trajectory perception interactive control module can track the user's limb movements and trajectories in real time through sensors or motion capture technology, accurately capture the speed, direction, amplitude and other information of the movement, thereby converting the user's limb behavior into system operation commands to achieve precise motion control and operation.

[0096] The brain-computer interaction control module can use brain waves or electroencephalogram (EEG) technology to capture and analyze the user's brain waves, identify the user's intentions or emotional state, and then achieve direct control with computers or other devices.

[0097] The myoelectric interaction control module can sense the electrical activity of the user's muscles through electromyography (EMG) sensors, monitor the user's muscle signals in real time, and convert them into control commands.

[0098] In the above embodiment, the human-computer interaction collaborative control module includes at least one of a visual positioning interaction control module, a tactile perception interaction control module, a speech recognition interaction control module, a limb trajectory perception interaction control module, a brain-computer interaction control module, and an electromyography interaction control module, so as to provide a flexible and efficient interaction method according to different target signals.

[0099] In some implementations, the user information management module is further used to manage historical feedback reports based on basic user information.

[0100] In this embodiment, the user information management module can also determine the historical feedback reports of all the trainings the user has participated in based on the user's basic information. The user can manage the historical feedback reports of all the trainings the user has participated in through the user information management module, for example, query the historical feedback reports of a specified training.

[0101] In the above implementation, the user information management module is also used to manage historical feedback reports based on the user's basic information, thereby improving work efficiency and query convenience.

[0102] In some embodiments, the system further includes a rehabilitation training and feedback module, which is used to output at least one of a visual training interface, a visual training report, a visual training feedback interface, a visual feedback report, and a historical feedback report.

[0103] In this embodiment, the visual training interface is designed to display real-time rehabilitation training data, and can present various rehabilitation training signal graphs or control interaction object signal graphs generated during the rehabilitation training process in real time. Among them, the rehabilitation training signal graph reflects the data obtained by the target sensor in real time during the user's rehabilitation process. The control interaction object signal graph shows the signal of the target interaction object that interacts with the user. Through the real-time display of the rehabilitation training signal graph or the control interaction object signal graph, the user can understand the effect and progress of the current training in real time.

[0104] In this embodiment, after the training is completed, a detailed visual training report will be generated and displayed. The signal graph indicator or signal change trend is determined based on the rehabilitation training signal graph and the control interaction object signal graph. The visual training report includes at least one of the signal graph indicators or signal change trends. The signal graph indicator will display the indicators of the rehabilitation training signal graph or the control interaction object signal graph to help users intuitively understand the effect of this training. The signal change trend will present the signal change trend based on the rehabilitation training signal graph and the control interaction object signal graph during the training process. The user can understand the training situation more clearly and understand whether abnormal fluctuations occur during the training process.

[0105] In this embodiment, the visual training feedback interface helps users understand their progress and effects in real time during the training process through intuitive visual and auditory feedback. Based on the preset target results, real-time effect feedback is provided on the user's actual performance. For example, the user can set specific goals, such as completing a specific task or performing multiple target tasks. Once the user achieves the set goal, the system will motivate and encourage the user through various forms of feedback. For example, the interface can display successful animation effects, dynamic charts or progress bars so that the user can clearly see his or her progress. In addition, positive feedback can also be strengthened through voice prompts. The system's feedback method is highly customizable, and users can customize feedback and reminder methods.

[0106] In this embodiment, the visual feedback report is a visualization content obtained by analyzing the visual training report through a visualization tool, which helps the user to clearly and intuitively understand the performance and progress of the training process, so as to optimize the training effect later. For example, the visual training report is analyzed to determine the achievement of the training goals, the duration of the training, and the degree of achievement of the training effect during the training process, and the obtained data is visualized to form a visual feedback report.

[0107] In this embodiment, the system will automatically generate and display a historical feedback report after the training of this target training requirement is completed, so as to help the user review and analyze the results and progress of past training processes. The historical feedback report may include a historical visualization training report and a historical visualization feedback report. Among them, the historical visualization training report displays the signal graph indicators and signal change trends of each past target training requirement of the user. Through these data, the user can intuitively see the signal fluctuations of each target training requirement. The historical visualization feedback report displays the visualization content of the visualization training report of each past target training requirement of the user for analysis, which is used to determine the content that has changed during the training process through data comparison, so as to adjust the training later.

[0108] In the above embodiment, the rehabilitation training and feedback module is used to output at least one of a visual training interface, a visual training report, a visual training feedback interface, a visual feedback report, and a historical feedback report, providing intuitive and detailed training data and feedback information to help users monitor and evaluate rehabilitation progress in real time, and then adjust the rehabilitation training plan to achieve the best rehabilitation effect.

[0109] This specification provides a training method based on multi-modal human-computer interactive collaborative control. Figure 5 , the training method based on multimodal human-computer interactive collaborative control can include the following steps:

[0110] S510. Determine target training items based on basic information of the user.

[0111] Specifically, in the user information management module, the user's basic information is determined by the user input or obtaining the user's detailed information from the system. Then, the user information management module analyzes the user's basic information to determine the items that the user needs to train, thereby determining the target training items.

[0112] S520: Determine target training requirements based on target training items and user basic information.

[0113] Specifically, in the target training item, the rehabilitation training selection module further analyzes the basic information of the user and selects the most appropriate detailed training requirement from multiple sub-training items included in the target training item, and determines the selected sub-training item as the target training requirement.

[0114] S530: Determine a mapping relationship between interactive object control commands and user control actions based on target training requirements.

[0115] Specifically, the signal acquisition and decoding module combines the target training requirements and uses a machine learning algorithm to determine the interactive object control command. Then, the signal acquisition and decoding module constructs a mapping relationship between the interactive object control command and the user control action, thereby realizing the control of the target interactive object.

[0116] S540. Determine the target interaction control method based on the target training requirements. When a user control action is received, determine the interaction object control command corresponding to the received user control action according to the mapping relationship through the target interaction control method, and control the target interaction object based on the interaction object control command to achieve training.

[0117] Specifically, the human-computer interaction collaborative control module determines the target interaction control mode by analyzing the target training requirements. When the user performs an operation according to the user control action, it is determined whether the data obtained through the target interaction control mode meets the execution conditions of the control target. If the conditions are met, the target interaction object controls the target interaction object according to the control command of the interaction object that has a mapping relationship with the user control action to achieve training.

[0118] In the above implementation, first, the user information management module determines the target training items based on the user's basic information, and can provide the user with a customized training plan, thereby improving the accuracy and pertinence of the training. Next, the rehabilitation training selection module determines the target training requirements based on the target training items and the user's basic information. Then, the signal acquisition and decoding module determines the mapping relationship between the interactive object control command and the user control action based on the target training requirements, thereby realizing the interaction between the target interactive object and the user. Finally, the human-computer interaction collaborative control module determines the target interactive control method based on the target training requirements. When the user control action is received, the interactive object control command corresponding to the received user control action is determined according to the mapping relationship through the target interactive control method, and the target interactive object is controlled based on the interactive object control command to realize training. The above system integrates multimodal human-computer interaction methods to make the training process more intelligent and personalized, thereby realizing different target training requirements.

[0119] The present specification also provides a training system based on multi-modal human-computer interactive collaborative control, see Figure 6The system includes a user information management module, rehabilitation training selection items, a signal acquisition and decoding module, a human-computer interaction collaborative control module, and a rehabilitation training and feedback module. In the user information management module, basic user information can be entered and user training item reports can be managed. The rehabilitation training selection module includes prosthetic multi-finger collaborative training items, arm muscle activity training items, attention ability training items, and cognitive ability training items. The signal acquisition and decoding module includes a signal selection module, a model selection module, a model training module, and a decoding control module. The human-computer interaction collaborative control module includes a visual positioning interaction control module, a tactile perception interaction control module, a speech recognition interaction control module, a limb trajectory perception interaction control module, a brain-computer interaction control module, and an electromyography interaction control module. The rehabilitation training and feedback module includes a visual training interface, a visual training report, a visual training feedback interface, a visual feedback report, and a personal history training and feedback report.

[0120] It should be noted that before the training begins, since the electromyographic signal itself is a very weak signal, the user must first clean the skin surface (remove body hair, wipe with 75% alcohol and wait for it to dry), and avoid collecting electromyographic activity in thicker fat tissue and near the heart area to reduce interference from sweat, stratum corneum, fat layer and other factors during the collection process.

[0121] Wireless electromyographic sensors are used to collect electromyographic signals from the subjects' forearms. Each signal collection action includes a series of standard gestures, such as "pinch" and "clench a fist". Each action is maintained for about 2 seconds and repeated in a relaxed state (each action is repeated 50 times). The arm needs to be suspended in the air and relaxed during the collection process to ensure the accuracy of the signal. Through the multimodal data analysis platform, the time domain, frequency domain and time-frequency domain features of the electromyographic signal, such as mean, standard deviation, spectral distribution, etc., are extracted to identify the electromyographic activity pattern associated with a specific gesture. The extracted features are input into the support vector machine (SVM) classifier for training. The model improves the accuracy by optimizing hyperparameters and enhancing data, and finally completes the pattern recognition of different hand movements (such as grasping, releasing, pinching, etc.), thereby realizing model training.

[0122] Specifically, in the process of implementing training, first, the user's basic information is input into the user information management module, and then the target training item is determined to be the prosthesis multi-finger collaborative control training item based on the user's basic information. In the rehabilitation training selection module, the target training requirement is determined to be the prosthesis multi-finger collaborative control training item. The signal selection module determines that the target signal is an electromyographic signal according to the prosthesis multi-finger collaborative control training item, determines the corresponding electromyographic sensor based on the electromyographic signal, and calibrates the electromyographic sensor. Then the model obtained by the above training is selected, and the interactive object control command is output according to the prosthesis multi-finger collaborative control training item. The decoding control module sets the user control action for the interactive object control command, and determines the mapping relationship between the interactive object control command and the user control action. The human-computer interactive collaborative control module determines that the target interactive control mode is electromyographic interactive control based on the prosthesis multi-finger collaborative control training item, and connects the matching prosthesis. When the prosthesis is successfully connected, in the case of receiving the user control action, the interactive object control command corresponding to the received user control action is determined according to the mapping relationship through the target interactive control mode, and the prosthesis is controlled based on the interactive object control command to implement training, and feedback is provided in real time.

[0123] The visual training interface can display the changes in electromyographic signals, changes in prosthetic movements, and the activities of related muscle groups in real time. Users can mobilize the corresponding muscle groups according to the activities of the related muscle groups to achieve the purpose of controlling the prosthesis.

[0124] The visual training feedback interface can be customized according to needs. For example, when the user completes the first training, when the muscle activity reaches the level that can control the prosthesis, the interface will display powerful animation feedback; if the user has trained multiple times and can coordinately control multiple muscle groups to achieve grasping movements, encouraging feedback will be provided.

[0125] In some embodiments, an electronic device is provided, whose internal structure diagram can be as follows: Figure 7As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method based on multimodal human-computer interactive collaborative control is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0126] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme disclosed in this specification, and does not constitute a limitation on the electronic device to which the scheme disclosed in this specification is applied. Specifically, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0128] An embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0129] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0130] It should be understood that various parts of this specification can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0132] In the description of this specification, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this specification and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on this specification.

[0133] In addition, the terms "first", "second", etc. used in the embodiments of this specification are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in this embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of this specification can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of this specification, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0134] In this specification, unless otherwise clearly specified or limited in the embodiments, the terms "install", "connect", "connect" and "fix" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this specification can be understood according to the specific implementation situation.

[0135] In this specification, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0136] Although the embodiments of the present specification have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limitations on the present specification. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present specification.

Claims

1. A training system based on multimodal human-computer interactive collaborative control, characterized in that: The system comprises at least: A user information management module, used to determine target training items based on user basic information; A rehabilitation training selection module, used to determine target training requirements based on the target training items and basic user information; A signal acquisition and decoding module, used to determine the mapping relationship between the interactive object control command and the user control action based on the target training requirement; The human-computer interaction collaborative control module is used to determine the target interaction control mode based on the target training requirements. When a user control action is received, the interaction object control command corresponding to the received user control action is determined according to the mapping relationship through the target interaction control mode, and the target interaction object is controlled based on the interaction object control command to achieve training.

2. The system according to claim 1, characterized in that The signal acquisition and decoding module includes a signal selection module: The signal selection module is used to determine the target signal according to the target training requirement, determine the corresponding target sensor based on the target signal, and calibrate the target sensor.

3. The system according to claim 1, characterized in that The signal acquisition and decoding module includes a model selection module and a decoding control module, and the determining of the mapping relationship between the interactive object control command and the user control action includes: The model selection module is used to output interactive object control commands according to the target training requirements; The decoding control module is used to set a user control action for the interactive object control command and determine a mapping relationship between the interactive object control command and the user control action.

4. The system according to claim 1, characterized in that The signal acquisition and decoding module also includes a model training module: The model training module is used to train the custom model and obtain interactive object control commands.

5. The system according to claim 1, characterized in that The step of determining a target interaction control mode based on the target training requirement, determining an interaction object control command corresponding to the received user control action according to the mapping relationship through the target interaction control mode, and controlling the target interaction object based on the interaction object control command to implement training, includes: Determining a target interaction object based on the target training requirement, and determining a target interaction control mode based on a target signal determined by the target training requirement; Sending an interactive object control command to the target interactive object; Based on the target interaction control mode and according to the user control action, when preset conditions are met, the target interaction object is controlled based on the interaction object control command to achieve training.

6. The system according to claim 5, characterized in that The human-computer interaction collaborative control module includes at least one of the following: a visual positioning interaction control module, a tactile perception interaction control module, a speech recognition interaction control module, a limb trajectory perception interaction control module, a brain-computer interaction control module, and an electromyography interaction control module.

7. The system according to claim 1, characterized in that The user information management module is also used to manage historical feedback reports based on the user basic information.

8. The system according to claim 1, characterized in that The system further includes a rehabilitation training and feedback module, which is used to output at least one of the following: A visual training interface, used to output at least one of a real-time rehabilitation training signal graph and a control interaction object signal graph; A visual training report, including at least one of a signal graph indicator and a signal change trend; Visual training feedback interface, used to provide feedback on training effects based on preset target results; Visual feedback report, including visualization content obtained by analyzing the visualization training report; The historical feedback report includes a historical visualization training report and a historical visualization feedback report of the target training requirement corresponding to the basic information of the user.

9. A training method based on multimodal human-computer interactive collaborative control, characterized in that: The method comprises: Determine target training items based on user basic information; Determine target training requirements based on the target training items and basic user information; Determine a mapping relationship between interactive object control commands and user control actions based on the target training requirements; The target interaction control method is determined based on the target training requirements. When a user control action is received, the interaction object control command corresponding to the received user control action is determined according to the mapping relationship through the target interaction control method, and the target interaction object is controlled based on the interaction object control command to achieve training.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images