Artificial limb control method and device, artificial limb control equipment and storage medium

Abstract

The invention provides an artificial limb control method and device, artificial limb control equipment and a storage medium, and belongs to the technical field of biological signal detection.The method comprises the steps that biological signals of a user in the artificial limb using process are obtained, and the biological signals comprise electromyographic signals and voice signals and / or electroencephalogram signals; identifying an action result corresponding to each biological signal by adopting a pre-trained action identification model, wherein the action result comprises a plurality of action types and respective corresponding prediction probabilities; determining a target action according to a plurality of action types and respective corresponding prediction probabilities in each action result; the artificial limb is controlled based on the target action, accurate control over the artificial limb is achieved, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biological signal detection, and in particular to a prosthesis control method and device, a prosthesis control apparatus, and a storage medium. BACKGROUND

[0002] With the acceleration of industrialization and urbanization, unexpected events such as traffic accidents and mechanical accidents occur frequently, resulting in an increasing number of amputees year by year. Arm amputees face many inconveniences in daily life, and installing a prosthesis is an important means to help them restore some functions. Therefore, it is particularly important to improve the intelligent level of the prosthesis to improve the user experience.

[0003] In related technologies, a plurality of commonly used action templates are usually preset for the prosthesis. When a certain action is recognized from the collected electromyographic signals, the preset template is called to complete the corresponding action. That is, the electromyographic signals are pattern-recognized to control the prosthesis to realize the matching of the preset action and the electromyographic signals, thereby controlling the prosthesis.

[0004] However, the electromyographic signals of different users may differ, and the existing technology is difficult to adapt to the individual needs of different users, resulting in unstable recognition accuracy of the pattern recognition of the electromyographic signals and reduced accuracy of the prosthesis control. SUMMARY

[0005] Therefore, it is necessary to provide a prosthesis control method, device, prosthesis control apparatus, and storage medium to solve the technical problem of low prosthesis control accuracy in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a prosthesis control method, comprising: obtaining biological signals of a prosthesis of a user during use of the prosthesis, the biological signals including electromyographic signals, and voice signals and / or electroencephalographic signals; recognizing an action result corresponding to each biological signal by using a pre-trained action recognition model, the action result including a plurality of action types and respective prediction probabilities; determining a target action according to the plurality of action types and respective prediction probabilities in each action result; controlling the prosthesis based on the target action.

[0007] In a possible implementation, the determining of the target action according to the plurality of action types and respective prediction probabilities in each action result comprises: determining a type weight of the action types in the action result according to a signal type of the biological signal corresponding to the action result; For each action type in the action result, a comprehensive probability of each action type is determined according to the type weight and the corresponding prediction probability; A target action is determined according to the comprehensive probability of each action type.

[0008] In a possible implementation, the action recognition model includes a signal type recognition sub-model and three action recognition sub-models in parallel, the signal type recognition sub-model is cascaded with the three action recognition sub-models respectively, and the three action recognition sub-models are respectively corresponding to the action recognition sub-models of the electromyographic signal, the speech signal and the electroencephalogram signal; and the action result corresponding to each biological signal is recognized by using the pre-trained action recognition model, including: The signal type of the biological signal is recognized by the signal type recognition sub-model; According to the signal type, the biological signal of the corresponding signal type is respectively recognized by using the corresponding action recognition sub-model to obtain the action result corresponding to each biological signal.

[0009] In a possible implementation, the action result corresponding to each biological signal is recognized by using the corresponding action recognition sub-model according to the signal type, including: In the case where the signal type is the electromyographic signal, the to-be-recognized electromyographic signal is obtained from the biological signal, and the first feature of the to-be-recognized electromyographic signal is input into the action recognition sub-model corresponding to the electromyographic signal to obtain a plurality of corresponding action types and respective corresponding first prediction probabilities; In the case where the signal type is the electroencephalogram signal, the to-be-recognized electroencephalogram signal is obtained from the biological signal, and the second feature of the to-be-recognized electroencephalogram signal is input into the action recognition sub-model corresponding to the electroencephalogram signal to obtain a plurality of corresponding action types and respective corresponding second prediction probabilities; In the case where the signal type is the speech signal, the to-be-recognized speech signal is obtained from the biological signal, and the third feature of the to-be-recognized speech signal is input into the action recognition sub-model corresponding to the speech signal to obtain a plurality of corresponding action types and respective corresponding third prediction probabilities.

[0010] In a possible implementation, the comprehensive probability of each action type is determined according to the type weight and the corresponding prediction probability for each action type in the action result, including: The type weight and the corresponding prediction probability are respectively weighted and summed to obtain the comprehensive probability.

[0011] In a possible implementation, the control of the artificial limb based on the target action includes: In a case where the biological signal includes a speech signal, the speech signal to be recognized is acquired; An adjustment parameter corresponding to the target action is determined according to the speech signal to be recognized; The prosthesis is controlled to perform the target action according to the adjustment parameter.

[0012] In a possible implementation, the type weight of the action type in the action result is determined according to the signal type of the biological signal corresponding to the action result, including: If the biological signal includes a speech signal and an electromyography signal, a first type weight corresponding to the speech signal is determined to be greater than a third type weight corresponding to the electromyography signal; If the biological signal includes an electroencephalogram signal and an electromyography signal, a third type weight corresponding to the electromyography signal is determined to be greater than a second type weight corresponding to the electroencephalogram signal; If the biological signal includes an electroencephalogram signal, an electroencephalogram signal and an electromyography signal, a first type weight corresponding to the speech signal is determined to be greater than a third type weight corresponding to the electromyography signal, and the third type weight corresponding to the electromyography signal is determined to be greater than a second type weight corresponding to the electroencephalogram signal.

[0013] In a second aspect, the present application further provides a prosthesis control device, including: An acquisition unit is configured to acquire a biological signal of a user in the process of using a prosthesis, the biological signal including an electromyography signal, a speech signal and / or an electroencephalogram signal; An identification unit is configured to identify an action result corresponding to each biological signal by using a pre-trained action recognition model, the action result including a plurality of action types and respective prediction probabilities; A determination unit is configured to determine a target action according to the plurality of action types and respective prediction probabilities in each action result; A control unit is configured to control the prosthesis based on the target action.

[0014] In a third aspect, the present application further provides a prosthesis control device, including a memory and a processor, wherein the memory is configured to store a program; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the prosthesis control method in any of the above implementation manners.

[0015] In a fourth aspect, the present application further provides a computer readable storage medium for storing a computer readable program or instruction, which can implement the steps of the prosthesis control method in any of the above implementation manners when executed by a processor.

[0016] The present application has the following beneficial effects: The prosthesis control method provided by this invention acquires biosignals from the user during prosthesis use, including electromyography (EMG) signals, speech signals, and / or electroencephalogram (EEG) signals. This ensures that the biosignals used for prosthesis control include at least EMG signals, providing direct, real-time, and rich information for precise prosthesis control. Furthermore, combining EMG signals with speech and / or EEG signals allows for better adaptation to the personalized needs of different users due to the complementarity of different signals. A pre-trained action recognition model identifies the action result corresponding to each biosignal. The action result includes multiple action types and their corresponding predicted probabilities. Identifying multiple action types and their predicted probabilities provides more candidate options for action recognition results, facilitating subsequent quantitative analysis based on the predicted probabilities of each action type and improving the accuracy of action prediction. Based on the multiple action types and their corresponding predicted probabilities in each action result, a target action is determined, enabling a more comprehensive and accurate analysis of the user's movement intention and ensuring the accuracy of the target action. Controlling the prosthesis based on the target action improves the precision of prosthesis control and enhances the user experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of an embodiment of the prosthetic limb control method provided by the present invention; Figure 2 This is a schematic diagram of the signal acquisition unit of the present invention; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S103; Figure 4 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 5 For the present invention Figure 4 A schematic diagram of an embodiment of S302; Figure 6 For the present invention Figure 1 A schematic diagram of an embodiment of S104; Figure 7 For the present invention Figure 3 A schematic flowchart of an embodiment of S201 Figure 8An embodiment structure schematic diagram of a prosthesis control device provided by the present application is shown in the figure. Figure 9 An embodiment structure schematic diagram of a prosthesis control device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0020] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. The association relationship of the associated objects is described by "and / or", which means that there can be three relationships, for example: A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone.

[0021] The "first", "second", and the like described in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the technical features limited by "first" and "second" can explicitly or implicitly include at least one of the features.

[0022] In this document, referring to "embodiments" means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The execution subject of the prosthesis control method in the embodiments of the present application can be the prosthesis control device provided by the present application, or a server device, a physical host or a user equipment (User Equipment, UE) integrated with the prosthesis control device, and the like. The prosthesis control device can be realized in hardware or software, and the UE can be a terminal device such as a smart phone, a tablet computer, a notebook computer, a palm computer, a desktop computer or a personal digital assistant (Personal Digital Assistant, PDA).

[0024] The present application provides a prosthesis control method, device, prosthesis control equipment and storage medium, which are described below.

[0025] Figure 1 An embodiment flowchart of the prosthesis control method provided by the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the prosthesis control method comprises: S101, acquiring biological signals of a user in the process of using a prosthesis, the biological signals comprising electromyographic signals, and speech signals and / or electroencephalographic signals.

[0026] In the present application, the prosthesis can be an upper limb or a lower limb.

[0027] The electromyographic signals refer to electrical signals generated when muscles contract, and are a direct manifestation of brain movement instructions, and can directly reflect the movement intention of the user. In the present embodiment, the biological signals used for prosthesis control at least include electromyographic signals, which can provide direct, real-time and rich information for precise control of the prosthesis, so as to improve the accuracy of action recognition and user experience. Moreover, the biological signals at least include two types, i.e., the electromyographic signals are combined with the speech signals and / or the electroencephalographic signals, so as to realize comprehensive analysis of multi-modal biological signals, and further improve the accuracy of action recognition and user experience.

[0028] Specifically, the electromyographic signals of the user at the prosthesis connection can be collected by an electromyographic signal sensor, the speech signals can be collected by a speech detection device such as a microphone, and the electroencephalographic signals can be detected by an electroencephalographic electrode, so as to acquire the biological signals. In the present embodiment, the biological signals of the user in the process of using the prosthesis include electromyographic signals, and speech signals and / or electroencephalographic signals, i.e., the biological signals used for prosthesis control at least include electromyographic signals, which can provide direct, real-time and rich information for precise control of the prosthesis, so as to improve the accuracy of action recognition and user experience. Moreover, the biological signals at least include two types, i.e., the electromyographic signals are combined with the speech signals and / or the electroencephalographic signals, which enriches the signal sources, improves the stability of the signals and the accuracy of recognition, and the complementarity of different signals can better adapt to the individual needs of different users. As shown in Table 1, a table of three modes of biological signals in the present embodiment, the in the table, indicates that the biological signals include the signal, and the in the table indicates that the biological signals do not include the signal.

[0029] Table 1, a table of three modes of biological signals

[0030] It should be noted that for a single signal type of biological signals, it can be used for device inspection, device calibration, device debugging and user data acquisition; in the scene of prosthesis control, according to the modes of a, b and c in Table 1, the biological signals of the user are detected and action prediction is made, and then the prosthesis is controlled to perform corresponding actions.

[0031] In one specific embodiment, the prosthesis is taken as an upper limb prosthesis for example, as shown in FIG. 2. Figure 2As shown, it is a structural schematic diagram of the signal acquisition unit 100. The signal acquisition unit 100 can include an electromyography signal acquisition module 110, a voice signal acquisition module 120, and an electroencephalogram signal acquisition module 130.

[0032] The electromyography signal acquisition module 110 detects electromyography signals through a multi-channel electromyography signal sensor. The electromyography signal sensor is placed on a specific muscle group of the user's amputated limb, which can capture useful electromyography signals to the greatest extent. It should be noted that the electromyography signal sensor is in direct contact with the skin without the need for additional conductive materials.

[0033] The voice signal acquisition module 120 detects voice signals through a microphone or the like. The microphone is placed at the uppermost end of the prosthesis close to the user's mouth, which facilitates clear reception of voice signals from the user in various environments.

[0034] The electroencephalogram signal acquisition module 130 detects electroencephalogram signals through an electroencephalogram electrode or the like. The electroencephalogram electrode is placed at a specific position on the user's scalp according to a pre-designed layout, which ensures that useful electroencephalogram signals can be captured to the greatest extent. It should be noted that, in order to improve signal stability, a conductive material needs to be applied between the electroencephalogram electrode and the scalp to ensure good electrical contact.

[0035] S102, a pre-trained action recognition model is used to identify the action result corresponding to each biological signal, the action result including a plurality of action types and respective corresponding prediction probabilities.

[0036] The action result includes a plurality of action types and respective corresponding prediction probabilities, wherein each action type corresponds to an action. For example, clenched fist, stretched palm, grabbing, etc. The prediction probability refers to the probability that the action result is judged as an action type, such as the prediction probability of the clenched fist action being 60%.

[0037] The pre-trained action recognition model is a pre-trained model for pattern recognition of input biological signals to output action results. It can be trained based on at least one of random forest, gradient boosting machine, support vector machine, K-nearest neighbor, neural network, and naive Bayes, and has obtained a plurality of action types and corresponding prediction probabilities.

[0038] Specifically, the pre-trained action recognition model is used to identify a plurality of action types corresponding to each biological signal and respective corresponding prediction probabilities. It can be understood that, in this embodiment, a plurality of action types and their prediction probabilities are identified, which can provide more candidate options for action recognition results compared to a single action result, while avoiding the problem of inaccurate single action result caused by poor biological signal quality, thereby improving the fault tolerance of action result prediction. In order to subsequently combine the prediction probabilities of each action type for quantitative analysis and improve the accuracy of action prediction.

[0039] S103, determining a target action according to the multiple action types in each action result and the respective corresponding prediction probabilities.

[0040] Specifically, the action types in each action result and the respective corresponding prediction probabilities can be comprehensively quantitatively analyzed to determine the target action. In this embodiment, compared with the traditional action recognition method which only selects the action type with the highest prediction probability as the target action and ignores other possible action types and their probability information, the motion intention of the user can be more comprehensively and accurately analyzed by considering multiple action types and their prediction probabilities, the errors caused by the limitations of single decision are avoided, and the accuracy of the target action is ensured.

[0041] S104, controlling the prosthesis based on the target action.

[0042] Specifically, the prosthesis control device issues a control instruction to the prosthesis to perform the target action, and controls the prosthesis to perform the target action. It can be understood that since the prosthesis is controlled according to the target action with higher accuracy, the precision of the prosthesis control is improved, and the user experience is improved.

[0043] To sum up, the prosthesis control method provided in the embodiments of the present application acquires the biological signals of the user in the process of using the prosthesis, the biological signals including electromyographic signals, and voice signals and / or electroencephalogram signals, so that the biological signals used for prosthesis control at least include electromyographic signals, which can provide direct, real-time and rich information for the accurate control of the prosthesis, and the combination of electromyographic signals with voice signals and / or electroencephalogram signals can better adapt to the individual needs of different users due to the complementarity of different signals; the pre-trained action recognition model is used to recognize the action results corresponding to each biological signal, the action results including multiple action types and respective corresponding prediction probabilities, since multiple action types and their prediction probabilities are recognized, more candidate options of action recognition results can be provided, so as to subsequently perform quantitative analysis combined with the prediction probabilities of each action type, and the accuracy of action prediction is improved; the target action is determined according to the multiple action types in each action result and the respective corresponding prediction probabilities, which can more comprehensively and accurately analyze the motion intention of the user, and ensures the accuracy of the target action; the prosthesis is controlled based on the target action, since the prosthesis is controlled according to the target action with higher accuracy, the precision of the prosthesis control is improved, and the user experience is improved.

[0044] In some embodiments of the present application, as shown in Figure 3 S103 includes: S201, determining a type weight of the action types in the action result according to the signal type of the biological signal corresponding to the action result; S202, for each action type in the action result, determining a comprehensive probability of each action type according to the type weight and the corresponding prediction probability; S203, determining a target action according to the comprehensive probability of each action type.

[0045] Specifically, since biological signals of different signal types have different reliability and importance, by determining the type weight of the action result corresponding to each signal type of biological signal, the contribution degree of the recognition result under different signal types in action recognition is reflected, and the accuracy of decision is improved. For the action result of each biological signal, each action type under the respective action result and the corresponding prediction probability are calculated to determine the comprehensive probability of each action type, and through the comprehensive probability, the most possible action type is selected as the target action, thereby improving the accuracy and reliability of action decision.

[0046] In some embodiments of the present application, the action recognition model comprises a signal type recognition sub-model and three parallel action recognition sub-models, the signal type recognition sub-model is cascaded with the three action recognition sub-models respectively, and the three action recognition sub-models are respectively the action recognition sub-models corresponding to the electromyographic signal, the speech signal and the electroencephalogram signal. Figure 4 As shown in the figure, step S102 comprises: S301, identifying the signal type of the biological signal through the signal type recognition sub-model; S302, according to the signal type, using the corresponding action recognition sub-model to respectively identify the biological signal of the corresponding signal type, to obtain the action result corresponding to each biological signal.

[0047] The signal type recognition sub-model is a model for identifying the signal type of the biological signal, and can train a classifier by using historical biological signals containing electromyographic signals, speech signals and electroencephalogram signals and the corresponding signal types as labels to obtain the signal type recognition sub-model.

[0048] The three action recognition sub-models are models for respectively identifying the action result under the electromyographic signal, the speech signal and the electroencephalogram signal, and can be trained based on at least one model in random forest, gradient boosting machine, support vector machine, K-nearest neighbor, neural network and naive Bayes to obtain the action recognition sub-model.

[0049] The signal type recognition sub-model is cascaded with the three action recognition sub-models respectively, and the three action recognition sub-models are parallel, to constitute the action recognition model.

[0050] Specifically, the signal type recognition sub-model can accurately distinguish between different types of biological signals. Each action recognition sub-model can be optimized according to the signal type it specifically processes, further improving the accuracy of action result recognition.

[0051] In some embodiments of the present invention, such as Figure 5 As shown, step S302 includes: S401. When the signal type is electromyography (EMG) signal, obtain the EMG signal to be identified from the biological signal, and input the first feature of the EMG signal to be identified into the action recognition sub-model corresponding to the EMG signal to obtain multiple corresponding action types and their respective first prediction probabilities. S402. When the signal type is an EEG signal, the EEG signal to be identified is obtained from the biological signal, and the second feature of the EEG signal to be identified is input into the action recognition sub-model corresponding to the EEG signal to obtain multiple corresponding action types and their respective second prediction probabilities. S403. When the signal type is a speech signal, obtain the speech signal to be recognized from the biosignal, and input the third feature of the speech signal to be recognized into the action recognition sub-model corresponding to the speech signal to obtain multiple corresponding action types and their respective third prediction probabilities.

[0052] The first feature is the characteristic of the electromyographic signal to be identified, such as time domain features (mean, root mean square, variance, etc.) and frequency domain features (short-time Fourier transform, wavelet packet transform, etc.).

[0053] The formula for the mean characteristic is as follows:

[0054] Indicates the characteristic of the mean; x n Indicates the first in the signal n The value of each sampling point; N This indicates the total number of sampling points for the signal.

[0055] The formula for the root mean square characteristic is as follows:

[0056] Indicates the root mean square characteristic; x n Indicates the first in the signal n The value of each sampling point; N This indicates the total number of sampling points for the signal.

[0057] The formula for variance characteristics is as follows:

[0058] denotes a variance feature; x n denotes a value of a i-th sample point in a signal, n denotes a mean value; denotes a mean value; N denotes a total number of sample points of a signal.

[0059] The formula of the short-time Fourier transform feature is as follows:

[0060] denotes a short-time Fourier transform feature; x denotes a signal, is a window function, is a frequency, t is a time.

[0061] The formula of the wavelet packet transform feature is as follows:

[0062] denotes a wavelet packet transform feature; denotes a wavelet basis function; j denotes a scale parameter; k denotes a translation parameter.

[0063] The second feature is a feature of the brain electrical signal to be identified, such as a time domain feature (mean value, root mean square, variance, etc.) and a frequency domain feature (short-time Fourier transform, wavelet packet transform, etc.), and the time domain feature and the frequency domain feature are consistent with the time domain feature and the frequency domain feature in the first feature, which will not be described here.

[0064] The third feature is a feature of the speech signal to be identified, such as a time domain feature (mean value, root mean square, variance, etc.) and a frequency domain feature (short-time Fourier transform, wavelet packet transform, etc.), and the time domain feature and the frequency domain feature are consistent with the time domain feature and the frequency domain feature in the first feature, which will not be described here.

[0065] Specifically, the electromyographic signal to be identified is preprocessed to remove noise and interference, so as to facilitate subsequent feature extraction and action recognition. A band-pass filter can be used to filter the electromyographic signal to be identified, and the electromyographic signal rich in information is retained. Then, the time domain feature and the frequency domain feature of the preprocessed electromyographic signal to be identified are extracted to obtain the first feature, which can better represent the characteristics of the electromyographic signal and provide useful information for action recognition. The first feature is taken as an action recognition sub-model corresponding to the electromyographic signal, and the output of the action recognition sub-model is a plurality of action types corresponding to the electromyographic signal and respective first prediction probabilities, Using the trained action recognition sub-model, the action type corresponding to the electromyographic signal and the first prediction probability thereof can be accurately identified.

[0066] The brain electrical signal to be identified is preprocessed to remove noise and interference, so that subsequent feature extraction and action recognition can be performed. A band-pass filter can be used to filter the brain electrical signal to be identified, retaining the information-rich brain electrical signal. Then, the time domain features and frequency domain features of the preprocessed brain electrical signal to be identified are extracted to obtain second features, which can better represent the characteristics of the brain electrical signal and provide useful information for action recognition. The second features are used as the action recognition sub-model corresponding to the brain electrical signal. The output of the action recognition sub-model is the multiple action types corresponding to the brain electrical signal and the respective second prediction probabilities thereof. Using the trained action recognition sub-model, the action type corresponding to the electromyographic signal and the second prediction probability thereof can be accurately identified.

[0067] The speech signal to be identified is preprocessed to remove noise, and then activity detection is performed to select a number of information-rich speech signals for subsequent feature extraction and action recognition. A band-pass filter can be used to filter the speech signal to be identified, retaining the information-rich speech signal. Then, the time domain features and frequency domain features of the preprocessed speech signal to be identified are extracted to obtain third features, which can better represent the characteristics of the speech signal and provide useful information for action recognition. The third features are used as the action recognition sub-model corresponding to the speech signal. The output of the action recognition sub-model is the multiple action types corresponding to the speech signal and the respective third prediction probabilities thereof. Using the trained action recognition sub-model, the action type corresponding to the electromyographic signal and the third prediction probability thereof can be accurately identified.

[0068] In some embodiments of the present application, step S202 comprises: S501, calculating the type weight and the corresponding prediction probability respectively by weighted summation to obtain the comprehensive probability.

[0069] Specifically, by calculating the type weight and the corresponding prediction probability by weighted summation, the comprehensive probability of each action type can be obtained. Since the advantages of multi-modal are fully utilized and the candidate options of multiple action types are fully utilized, the quantization result is combined with the prediction probability as the comprehensive probability, which improves the accuracy and rationality of the comprehensive probability.

[0070] In one specific embodiment, when the biological signal includes electromyographic signals and speech signals: the speech signal is the main one, and the electromyographic signal is the auxiliary one. At this time, the calculation formula of the comprehensive probability is as follows:

[0071]

[0072] wherein, represents the comprehensive probability of the action type being ; respectively represent the type weight of the electromyography signal and the speech signal for the action type being i , and the sum of the two is 1, and the type weight corresponding to the speech signal is greater than the type weight corresponding to the electromyography signal; and respectively represent the prediction probability of the electromyography signal and the speech signal for the action type being i .

[0073] In another specific embodiment, when the biosignal includes the electromyography signal and the electroencephalogram signal: the electromyography signal is given priority, and the electroencephalogram signal is given secondary priority. At this time, the calculation formula of the comprehensive probability is as follows:

[0074]

[0075] wherein, P i represents the comprehensive probability of the action type being i ; respectively represent the type weight of the electromyography signal and the electroencephalogram signal for the action type being i , and the sum of the two is 1, and the type weight corresponding to the electromyography signal is greater than the type weight corresponding to the electroencephalogram signal; and respectively represent the prediction probability of the electromyography signal and the electroencephalogram signal for the action type being i .

[0076] In yet another specific embodiment, when the biosignal includes the electromyography signal, the speech signal and the electroencephalogram signal: the speech signal is given priority, and the electromyography signal and the electroencephalogram signal are given secondary priority. At this time, the calculation formula of the comprehensive probability is as follows:

[0077]

[0078] wherein, P i represents the comprehensive probability of the action type being i ; respectively represent the type weight of the electromyography signal, the speech signal and the electroencephalogram signal for the action type being i , and the sum of the three is 1, and the type weight corresponding to the speech signal is greater than the sum of the type weights corresponding to the other two signals; respectively represent the prediction probability of the electromyography signal, the speech signal and the electroencephalogram signal for the action type being i .

[0079] In some embodiments of the present application, as shown in Figure 6 Step S104 includes: S601, in the case where the biological signal includes a speech signal, acquiring a speech signal to be recognized; S602, determining an adjustment parameter corresponding to the target action according to the speech signal to be recognized; S603, controlling the prosthesis to perform the target action according to the adjustment parameter.

[0080] Specifically, the speech signal can be converted into text using a speech recognition model, such as a deep learning-based speech recognition system. The recognized text can contain action adjustment instructions, such as "quickly grab" and "lightly stretch". According to the content of the recognized text, the specific adjustment parameter can be parsed. "Quickly grab" can be parsed as the target action "grab" and the adjustment parameter "speed = quickly". A preset parameter mapping table is established to map the speech instruction to the specific adjustment parameter, and the prosthesis is controlled to perform the target action according to the adjustment parameter, further improving the accuracy of the prosthesis and improving the user's experience.

[0081] For example, the "quick" part in "quickly make a fist" in the speech signal is identified and mapped by the parameter adjustment module. The obtained control parameter will be a multiple of the normal fist action control parameter (such as time will be 1 / 2).

[0082] In some embodiments of the present application, as shown in Figure 7 Step S201 includes: S701, if the biological signal includes a speech signal and an electromyography signal, determining that the first type weight corresponding to the speech signal is greater than the third type weight corresponding to the electromyography signal; S702, if the biological signal includes an electroencephalogram signal and an electromyography signal, determining that the third type weight corresponding to the electromyography signal is greater than the second type weight corresponding to the electroencephalogram signal; S703, if the biological signal includes an electroencephalogram signal, an electroencephalogram signal, and an electromyography signal, determining that the first type weight corresponding to the speech signal is greater than the third type weight corresponding to the electromyography signal, and the third type weight corresponding to the electromyography signal is greater than the second type weight corresponding to the electroencephalogram signal.

[0083] Specifically, since the speech signal is generally considered more direct and more explicit, it is assigned the highest type weight. The electromyography signal is more direct than the electroencephalogram information, so it is assigned the second highest type weight. The electroencephalogram signal is assigned the lowest type weight. Through the preset weight rule, the weights of different signal types are reasonably distributed, and the accuracy of action recognition is improved.

[0084] In order to better implement the prosthesis control method in the embodiments of the present application, on the basis of the prosthesis control method, as shown in Figure 8 The present application also provides a prosthesis control device, as shown in The acquisition unit 801 is configured to acquire biological signals of a user during use of a prosthesis, wherein the biological signals include electromyographic signals, and voice signals and / or electroencephalographic signals. The identification unit 802 is configured to identify an action result corresponding to each biological signal by using a pre-trained action recognition model, wherein the action result includes a plurality of action types and respective corresponding prediction probabilities. The determination unit 803 is configured to determine a target action according to the plurality of action types and respective corresponding prediction probabilities in each action result. The control unit 804 is configured to control the prosthesis based on the target action.

[0085] The prosthesis control device 800 provided in the above embodiments can implement the technical solutions described in the above prosthesis control method embodiments, and the principles of the implementation of the above modules or units can be referred to the corresponding content in the above prosthesis control method embodiments, which will not be described here.

[0086] As shown in Figure 9 The present application also provides a prosthesis control device 900. The prosthesis control device 900 includes a processor 901, a memory 902 and a display 903. Figure 9 Only part of the components of the prosthesis control device 900 are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0087] The processor 901 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, and is configured to run program codes or process data stored in the memory 902, such as the prosthesis control method in the present application.

[0088] In some embodiments, the processor 901 can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor 901 can be local or remote. In some embodiments, the processor 901 can be implemented in a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-internal, a multiple cloud, etc., or any combination thereof.

[0089] The memory 902 can be an internal storage unit of the prosthetic control device 900 in some embodiments, such as a hard disk or a memory of the prosthetic control device 900. The memory 902 can also be an external storage device of the prosthetic control device 900 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the prosthetic control device 900.

[0090] Further, the memory 902 can include both an internal storage unit and an external storage device of the prosthetic control device 900. The memory 902 is used to store application software and various data installed on the prosthetic control device 900.

[0091] The display 903 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 903 is used to display information of the prosthetic control device 900 and to display a visualized user interface. The components 901-903 of the prosthetic control device 900 communicate with each other through a system bus.

[0092] In an embodiment, the following steps can be implemented when the processor 901 executes the prosthetic control program in the memory 902: Obtaining biological signals of a user in the process of using a prosthetic, the biological signals including electromyographic signals, voice signals and / or electroencephalographic signals; Identifying an action result corresponding to each biological signal using a pre-trained action recognition model, the action result including a plurality of action types and respective corresponding prediction probabilities; Determining a target action according to the plurality of action types and respective corresponding prediction probabilities in each action result; Controlling the prosthetic based on the target action.

[0093] It should be understood that, in addition to the above functions, the processor 901 can also implement other functions when executing the prosthetic control program in the memory 902, which can be specifically referred to the description of the corresponding method embodiments.

[0094] Further, the embodiments of the present application do not make specific limitation on the type of the prosthesis control device 900 mentioned above, and the prosthesis control device 900 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, or the like. Exemplary embodiments of the portable prosthesis control device include, but are not limited to, a portable prosthesis control device running an IOS, an android, a microsoft, or other operating system. The portable prosthesis control device described above can also be other portable prosthesis control devices, such as a laptop computer having a touch-sensitive surface (e.g., a touch panel), and the like. It should also be understood that in some other embodiments of the present application, the prosthesis control device 800 can also not be a portable prosthesis control device, but a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0095] Correspondingly, the embodiments of the present application also provide a computer readable storage medium for storing computer readable programs or instructions, which, when executed by a processor, can implement the steps or functions in the prosthesis control method provided by the above-mentioned method embodiments.

[0096] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing relevant hardware (such as a processor, a controller, etc.) to complete, and the computer program can be stored in a computer readable storage medium. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0097] The above describes in detail the prosthesis control method, device, prosthesis control device, and storage medium provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed; and in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for controlling a prosthesis, characterized in that, include: Acquire biosignals of the user during the use of the prosthesis, including electromyographic signals, as well as speech signals and / or electroencephalogram (EEG) signals; A pre-trained action recognition model is used to identify the action result corresponding to each biosignal. The action result includes multiple action types and their respective predicted probabilities. The target action is determined based on the multiple action types and their corresponding predicted probabilities in each action result; The prosthesis is controlled based on the target action.

2. The prosthesis control method according to claim 1, characterized in that, The step of determining the target action based on multiple action types and their corresponding predicted probabilities in each action result includes: Based on the signal type of the biological signal corresponding to the action result, determine the type weight of the action type in the action result; For each action type in the action results, the comprehensive probability of each action type is determined based on the type weight and the corresponding prediction probability; The target action is determined based on the combined probability of each action type.

3. The prosthetic limb control method according to claim 1, characterized in that, The action recognition model includes a signal type recognition sub-model and three parallel action recognition sub-models. The signal type recognition sub-model is cascaded with each of the three action recognition sub-models, which are action recognition sub-models corresponding to electromyography (EMG) signals, speech signals, and electroencephalography (EEG) signals, respectively. The step of using a pre-trained action recognition model to identify the action result corresponding to each biosignal includes: The signal type of the biological signal is identified through the signal type identification sub-model. Based on the signal type, the corresponding action recognition sub-model is used to identify the biological signals of the corresponding signal type, and the action result corresponding to each biological signal is obtained.

4. The prosthesis control method according to claim 3, characterized in that, The step of identifying the biological signals of the corresponding signal type using the corresponding action recognition sub-model according to the signal type, and obtaining the action result corresponding to each biological signal, includes: When the signal type is electromyography (EMG) signal, the EMG signal to be identified is obtained from the biological signal, and the first feature of the EMG signal to be identified is input into the action recognition sub-model corresponding to the EMG signal to obtain multiple corresponding action types and their respective first prediction probabilities. When the signal type is an EEG signal, the EEG signal to be identified is obtained from the biological signal, and the second feature of the EEG signal to be identified is input into the action recognition sub-model corresponding to the EEG signal to obtain multiple corresponding action types and their respective second prediction probabilities. When the signal type is a speech signal, the speech signal to be recognized is obtained from the biosignal, and the third feature of the speech signal to be recognized is input into the action recognition sub-model corresponding to the speech signal to obtain multiple corresponding action types and their respective third prediction probabilities.

5. The prosthesis control method according to claim 2, characterized in that, For each action type in the action results, the comprehensive probability of each action type is determined based on the type weight and the corresponding predicted probability, including: The combined probability is obtained by weighting and summing the type weights and their corresponding predicted probabilities.

6. The prosthetic limb control method according to any one of claims 1-5, characterized in that, The control of the prosthesis based on the target action includes: In the case where the biosignal includes a speech signal, the speech signal to be identified is acquired; The adjustment parameters corresponding to the target action are determined based on the speech signal to be recognized. Control the prosthesis to perform the target action according to the adjustment parameters.

7. The prosthesis control method according to claim 2, characterized in that, The step of determining the type weight of the action type in the action result based on the signal type of the biosignal corresponding to the action result includes: If the biosignal includes a speech signal and an electromyographic signal, the first type weight corresponding to the speech signal is determined to be greater than the third type weight corresponding to the electromyographic signal. If the biological signal includes electroencephalogram (EEG) signal and electromyogram (EMG) signal, the third type weight corresponding to the EMG signal is determined to be greater than the second type weight corresponding to the EEG signal. If the biosignal includes EEG signal, EEG signal and EMG signal, the first type weight corresponding to the speech signal is determined to be greater than the third type weight corresponding to the EMG signal, and the third type weight corresponding to the EMG signal is greater than the second type weight corresponding to the EEG signal.

8. A prosthetic limb control device, characterized in that, include: The acquisition unit is used to acquire the user's biosignals during the use of the prosthesis, including electromyographic signals, as well as speech signals and / or electroencephalogram (EEG) signals. The recognition unit is used to identify the action result corresponding to each biosignal using a pre-trained action recognition model. The action result includes multiple action types and their respective predicted probabilities. The determination unit is used to determine the target action based on multiple action types and their corresponding predicted probabilities in each action result; A control unit for controlling the prosthesis based on the target action.

9. A prosthetic limb control device, characterized in that, Includes a memory and a processor, wherein the memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the prosthetic control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the prosthetic control method according to any one of claims 1 to 7.

Images