Adaptive wheelchair control method and system based on multimodal motion signals of neck

Abstract

The present invention discloses an adaptive wheelchair control method and system based on neck multimodal motion signals, the method comprising: obtaining neck electromyographic signals and neck strain signals, constructing neck multimodal motion signals according to the neck electromyographic signals and neck strain signals; training the RTNMRM network model according to the neck multimodal motion signals by using the random gradient descent method, calculating the loss function, updating the network model parameters until the loss function converges, and obtaining a trained neck motion recognition model; inputting the acquired real-time neck multimodal motion signal into the neck motion recognition model, the neck motion recognition model outputs the neck motion recognition result, and controlling the wheelchair to execute the walking command according to the neck motion recognition result. The lightweight network model is constructed to overcome several shortcomings of the traditional neck motion recognition control wheelchair, solve the problem of time window alignment, significantly increase the adaptive ability of recognition, and improve the efficiency and accuracy of recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction control, and in particular relates to an adaptive wheelchair control method and system based on neck multimodal motion signals. Background Art

[0002] Starting from the main body of human movement, interacting with wheelchairs through human habits and natural ways such as voice and body language, thereby getting rid of the constraints of traditional human-computer interfaces such as joysticks and buttons, and replacing the missing motor functions of disabled people, has become an important research field of human-computer interaction. Among them, the voice interaction method has the problem of large noise interference and requires a high level of quietness in the environment. In body language, head and neck movements such as turning the head left, right, bending forward, and leaning back can effectively reflect the user's operating intention, similar to a natural rocker. For identifying head and neck movements, visual recognition is a commonly used method, but this method has the disadvantages of large light source influence and high cost. In addition, inertial sensors can also reflect the posture of the head and neck, but this method is used in wheelchair control and will be disturbed by the external terrain, and it is not wearable and flexible enough. Therefore, seeking other methods to effectively reflect the movement state of the head and neck is of great significance for human-computer interaction applications.

[0003] According to current research results, controlling the wheelchair by identifying the intention of neck movement is an effective method to help upper limb disabled people walk; however, there are the following shortcomings:

[0004] 1. The neck movement signal is not comprehensive enough;

[0005] 2. The recognition method using a fixed time window mixes the active segment and inactive segment signals;

[0006] 3. The method of dividing time windows using activity thresholds is difficult to align and lacks adaptive capabilities;

[0007] 4. The number of network layers is large, the computing efficiency is low, and it is difficult to adapt to real-time requirements;

[0008] Therefore, there are still some defects in neck motion recognition, and the ideal control effect is often not achieved when controlling the wheelchair. Therefore, there is an urgent need for an adaptive wheelchair control method and system based on neck multimodal motion signals. Summary of the invention

[0009] The present invention provides an adaptive wheelchair control method and system based on multimodal neck motion signals, which overcomes several shortcomings of traditional neck motion recognition and wheelchair control, solves the problem of time window alignment, significantly increases the adaptive ability of recognition, and improves the efficiency and accuracy of recognition.

[0010] In a first aspect, the present invention provides an adaptive wheelchair control method based on a multimodal motion signal of a neck, comprising:

[0011] Acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal;

[0012] According to the neck multimodal motion signal, the RTNMRM network model is trained by using the stochastic gradient descent method, the loss function is calculated, and the network model parameters are updated until the loss function converges to obtain a trained neck motion recognition model;

[0013] The acquired real-time neck multimodal motion signal is input into the neck motion recognition model, and the neck motion recognition model outputs a neck motion recognition result. The wheelchair is controlled to execute a walking command according to the neck motion recognition result.

[0014] In a second aspect, the present invention provides an adaptive wheelchair control system based on a multimodal motion signal of the neck, comprising:

[0015] An acquisition module is configured to acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal;

[0016] A training module is configured to train the RTNMRM network model according to the neck multimodal motion signal using a stochastic gradient descent method, calculate a loss function, and update network model parameters until the loss function converges to obtain a trained neck motion recognition model;

[0017] The output module is configured to input the acquired real-time neck multimodal motion signal into the neck motion recognition model, the neck motion recognition model outputs the neck motion recognition result, and controls the wheelchair to execute the walking command according to the neck motion recognition result.

[0018] In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the adaptive wheelchair control method based on neck multimodal motion signals of any embodiment of the present invention.

[0019] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes the steps of the adaptive wheelchair control method based on multimodal motion signals of the neck according to any embodiment of the present invention.

[0020] The adaptive wheelchair control method and system based on neck multimodal motion signals of the present application enrich the neck multimodal motion signals by integrating neck electromyography signals and neck strain signals. In the signal preprocessing stage, activity threshold detection, filtering and normalization processing are performed to obtain a symmetrical feature matrix that is not constrained by the length of the time window. In the feature extraction stage, features are effectively extracted through improved two-dimensional wavelet transform. In the classification and recognition stage, convolutional networks, attention mechanisms and memory networks are combined to construct a lightweight network model, which overcomes several shortcomings of traditional neck motion recognition and control wheelchairs, solves the problem of time window alignment, significantly increases the adaptive ability of recognition, and improves the efficiency and accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an adaptive wheelchair control method based on multimodal neck motion signals provided by an embodiment of the present invention;

[0023] Figure 2 A structural block diagram of an adaptive wheelchair control system based on multimodal neck motion signals provided by an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 , which shows a flow chart of an adaptive wheelchair control method based on neck multimodal motion signals of the present application.

[0027] like Figure 1 As shown, the adaptive wheelchair control method based on the multimodal motion signal of the neck specifically includes the following steps:

[0028] Step S101, acquiring a neck electromyographic signal and a neck strain signal, and constructing a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal.

[0029] In this step, the neck electromyographic signal and neck strain signal generated by the neck muscle group when the user's neck moves are obtained. Before obtaining, the user's neck surface skin is wiped clean with alcohol to reduce the appearance of motion artifacts. The muscle group refers to the upper trapezius muscle of the human body. The neck electromyographic signal is collected by the electromyographic sensor placed symmetrically on the upper trapezius muscle. Two electromyographic leads placed differentially 2 cm apart constitute one neck electromyographic signal, with a total of two channels; the neck strain signal is collected by the flexible strain sensor placed on the upper trapezius muscle, with a total of four channels; the neck electromyographic signal and the neck strain signal are collected synchronously, and the total six-channel signal is used to construct a neck multimodal motion signal, and the expression is:

[0030] , (1)

[0031] In the formula, For the Channel signals, where channels 1 and 2 are neck electromyography signals, and channels 3, 4, 5, and 6 are neck strain signals. is the multimodal motion signal of the neck composed of all channels, is the sampling time.

[0032] Step S102, according to the neck multimodal motion signal, the RTNMRM network model is trained using the stochastic gradient descent method, the loss function is calculated, and the network model parameters are updated until the loss function converges to obtain a trained neck motion recognition model.

[0033] In this step, the multimodal motion signal of the neck is preprocessed, feature extracted, and classified. Preprocessing refers to improving signal quality, reducing noise, and making the signal better adaptable to subsequent algorithms, which is achieved by the TIMEEMD algorithm. The feature extraction refers to converting the preprocessed signal into a more discriminative feature set, reducing the complexity and redundancy of the data, and enhancing task-related information for subsequent classification, which is achieved by the XmvfoNet algorithm. The classification is to map the feature-extracted information to the neck motion result label after training and iterative learning, which is achieved by the LclaNet network.

[0034] The multimodal motion signal of the neck is preprocessed, specifically:

[0035] Calculate the average value of all sampling points within 2 seconds in front of the neck, and record it as , 10 consecutive sampling points are greater than the average value The signal is taken as the starting active point, and 10 consecutive sampling points are all less than the average value. The signal of the starting activity point and the ending activity point constitute the signal in the activity time window , the expression is:

[0036] , (2)

[0037] In the formula, The time point to terminate the activity. is the time point at which the activity starts. is the longest activity period, is the signal in the active time window, is the multimodal motion signal of the neck composed of all channels, is the multimodal motion signal of the neck at the starting point, Multimodal motion signals of the neck for the stop activity point;

[0038] Perform empirical mode decomposition on the signals of all channels in the active time window, and the expression is:

[0039] , (3)

[0040] In the formula, is the mode number, is the total number of modes, For the Channel No. Modal components, is the residual signal;

[0041] The specific calculation steps are as follows: connect the signal with the local extreme point to obtain the upper and lower envelopes, and calculate the mean of the upper and lower envelopes as shown in formula (4); then, subtract the original signal from the mean signal to obtain a new signal as shown in formula (5); then determine whether the signal meets the IMF satisfaction condition: the difference between the number of extreme points and the number of zero-crossing points is not greater than and the signal envelope composed of local minimum points and local maximum points is locally symmetrical, that is, their average value approaches zero. If the condition is met, the signal is an IMF, as shown in formula (6); otherwise, repeat formulas (4) and (5) until the condition is met. Then, subtract the IMF from the original signal to obtain the residual signal, as shown in formula (7); if the residual signal is a monotonic function or a constant, the calculation stops; otherwise, use the residual signal as the base signal and calculate the next IMF until all IMFs that meet the requirements are found.

[0042] , (4)

[0043] , (5)

[0044] , (6)

[0045] , (7)

[0046] In the formula, is the mean signal, is the upper envelope, is the lower envelope, is the difference between the original signal and the mean signal, is the original signal, For the Modal components, is the residual signal after subtracting the high-frequency component from the original signal. For the Channel No. modal components;

[0047] The first four IMF components are added in pairs to calculate the filtered active segment signal, which is expressed as:

[0048] ,

[0049] In the formula, is the filtered active segment signal;

[0050] The filtered active segment signal is normalized with the maximum and minimum values ​​of the offset, and the active segment signal is normalized to The interval expression is:

[0051] ,

[0052] In the formula, is the normalized signal, is the sample offset, is the maximum value of the signal, is the minimum value of the signal, is the maximum offset;

[0053] In order to unify the time windows of different lengths and improve the network's adaptive ability, the normalized signal , and the normalized signal Multiply the transpose of to get the normalized symmetric matrix, the expression is:

[0054] ,

[0055] In the formula, is the normalized symmetric matrix, is the transpose of the normalized signal.

[0056] Feature extraction is performed on the multimodal motion signal of the neck, specifically:

[0057] According to the improved two-dimensional discrete wavelet transform, the symmetric matrix Decompose into approximate coefficients at different scales and directions and detail factor , the expression is:

[0058] ,

[0059] ,

[0060] ,

[0061] ,

[0062] In the formula, For the The layer approximation coefficients are at positions The value of is the dimension size of the symmetric matrix, is a symmetric matrix in position The value of is the Xmvfo wavelet function, is the index of the scale, is the index of the position, For the Layer detail factor at position The value of is the adjoint wavelet function of Xmvfo, is the dynamic factor of the wavelet function, is the bandwidth parameter of the wavelet, is a positional parameter, is the frequency parameter of the wavelet;

[0063] By combining the approximation coefficients and detail factor Reconstruct the symmetric matrix, the expression is:

[0064] ,

[0065] In the formula, is the scattering coefficient matrix after wavelet reconstruction, is the scale factor, which determines the scale level of decomposition. is the direction quantity factor, which represents the number of decomposition directions.

[0066] The multimodal motion signal of the neck is classified and identified, specifically:

[0067] Construct a scattering coefficient feature extraction module, which includes three feature extraction units in sequence. Each feature extraction unit includes: a convolution layer, an activation function layer, and a maximum pooling layer. The convolution kernel size of the convolution layer is 3. 3, the step size is 1, the activation function layer is the Relu activation function, and the window size of the maximum pooling layer is 2 2, the step size is 2;

[0068] Construct an attention convergence module, which consists of a multi-head attention mechanism, and the expression is:

[0069] ,

[0070] In the formula, is the output value of the u-th attention head, is the query matrix, is the key-value matrix, is the output value matrix, is the offset parameter, is the scaling parameter, is the query matrix dimension;

[0071] Construct a target domain classification module, which includes 64 LSTM hidden units and a dimension of Fully connected layers;

[0072] Constructing a weighted cross entropy loss function with L2 parameter penalty , the expression is:

[0073] ,

[0074] ,

[0075] In the formula, is the regularization coefficient, is the number of network layers, For the The weight matrix of the layer, is the weighted cross entropy loss function, For the The weight of the category, For the The sample The true class label, The model predicts The samples belong to The probability of the class, is the number of categories, is the number of samples;

[0076] Based on the scattering coefficient feature extraction module, the attention convergence module, the target domain classification module and the weighted cross entropy loss function An LclaNet network is constructed, and the multimodal motion signal of the neck is classified and identified according to the LclaNet network.

[0077] Step S103, inputting the acquired real-time neck multimodal motion signal into the neck motion recognition model, the neck motion recognition model outputs a neck motion recognition result, and controlling the wheelchair to execute a walking command according to the neck motion recognition result.

[0078] In this step, the user's neck multimodal motion signal is obtained and input into the trained neck motion recognition model to obtain the neck motion recognition result. The neck motion recognition result is transmitted to the controller of the wheelchair by wireless transmission, and the wireless transmission mode is Bluetooth wireless transparent transmission. The controller of the wheelchair is composed of an STM32 single-chip microcomputer and its motor drive controller. The controller of the wheelchair executes the corresponding control command according to the motion instruction table, controls the rotation of the left and right motors, and realizes the walking of the wheelchair according to the user's neck motion intention. The control instructions are shown in Table 1.

[0079] ,

[0080] In summary, the method of the present application enriches the multimodal motion signal of the neck by integrating the neck electromyography signal and the neck strain signal. In the signal preprocessing stage, a symmetrical feature matrix that is not constrained by the length of the time window is obtained through activity threshold detection, filtering and normalization processing. In the feature extraction stage, features are effectively extracted through improved two-dimensional wavelet transform. In the classification and recognition stage, convolutional networks, attention mechanisms and memory networks are combined to construct a lightweight network model, which overcomes several shortcomings of traditional neck motion recognition and control wheelchairs, solves the problem of time window alignment, significantly increases the adaptive ability of recognition, and improves the efficiency and accuracy of recognition.

[0081] See also Figure 2 , which shows a structural block diagram of an adaptive wheelchair control system based on neck multimodal motion signals of the present application.

[0082] like Figure 2 As shown, the adaptive wheelchair control system 200 includes an acquisition module 210 , a training module 220 and an output module 230 .

[0083] Among them, the acquisition module 210 is configured to acquire neck electromyographic signals and neck strain signals, and construct neck multimodal motion signals according to the neck electromyographic signals and the neck strain signals; the training module 220 is configured to train the RTNMRM network model according to the neck multimodal motion signals using the stochastic gradient descent method, calculate the loss function, and update the network model parameters until the loss function converges to obtain a trained neck motion recognition model; the output module 230 is configured to input the acquired real-time neck multimodal motion signal into the neck motion recognition model, and the neck motion recognition model outputs the neck motion recognition result, and controls the wheelchair to execute the walking command according to the neck motion recognition result.

[0084] It should be understood that Figure 2 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 2 The modules in it will not be described in detail here.

[0085] In some other embodiments, the embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the adaptive wheelchair control method based on the multimodal motion signal of the neck in any of the above method embodiments;

[0086] As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0087] Acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal;

[0088] According to the neck multimodal motion signal, the RTNMRM network model is trained by using the stochastic gradient descent method, the loss function is calculated, and the network model parameters are updated until the loss function converges to obtain a trained neck motion recognition model;

[0089] The acquired real-time neck multimodal motion signal is input into the neck motion recognition model, and the neck motion recognition model outputs a neck motion recognition result. The wheelchair is controlled to execute a walking command according to the neck motion recognition result.

[0090] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of an adaptive wheelchair control system based on a multimodal motion signal of the neck, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the adaptive wheelchair control system based on a multimodal motion signal of the neck via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 In the example, the bus connection is used. The memory 320 is the above-mentioned computer-readable storage medium. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the adaptive wheelchair control method based on the multimodal motion signal of the neck in the above-mentioned method embodiment is realized. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the adaptive wheelchair control system based on the multimodal motion signal of the neck. The output device 340 may include a display device such as a display screen.

[0092] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0093] As an implementation mode, the electronic device is applied to an adaptive wheelchair control system based on a multimodal motion signal of the neck, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:

[0094] Acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal;

[0095] According to the neck multimodal motion signal, the RTNMRM network model is trained by using the stochastic gradient descent method, the loss function is calculated, and the network model parameters are updated until the loss function converges to obtain a trained neck motion recognition model;

[0096] The acquired real-time neck multimodal motion signal is input into the neck motion recognition model, and the neck motion recognition model outputs a neck motion recognition result. The wheelchair is controlled to execute a walking command according to the neck motion recognition result.

[0097] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive wheelchair control method based on multimodal neck motion signals, characterized in that: include: Acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal; According to the neck multimodal motion signal, the RTNMRM network model is trained by using the stochastic gradient descent method, the loss function is calculated, and the network model parameters are updated until the loss function converges to obtain a trained neck motion recognition model. Among them, before training the RTNMRM network model using the stochastic gradient descent method according to the multimodal motion signal of the neck, the method further includes: The multimodal motion signal of the neck is preprocessed, feature extracted, and classified and identified, specifically: Calculate the average value of all sampling points within 2 seconds in front of the neck, and record it as , 10 consecutive sampling points are greater than the average value The signal is taken as the starting active point, and 10 consecutive sampling points are all less than the average value. The signal of the starting activity point and the ending activity point constitute the signal in the activity time window , the expression is: ， In the formula, The time point to terminate the activity. is the time point at which the activity starts. is the longest activity period, is the signal in the active time window, is the multimodal motion signal of the neck composed of all channels, is the multimodal motion signal of the neck at the starting point, Multimodal motion signals of the neck for the stop activity point; Perform empirical mode decomposition on the signals of all channels in the active time window, and the expression is: ， In the formula, is the mode number, is the total number of modes, For the Channel No. Modal components, is the residual signal; The first four IMF components are added in pairs to calculate the filtered active segment signal, which is expressed as: ， In the formula, is the filtered active segment signal; The filtered active segment signal is normalized with the maximum and minimum values ​​of the offset, and the active segment signal is normalized to The interval expression is: ， In the formula, is the normalized signal, is the sample offset, is the maximum value of the signal, is the minimum value of the signal, is the maximum offset; The normalized signal , and the normalized signal Multiply the transpose of to get the normalized symmetric matrix, the expression is: ， In the formula, is the normalized symmetric matrix, is the transpose of the normalized signal; Feature extraction is performed on the multimodal motion signal of the neck, specifically: According to the improved two-dimensional discrete wavelet transform, the symmetric matrix Decompose into approximate coefficients at different scales and directions and detail factor , the expression is: ， ， ， ， In the formula, For the The layer approximation coefficients are at positions The value of is the dimension size of the symmetric matrix, is a symmetric matrix in position The value of is the Xmvfo wavelet function, is the index of the scale, is the index of the position, For the Layer detail factor at position The value of is the adjoint wavelet function of Xmvfo, is the dynamic factor of the wavelet function, is the bandwidth parameter of the wavelet, is a positional parameter, is the frequency parameter of the wavelet; By combining the approximation coefficients and detail factor Reconstruct the symmetric matrix, the expression is: ， In the formula, is the scattering coefficient matrix after wavelet reconstruction, is the scale factor, which determines the scale level of decomposition. is the direction quantity factor, representing the number of decomposition directions; The acquired real-time neck multimodal motion signal is input into the neck motion recognition model, and the neck motion recognition model outputs a neck motion recognition result. The wheelchair is controlled to execute a walking command according to the neck motion recognition result.

2. The adaptive wheelchair control method based on neck multimodal motion signals according to claim 1, characterized in that: The step of acquiring a neck electromyographic signal and a neck strain signal, and constructing a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal comprises: Acquire the neck electromyographic signal and neck strain signal generated by the neck muscle group when the user moves his neck, wherein the neck electromyographic signal is acquired by electromyographic sensors symmetrically placed on the upper trapezius muscle, with a total of two channels, and the neck strain signal is acquired by flexible strain sensors placed on the upper trapezius muscle, with a total of four channels; A neck multimodal motion signal is constructed according to the neck electromyographic signal and the neck strain signal. The expression of the neck multimodal motion signal is: ， In the formula, For the Channel signals, where channels 1 and 2 are neck electromyography signals, and channels 3, 4, 5, and 6 are neck strain signals. is the multimodal motion signal of the neck composed of all channels, is the sampling time.

3. The adaptive wheelchair control method based on neck multimodal motion signals according to claim 1, characterized in that: The multimodal motion signal of the neck is classified and identified, specifically: Construct a scattering coefficient feature extraction module, which includes three feature extraction units in sequence, and each feature extraction unit includes: a convolution layer, an activation function layer, and a maximum pooling layer in sequence. The convolution kernel size of the convolution layer is 3*3, the step size is 1, the activation function layer is a Relu activation function, and the window size of the maximum pooling layer is 2*2, and the step size is 2; Construct an attention convergence module, which consists of a multi-head attention mechanism, and the expression is: ， In the formula, is the output value of the u-th attention head, is the query matrix, is the key-value matrix, is the output value matrix, is the offset parameter, is the scaling parameter, is the query matrix dimension; Construct a target domain classification module, which includes 64 LSTM hidden units and a dimension of Fully connected layers; Constructing a weighted cross entropy loss function with L2 parameter penalty , the expression is: ， In the formula, is the regularization coefficient, is the number of network layers, For the The weight matrix of the layer, is the weighted cross entropy loss function; Based on the scattering coefficient feature extraction module, the attention convergence module, the target domain classification module and the weighted cross entropy loss function An LclaNet network is constructed, and the multimodal motion signal of the neck is classified and identified according to the LclaNet network.

4. An adaptive wheelchair control system based on multimodal neck motion signals, characterized in that: include: An acquisition module is configured to acquire a neck electromyographic signal and a neck strain signal, and construct a neck multimodal motion signal according to the neck electromyographic signal and the neck strain signal; The training module is configured to train the RTNMRM network model according to the neck multimodal motion signal using the stochastic gradient descent method, calculate the loss function, and update the network model parameters until the loss function converges to obtain a trained neck motion recognition model. Among them, before training the RTNMRM network model using the stochastic gradient descent method according to the multimodal motion signal of the neck, the method further includes: The multimodal motion signal of the neck is preprocessed, feature extracted, and classified and identified, specifically: Calculate the average value of all sampling points within 2 seconds in front of the neck, and record it as , 10 consecutive sampling points are greater than the average value The signal is taken as the starting active point, and 10 consecutive sampling points are all less than the average value. The signal of the starting activity point and the ending activity point constitute the signal in the activity time window , the expression is: ， In the formula, The time point to terminate the activity. is the time point at which the activity starts. is the longest activity period, is the signal in the active time window, is the multimodal motion signal of the neck composed of all channels, is the multimodal motion signal of the neck at the starting point, Multimodal motion signals of the neck for the stop activity point; Perform empirical mode decomposition on the signals of all channels in the active time window, and the expression is: ， In the formula, is the mode number, is the total number of modes, For the Channel No. Modal components, is the residual signal; The first four IMF components are added in pairs to calculate the filtered active segment signal, which is expressed as: ， In the formula, is the filtered active segment signal; The filtered active segment signal is normalized with the maximum and minimum values ​​of the offset, and the active segment signal is normalized to The interval expression is: ， In the formula, is the normalized signal, is the sample offset, is the maximum value of the signal, is the minimum value of the signal, is the maximum offset; The normalized signal , and the normalized signal Multiply the transpose of to get the normalized symmetric matrix, the expression is: ， In the formula, is the normalized symmetric matrix, is the transpose of the normalized signal; Feature extraction is performed on the multimodal motion signal of the neck, specifically: According to the improved two-dimensional discrete wavelet transform, the symmetric matrix Decompose into approximate coefficients at different scales and directions and detail factor , the expression is: ， ， ， ， In the formula, For the The layer approximation coefficients are at positions The value of is the dimension size of the symmetric matrix, is a symmetric matrix in position The value of is the Xmvfo wavelet function, is the index of the scale, is the index of the position, For the Layer detail factor at position The value of is the adjoint wavelet function of Xmvfo, is the dynamic factor of the wavelet function, is the bandwidth parameter of the wavelet, is a positional parameter, is the frequency parameter of the wavelet; By combining the approximation coefficients and detail factor Reconstruct the symmetric matrix, the expression is: ， In the formula, is the scattering coefficient matrix after wavelet reconstruction, is the scale factor, which determines the scale level of decomposition. is the direction quantity factor, representing the number of decomposition directions; The output module is configured to input the acquired real-time neck multimodal motion signal into the neck motion recognition model, the neck motion recognition model outputs the neck motion recognition result, and controls the wheelchair to execute the walking command according to the neck motion recognition result.

5. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images