Multi-modal signal-based action recognition method, device, equipment and program product

Abstract

The invention relates to the technical field of action detection, and provides an action recognition method and device based on a multi-modal signal, electronic equipment and a computer program product. The method comprises the following steps: acquiring personal information of a user and inertial signals and surface electromyogram signals of body parts of the user; selecting at least one signal from the inertial signal and the surface electromyogram signal as a target signal according to the personal information; and determining an action recognition result of the user according to the target signal. By adopting the method, the target signal for recognizing the human body action can be reasonably selected from the inertial signal and the surface electromyogram signal according to the personal information of the user, so that the accuracy of action recognition is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of motion detection, and particularly to a method, device, electronic device and computer program product for action recognition based on multi-modal signals. Background Art

[0002] As a convenient human-computer interaction method, the limb movements of the human body are applied in more and more scenarios, and action recognition has become a current research hotspot for technicians. Since the surface electromyogram signal of the human body can reflect muscle contraction, the flexion and extension of human joints, as well as the shape and position of the limbs, the recognition of human actions can be achieved by collecting the surface electromyogram signals of body parts such as the arms or feet of the human body. However, the surface electromyogram signal has obvious signal differences and is affected by personal factors such as the age, gender and skin condition of the user. Personal factors will significantly increase the error of the collected surface electromyogram signal in some cases, thus resulting in a low accuracy of action recognition. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a method, device, electronic device and computer program product for action recognition based on multi-modal signals, which can improve the accuracy of action recognition.

[0004] The first aspect of the embodiments of the present application provides a method for action recognition based on multi-modal signals, including:

[0005] Obtaining personal information of the user, inertial signals and surface electromyogram signals of the user's body parts;

[0006] Selecting at least one signal from the inertial signals and surface electromyogram signals as the target signal according to the personal information;

[0007] Determining the action recognition result of the user according to the target signal.

[0008] In the technical solution of the embodiments of the present application, the inertial signals and surface electromyogram signals of the user's body parts such as the arms and feet are collected together, and the personal information of the user is also obtained. According to the personal information, at least one signal is selected from the inertial signals and surface electromyogram signals as the target signal, and finally the action recognition result of the user is determined according to the target signal. By setting like this, the target signal for identifying human actions can be reasonably selected from the inertial signals and surface electromyogram signals according to the personal information of the user. For example, assuming that the personal factors of the user will cause a large error in the collected surface electromyogram signal, the inertial signal can be selected as the target signal, or the surface electromyogram signal and the inertial signal can be combined as the target signal together. Since the inertial signal is not easily affected by personal factors such as the age, gender and skin condition of the user, such processing can effectively improve the accuracy of action recognition.

[0009] In one implementation of the embodiment of the present application, the personal information includes the age of the user; according to the personal information, at least one signal is selected from the inertial signal and the surface electromyogram signal as the target signal, including:

[0010] If the age is within the preset range, the surface electromyogram signal is used as the target signal, or both the inertial signal and the surface electromyogram signal are used as the target signal;

[0011] If the age is outside the preset range, the inertial signal is used as the target signal.

[0012] In another implementation of the embodiment of the present application, the personal information includes the skin condition of the body part; according to the personal information, at least one signal is selected from the inertial signal and the surface electromyogram signal as the target signal, including:

[0013] If the skin condition is normal skin, the surface electromyogram signal is used as the target signal, or both the inertial signal and the surface electromyogram signal are used as the target signal;

[0014] If the skin condition is abnormal skin, the inertial signal is used as the target signal.

[0015] In one implementation of the embodiment of the present application, according to the target signal, the action recognition result of the user is determined, including:

[0016] Extract the signal feature value of the target signal;

[0017] Determine the target range interval where the signal feature value is located;

[0018] Based on the correspondence relationship between each preset range interval and each preset action category stored in advance, determine the action recognition result according to the target range interval.

[0019] In one implementation of the embodiment of the present application, determining the target range interval where the signal feature value is located includes:

[0020] If the target signal is a surface electromyogram signal, obtain the muscle fatigue degree at the current moment and the body part;

[0021] According to the current moment, the body part and the muscle fatigue degree, determine the target range interval where the signal feature value is located.

[0022] In one implementation of the embodiment of the present application, the inertial signal is obtained in the following manner:

[0023] Through the IIC interface, receive the inertial signal collected by the multi-axis inertial sensor arranged on the body part.

[0024] In one implementation manner of the embodiments of the present application, after determining the action recognition result of the user according to the target signal, it further includes:

[0025] Generating a control instruction according to the action recognition result;

[0026] Performing operation control on the controlled device based on the control instruction.

[0027] The second aspect of the embodiments of the present application provides an action recognition device based on multimodal signals, including:

[0028] A signal acquisition module, configured to acquire the personal information of the user, the inertial signal and the surface electromyogram signal of the user's body part;

[0029] A signal selection module, configured to select at least one signal from the inertial signal and the surface electromyogram signal as the target signal according to the personal information;

[0030] An action recognition module, configured to determine the action recognition result of the user according to the target signal.

[0031] The third aspect of the embodiments of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the action recognition method based on multimodal signals provided in the first aspect of the embodiments of the present application.

[0032] The fourth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on an electronic device, it causes the electronic device to execute the action recognition method based on multimodal signals provided in the first aspect of the embodiments of the present application.

[0033] The fifth aspect of the embodiments of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the action recognition method based on multimodal signals provided in the first aspect of the embodiments of the present application.

[0034] It can be understood that the beneficial effects of the above second aspect to the fifth aspect can refer to the relevant descriptions in the above first aspect, and will not be elaborated here. Description of the Drawings

[0035] Figure 1 is a flowchart of an action recognition method based on multimodal signals provided by an embodiment of the present application;

[0036] Figure 2 is a structural framework diagram of a wearable device provided by an embodiment of the present application;

[0037] Figure 3It is a schematic connection diagram of an inertial sensor and a main control module provided by an embodiment of the present application;

[0038] Figure 4 It is a schematic operation flow diagram of action recognition and device control based on inertial signals provided by an embodiment of the present application;

[0039] Figure 5 It is a schematic structural diagram of an action recognition device based on multi-modal signals provided by an embodiment of the present application;

[0040] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0041] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application. Additionally, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0042] Limb movements such as human gestures can convey people's thoughts, emotions, and certain command information. The action recognition technology based on surface electromyogram signals classifies and recognizes the motion intentions of users, making the interaction between people and various intelligent devices more natural and convenient, and has been widely applied. However, due to the characteristics of surface electromyogram signals such as weakness, non-stationarity, and poor anti-interference ability, and being easily affected by the surrounding environment and personal factors of users, the accuracy of action recognition is relatively low in some cases.

[0043] To address the above technical problems, the embodiments of the present application propose an action recognition method, device, electronic device, and computer program product based on multi-modal signals. By combining surface electromyogram signals and inertial signals, more accurate action recognition can be achieved. For more specific technical implementation details of the embodiments of the present application, please refer to the various method embodiments described below.

[0044] It should be understood that the execution subject of each method embodiment proposed in this application can be various types of electronic devices. For example, it can be a mobile phone, a tablet computer, a desktop computer, a wearable device, a vehicle-mounted terminal, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a large-screen TV, etc. The specific type of the electronic device is not limited in the embodiments of this application.

[0045] Please refer to Figure 1 , which shows an action recognition method based on multi-modal signals provided by an embodiment of this application, including:

[0046] 101. Obtain the personal information of the user, the inertial signal and the surface electromyogram signal of the user's body part;

[0047] First, obtain the personal information of the user, the inertial signal and the surface electromyogram signal of the user's body part. Among them, the personal information may include, but is not limited to: age, gender, race, and the skin conditions of each body part, etc. Inertial sensors and electromyogram signal acquisition chips can be installed on the designated body parts of the user (such as the arm, foot or torso, etc.). When the user makes a certain action, the corresponding inertial signal can be collected by using the inertial sensor, and the corresponding surface electromyogram signal can be collected by using the electromyogram signal acquisition chip.

[0048] In an implementation manner of the embodiment of this application, the inertial signal is obtained in the following way:

[0049] Receive the inertial signal collected by the multi-axis inertial sensor disposed on the body part through the IIC interface.

[0050] A multi-axis inertial sensor can be installed on the designated body part of the user to collect the multi-axis inertial signal of the designated body part. For example, a nine-axis inertial sensor can be installed, which includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, and is respectively used to collect the acceleration, angular velocity, and magnetic field strength in the X, Y, and Z directions in real time to judge the pose, speed, and direction information of the user's limb movement. The multi-axis inertial sensor can be connected to the main control module MCU through the IIC interface, and the interrupt control can be realized by means of embedded code, etc. The data of the three-axis accelerometer, the three-axis gyroscope, and the three-axis magnetometer can be stored in different registers respectively. The main control module MCU can obtain the inertial signal including acceleration, angular velocity, and magnetic field strength by reading the corresponding registers.

[0051] As an example, the execution subject of the method embodiment can be a certain wearable device, and the structural framework of the wearable device is as follows Figure 2 shown. Please refer to Figure 2 . The wearable device is composed of a main control module MCU, an inertial sensor, and an electromyogram signal acquisition chip. The inertial sensor can transmit and communicate data with the main control module MCU through communication protocols such as IIC. The electromyogram signal acquisition chip can be sealed together with the main control module MCU to improve the signal transmission efficiency. In addition, the wearable device can also include a power module and a clock module. When the user wears the wearable device and makes a certain action, the inertial sensor can collect the corresponding inertial signal and transmit it to the main control module MCU. The electromyogram signal acquisition chip can collect the corresponding surface electromyogram signal and transmit it to the main control module MCU. Then, the main control module MCU processes, classifies, and completes action recognition on the inertial signal and the surface electromyogram signal. Finally, through communication protocols such as RS485, it can control the controlled device to complete corresponding operations, such as controlling the manipulator to make corresponding gesture actions, and so on.

[0052] Figure 3 is a schematic connection diagram of an inertial sensor and a main control module provided by an embodiment of the present application. The inertial sensor is connected to the main control module MCU through the IIC communication protocol. Among them, the SDA data line of the inertial sensor is connected to the general-purpose input / output port GPIO B9 of the main control module MUC, the SCL serial clock line of the inertial sensor is connected to the general-purpose input / output port GPIO B8 of the main control module MUC, and the SA0 address selection line of the inertial sensor is connected to the general-purpose input / output port GPIO C13 of the main control module MUC. By using the SA0 address selection line, the IIC address of the inertial sensor can be configured. For example, when SA0 is at a low level, the IIC address of the inertial sensor is set to 0x68, and when SA0 is at a high level, the IIC address of the inertial sensor is set to 0x69. The SDA data line is a serial data line used to transmit communication data, including slave device addresses, read / write directions, data bytes, and acknowledgment signals, etc. The SCL serial clock line provides a communication clock signal for synchronizing the data transmission between the master device and the slave device. The master device generates clock pulses, and the slave device synchronizes to send or receive data according to the clock signal.

[0053] The electromyogram signal acquisition chip combined with the gel flexible electrode can send the collected surface electromyogram signal to the main control module MCU. In addition, the inertial sensor can send the collected relationship signal to the main control module MCU through the IIC interface. Then, the main control module MCU processes, classifies, and performs action recognition on the received inertial signal and surface electromyogram signal, and controls the controlled device (such as a manipulator) based on the action recognition result.

[0054] Due to the thermal motion of internal electronic components in the inertial sensor and the internal magnetic field of the device, etc., high-frequency small-amplitude random oscillations and drifts and other noises will occur in the inertial signal. In order to further improve the accuracy of action recognition, corresponding denoising processing can be performed on the inertial signal. Generally speaking, since the frequency of the limb movement signal is between 1 Hz and 5 Hz, while the frequency of the high-frequency noise signal is between dozens of Hz and hundreds of Hz, an IIR low-pass filter can be used to filter the inertial signal to filter out high-frequency noises such as mechanical vibrations and electromagnetic interference, and extract the low-frequency effective signal. The cut-off frequency of the IIR low-pass filter can be set to 5 Hz, and the general expression is where M and N represent the parameters related to the order of the filter. In actual operation, a first-order IIR low-pass filter can be used, that is, let M = N = 1.

[0055] 102. Select at least one signal from the inertial signal and the surface electromyogram signal as the target signal according to the personal information;

[0056] According to the user's personal information, the target signal for identifying human actions can be reasonably selected from the inertial signal and the surface electromyogram signal. For example, assuming that the user's personal factors cause large errors in the collected surface electromyogram signal, the inertial signal can be selected as the target signal, or the surface electromyogram signal and the inertial signal can be combined as the target signal. Another example, assuming that the user's personal factors do not cause large errors in the collected surface electromyogram signal, the surface electromyogram signal can be selected as the target signal, or the surface electromyogram signal and the inertial signal can be combined as the target signal.

[0057] In an implementation manner of the embodiment of the present application, the personal information includes the age of the user; selecting at least one signal from the inertial signal and the surface electromyogram signal as the target signal according to the personal information includes:

[0058] (1) If the age is within the preset range, the surface electromyogram signal is used as the target signal, or both the inertial signal and the surface electromyogram signal are used as the target signal;

[0059] (2) If the age is outside the preset range, the inertial signal is used as the target signal.

[0060] If the age of the user is within a certain preset range, for example, between 16 and 60, it indicates that the user's skin and muscle tissues are in a normal state. At this time, the surface electromyogram signal collected is relatively accurate. Therefore, the surface electromyogram signal can be used alone as the target signal, or both the inertial signal and the surface electromyogram signal can be used as the target signals. On the contrary, if the age of the user is outside this preset range, it means that the user is a minor or an elderly person. The muscle generation of minors is not yet complete, and the elderly may have phenomena such as wrinkled skin, which will all lead to a decrease in the accuracy rate of the surface electromyogram signal. Therefore, only the inertial signal is used alone as the target signal. By setting like this, the target signal for identifying human movements can be reasonably selected according to the age of the user.

[0061] In another implementation manner of the embodiment of the present application, the personal information includes the skin condition of the body part; according to the personal information, selecting at least one signal from the inertial signal and the surface electromyogram signal as the target signal includes:

[0062] (1) If the skin condition is normal skin, then use the surface electromyogram signal as the target signal, or use both the inertial signal and the surface electromyogram signal as the target signals;

[0063] (2) If the skin condition is abnormal skin, then use the inertial signal as the target signal.

[0064] If the skin condition of the user is normal skin, that is, there is no skin disease or skin damage, etc., the surface electromyogram signal collected is relatively accurate. Therefore, the surface electromyogram signal can be used alone as the target signal, or both the inertial signal and the surface electromyogram signal can be used as the target signals. On the contrary, if the skin condition of the user is abnormal skin, that is, there is skin disease or skin damage, etc., it will lead to a decrease in the accuracy rate of the surface electromyogram signal. Therefore, only the inertial signal is used alone as the target signal. By setting like this, the target signal for identifying human movements can be reasonably selected according to the skin condition of the user.

[0065] 103. Determine the action recognition result of the user according to the target signal.

[0066] After selecting the target signal, the features of the target signal are extracted and analyzed to determine the user's action recognition result. Specifically, the correspondence between each preset action category and the features of the target signal can be pre-stored, so that the corresponding action category can be inferred based on the features of the target signal, thereby obtaining the user's action recognition result. For example, assuming the target signal is an inertial signal, if both the acceleration and angular velocity are close to 0, such as the average acceleration is less than 0.1g and the angular velocity is less than 5° / s, it can be considered that the user's action is stationary; if the acceleration changes significantly and the angular velocity is close to 0, it can be considered that the user's action is translation; if the angular velocity changes significantly, such as the angular velocity change exceeds 50° / s, it can be considered that the user's action is rotation; for complex actions, they can be composed of multiple basic actions such as grasping, translation, and rotation.

[0067] As an example, assume the target signal is a surface electromyogram signal. The surface electromyogram signal obtained by using a flexible electrode is the comprehensive potential generated by the relevant muscle movement at the detection point, which can sensitively transmit limb information, and even non-obvious limb movements can achieve good recognition effects. However, the surface electromyogram signal is a weak non-stationary signal and is itself vulnerable to environmental interference.

[0068] As another example, assume the target signal is an inertial signal. By utilizing the characteristics that the inertial signal is not easily affected by environmental interference and user personal factors, high-precision user action recognition can be achieved.

[0069] As yet another example, assume the target signal includes a surface electromyogram signal and an inertial signal. Then the target signal is a multi-modal signal. By utilizing the characteristics that the surface electromyogram signal is more sensitive to detecting fine hand movements and the inertial signal has advantages in detecting large-scale movements, the accuracy of action recognition can be fully improved.

[0070] In one implementation manner of the embodiment of the present application, determining the user's action recognition result according to the target signal includes:

[0071] (1) Extracting the signal feature value of the target signal;

[0072] (2) Determining the target range interval where the signal feature value is located;

[0073] (3) Based on the correspondence between each pre-stored preset range interval and each preset action category, determining the action recognition result according to the target range interval.

[0074] When determining the action recognition result based on the target signal, first extract the signal feature values of the target signal. For example, for the electromyogram signal, extract feature values such as root mean square, variance, waveform length, etc., and for the inertial signal, extract feature values such as average value, waveform length, instantaneous peak value, slope sign change, and zero crossing point, etc. Then, determine the target range interval where the signal feature value is located. The corresponding relationship between each preset range interval and each preset action category is stored in advance, so that the corresponding action category can be found according to the target range interval, thereby determining the action recognition result.

[0075] In one implementation manner of the embodiment of the present application, determining the target range interval where the signal feature value is located includes:

[0076] (1) If the target signal is a surface electromyogram signal, obtain the muscle fatigue degree at the current moment and the body part.

[0077] (2) Determine the target range interval where the signal feature value is located according to the current moment, the body part, and the muscle fatigue degree.

[0078] Even for the same human body, there will be significant signal differences in the surface electromyogram signals collected under different moments, different muscle fatigue degrees, and different body parts. Therefore, if the target signal is a surface electromyogram signal, obtain the muscle fatigue degree at the current moment and the specified body part, and then determine the target range interval where the signal feature value is located according to the current moment, the specified body part, and the muscle fatigue degree, that is, different moments, different body parts, and different muscle fatigue degrees can respectively correspond to different target range intervals. By setting like this, it is possible to adaptively adjust the target range interval where the signal feature value is located according to different moments, different body parts, and different muscle fatigue degrees, thereby further improving the accuracy of action recognition.

[0079] In one implementation manner of the embodiment of the present application, after determining the action recognition result of the user based on the target signal, it further includes:

[0080] (1) Generate a control command according to the action recognition result;

[0081] (2) Perform operation control on the controlled device based on the control command.

[0082] After determining the action recognition result of the user based on the target signal, a corresponding control command can be generated according to this action recognition result to perform operation control on a certain controlled device. For example, assuming that the action recognition result is a grasping action and the controlled device is a certain manipulator, a corresponding operation command can be sent to the manipulator to control the manipulator to perform the grasping action.

[0083] As an example, Figure 4It is a schematic diagram of an operation process for realizing action recognition and device control based on inertial signals provided by an embodiment of the present application. In Figure 4 when a user makes a certain gesture action, the inertial sensor can collect the corresponding inertial signals and send the inertial signals to the main control module MCU through the IIC interface; the main control module MCU processes, classifies, and recognizes the inertial signals, and outputs corresponding control signals to the manipulator based on the action recognition result to control the manipulator to make the same gesture action.

[0084] In the technical solution of the embodiment of the present application, the inertial signals and surface electromyography signals of the user's body parts such as the arms and feet are collected together. In addition, the personal information of the user is obtained, and at least one signal is selected from the inertial signals and surface electromyography signals as the target signal according to the personal information. Finally, the action recognition result of the user is determined according to the target signal. By setting like this, the target signal for identifying human actions can be reasonably selected from the inertial signals and surface electromyography signals according to the personal information of the user. For example, assuming that the personal factors of the user cause large errors in the collected surface electromyography signals, the inertial signals can be selected as the target signal, or the surface electromyography signals and inertial signals can be combined as the target signal together. Since the inertial signals are not easily affected by personal factors such as the age, gender, and skin condition of the user, such processing can effectively improve the accuracy of action recognition.

[0085] In summary, the embodiment of the present application proposes an action recognition method based on multi-modal signals. By combining surface electromyography signals and inertial signals for processing, classification, and recognition, the accuracy of action recognition can be effectively improved.

[0086] It should be understood that the magnitudes of the sequence numbers of the steps in the above respective embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

[0087] The above mainly describes an action recognition method based on multi-modal signals. Next, an action recognition device based on multi-modal signals will be described.

[0088] Please refer to Figure 5 , which shows an action recognition device based on multi-modal signals provided by an embodiment of the present application, including:

[0089] A signal acquisition module 501, configured to acquire the personal information of the user, the inertial signals and surface electromyography signals of the user's body parts;

[0090] A signal selection module 502, configured to select at least one signal from the inertial signals and surface electromyography signals as the target signal according to the personal information;

[0091] The action recognition module 503 is configured to determine the action recognition result of the user according to the target signal.

[0092] In an implementation manner of the embodiment of the present application, the personal information includes the age of the user; the signal selection module includes:

[0093] The first signal selection unit is configured to, if the age is within a preset range, use the surface electromyogram signal as the target signal, or use both the inertial signal and the surface electromyogram signal as the target signals;

[0094] The second signal selection unit is configured to, if the age is outside the preset range, use the inertial signal as the target signal.

[0095] In another implementation manner of the embodiment of the present application, the personal information includes the skin condition of the body part; the signal selection module includes:

[0096] The third signal selection unit is configured to, if the skin condition is normal skin, use the surface electromyogram signal as the target signal, or use both the inertial signal and the surface electromyogram signal as the target signals;

[0097] The fourth signal selection unit is configured to, if the skin condition is abnormal skin, use the inertial signal as the target signal.

[0098] In an implementation manner of the embodiment of the present application, the action recognition module includes:

[0099] The feature extraction unit is configured to extract the signal feature value of the target signal;

[0100] The range determination unit is configured to determine the target range interval where the signal feature value is located;

[0101] The action recognition unit is configured to, based on the correspondence between each preset range interval and each preset action category stored in advance, determine the action recognition result according to the target range interval.

[0102] In an implementation manner of the embodiment of the present application, the range determination unit includes:

[0103] The parameter acquisition subunit is configured to, if the target signal is a surface electromyogram signal, acquire the muscle fatigue degree at the current moment and the body part;

[0104] The range determination subunit is configured to determine the target range interval where the signal feature value is located according to the current moment, the body part, and the muscle fatigue degree.

[0105] In an implementation manner of the embodiment of the present application, the signal acquisition module includes:

[0106] A signal receiving unit, configured to receive inertial signals collected by a multi-axis inertial sensor disposed on a body part through an IIC interface.

[0107] In an implementation manner of an embodiment of the present application, the action recognition device based on multi-modal signals further includes:

[0108] An instruction generation module, configured to generate a control instruction according to an action recognition result;

[0109] A device control module, configured to perform operation control on a controlled device based on the control instruction.

[0110] An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the action recognition method based on multi-modal signals described in any one of the above embodiments is implemented.

[0111] An embodiment of the present application further provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute the action recognition method based on multi-modal signals described in any one of the above embodiments.

[0112] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the embodiments of the above various action recognition methods based on multi-modal signals are implemented, for example Figure 1 the steps 101-step 103 shown. Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above device embodiments are implemented, for example, implementing Figure 5 the functions of module 501-module 503 of the device shown.

[0113] The computer program 62 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the electronic device 6.

[0114] The so-called processor 60 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0115] The memory 61 may be an internal storage unit of the electronic device 6, such as the hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk equipped on the electronic device 6, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 61 may also include both the internal storage unit of the electronic device 6 and the external storage device. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 may also be used to temporarily store the data that has been output or will be output.

[0116] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0117] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0118] In the above embodiments, the descriptions of the respective embodiments each have their own emphasis. For parts not described in detail or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0119] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0120] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this application.

[0122] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0124] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A method for action recognition based on multimodal signals, characterized in that Including: Obtaining personal information of the user, inertial signals and surface electromyography signals of the body part of the user; Selecting at least one signal from the inertial signals and the surface electromyography signals as a target signal according to the personal information; Determining an action recognition result of the user according to the target signal.

2. The method according to claim 1, wherein, The personal information includes the age of the user; the selecting at least one signal from the inertial signals and the surface electromyography signals as a target signal according to the personal information includes: If the age is within a preset range, taking the surface electromyography signal as the target signal, or taking both the inertial signal and the surface electromyography signal as the target signal; If the age is outside the preset range, taking the inertial signal as the target signal.

3. The method according to claim 1, wherein The personal information includes the skin condition of the body part; the selecting at least one signal from the inertial signals and the surface electromyography signals as a target signal according to the personal information includes: If the skin condition is normal skin, taking the surface electromyography signal as the target signal, or taking both the inertial signal and the surface electromyography signal as the target signal; If the skin condition is abnormal skin, taking the inertial signal as the target signal.

4. The method according to claim 1, wherein The determining an action recognition result of the user according to the target signal includes: Extracting a signal feature value of the target signal; Determining a target range interval where the signal feature value is located; Based on the corresponding relationship between each preset range interval and each preset action category stored in advance, determining the action recognition result according to the target range interval.

5. The method according to claim 4, wherein The determining a target range interval where the signal feature value is located includes: If the target signal is the surface electromyography signal, obtaining the current time and the muscle fatigue degree of the body part; Determining a target range interval where the signal feature value is located according to the current time, the body part and the muscle fatigue degree.

6. The method according to claim 1, wherein The inertial signal is obtained in the following manner: Receiving the inertial signal collected by a multi-axis inertial sensor disposed on the body part through an IIC interface.

7. The method according to any one of claims 1 to 6, characterized in that, After the determining an action recognition result of the user according to the target signal, it further includes: Generating a control instruction according to the action recognition result; Performing operation control on a controlled device based on the control instruction.

8. An action recognition device based on multimodal signals, characterized in that Including: A signal acquisition module, configured to obtain personal information of the user, inertial signals and surface electromyography signals of the body part of the user; A signal selection module, configured to select at least one signal from the inertial signals and the surface electromyography signals as a target signal according to the personal information; An action recognition module, configured to determine an action recognition result of the user according to the target signal.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the action recognition method based on multi-modal signals according to any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program product runs on an electronic device, it causes the electronic device to execute the action recognition method based on multi-modal signals according to any one of claims 1 to 7.

Images