Method for determining a human posture and a device for determining a human posture

Abstract

The application provides a human posture determination method and a determination device thereof. The method comprises the following steps: firstly, acquiring first sensor group data, second sensor group data and third sensor group data, wherein the first sensor group data is used for representing data of multiple sensors corresponding to EMG, the second sensor group data is used for representing data of multiple sensors corresponding to EIT, and the third sensor group data is used for representing data of multiple sensors corresponding to FMG; then, performing data enhancement processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data; finally, inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of a target object. The accuracy of the target posture data is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human posture, in particular, to a human posture determination method, a human posture determination device, a computer readable storage medium, a processor and a human posture determination system. BACKGROUND

[0002] Currently, gesture and other human posture estimation is mainly performed by Leap Motion (a body sensing controller) or Kinect, which is an optical-based sensing method. However, the optical-based solution has the difficulties of easy occlusion, high cost, and difficult large-scale deployment. In addition, there is also a solution based on biological sensors, but due to the small amount of data and low dimensionality, and the use of traditional machine learning algorithms, the accuracy of posture continuity estimation is low.

[0003] Therefore, there is an urgent need for a human posture determination method with low cost and high accuracy.

[0004] The above information disclosed in the background section is only used to enhance the understanding of the background of the technology described herein, therefore, the background section can contain certain information which is not known to the prior art in the country. SUMMARY

[0005] The main purpose of the present application is to provide a human posture determination method, a human posture determination device, a computer readable storage medium, a processor and a human posture determination system, to solve the problem of high cost and low accuracy in determining human posture in the prior art.

[0006] According to an aspect of some embodiments of the present application, there is provided a method for determining a human pose, the method comprising: obtaining first sensor group data, second sensor group data and third sensor group data, the first sensor group data being indicative of data from a plurality of sensors corresponding to EMG (Electro Magnetic Gun), the second sensor group data being indicative of data from a plurality of sensors corresponding to EIT (Electrical Impedance Tomography), the third sensor group data being indicative of data from a plurality of sensors corresponding to FMG (Force Myography); performing a first predetermined processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data, wherein the first predetermined processing comprises data augmentation processing; inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target pose data of a target object, the neural network model being trained by machine learning using a plurality of predetermined data, each of the plurality of predetermined data comprising first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual pose data, wherein the first historical sensor group data is indicative of historical data from a plurality of sensors corresponding to EMG, the second historical sensor group data is indicative of historical data from a plurality of sensors corresponding to EIT, the third historical sensor group data is indicative of historical data from a plurality of sensors corresponding to FMG.

[0007] Optionally, the first target sensor group data, the second target sensor group data and the third target sensor group data are input into a neural network model to obtain target posture data of the target object, comprising: obtaining the neural network model, wherein the neural network model comprises a first neural network model, a second neural network model, a third neural network model and a fourth neural network model, the first neural network model is trained by machine learning according to at least a plurality of sets of the first historical sensor group data, the second neural network model is trained by machine learning according to at least a plurality of sets of the second historical sensor group data, the third neural network model is trained by machine learning according to at least a plurality of sets of the third historical sensor group data, and the fourth neural network model is trained by machine learning according to at least the historical actual posture data; the first target sensor group data, the second target sensor group data and the third target sensor group data are input into the first neural network model, the second neural network model and the third neural network model one by one to obtain corresponding first posture data, second posture data and third posture data; the first posture data, the second posture data and the third posture data are subjected to a second predetermined processing to obtain corresponding first images, second images and third images, and the second predetermined processing comprises a pooling processing; and the target posture data is determined according to at least the first images, the second images and the third images.

[0008] Optionally, the target posture data is determined according to at least the first images, the second images and the third images, comprising: the first images, the second images and the third images are subjected to a third predetermined processing to obtain fourth images, the third predetermined processing comprises a Concat processing; and the fourth images are input into the fourth neural network model to obtain the target posture data.

[0009] Optionally, the first sensor group data, the second sensor group data and the third sensor group data are obtained, comprising: the first sensor group data, the second sensor group data and the third sensor group data within a predetermined time are obtained, the first sensor group data, the second sensor group data and the third sensor group data are data obtained after a fourth predetermined processing, and the fourth predetermined processing comprises a high-pass filtering and / or a low-pass filtering.

[0010] Optionally, before the first sensor group data, the second sensor group data and the third sensor group data are acquired, the method further comprises: acquiring a plurality of different predetermined actions of the target object under the condition that the target object wears the first sensor group, the second sensor group and the third sensor group, to obtain first predetermined sensor group data, second predetermined sensor group data and third predetermined sensor group data; acquiring a predetermined neural network model; and learning the predetermined neural network model according to at least the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data to obtain the neural network model.

[0011] Optionally, the first neural network model, the second neural network model and the third neural network model each comprise M layers of convolutional layers, and the fourth neural network model comprises N layers of the convolutional layers, where 0 < M < N, and M and N are integers.

[0012] According to another aspect of the embodiments of the present application, a device for determining human posture is further provided. The device comprises a first acquisition unit, a first processing unit and an input unit. The first acquisition unit is configured to acquire first sensor group data, second sensor group data and third sensor group data. The first sensor group data is used to represent data of a plurality of sensors corresponding to EMG. The second sensor group data is used to represent data of a plurality of sensors corresponding to EIT. The third sensor group data is used to represent data of a plurality of sensors corresponding to FMG. The first processing unit is configured to perform first predetermined processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data. The first predetermined processing comprises data enhancement processing. The input unit is configured to input the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of a target object. The neural network model is trained by machine learning using a plurality of groups of predetermined data. Each group of the predetermined data comprises first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data. The first historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EMG. The second historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EIT. The third historical sensor group data is used to represent historical data of a plurality of sensors corresponding to FMG.

[0013] According to still another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which includes a stored program, wherein the program is used to perform any of the methods.

[0014] According to still another aspect of the embodiments of the present application, a processor is also provided, which is used to run a program, wherein the program performs any of the methods when running.

[0015] According to still another aspect of the embodiments of the present application, a human posture determination system is also provided, which includes a controller, a first sensor group, a second sensor group, and a third sensor group, wherein the controller is used to perform any of the methods; the first sensor group is used to provide first sensor group data, which is used to represent data of a plurality of sensors corresponding to EMG; the second sensor group is used to provide second sensor group data, which is used to represent data of a plurality of sensors corresponding to EIT; and the third sensor group is used to provide third sensor group data, which is used to represent data of a plurality of sensors corresponding to FMG.

[0016] In the embodiment of the present application, in the human body posture determination method, first, the first sensor group data, the second sensor group data and the third sensor group data are obtained, the first sensor group data is used to represent the data of a plurality of sensors corresponding to EMG, the second sensor group data is used to represent the data of a plurality of sensors corresponding to EIT, and the third sensor group data is used to represent the data of a plurality of sensors corresponding to FMG; then, the first sensor group data, the second sensor group data and the third sensor group data are respectively subjected to data enhancement processing to obtain first target sensor group data, second target sensor group data and third target sensor group data; finally, the first target sensor group data, the second target sensor group data and the third target sensor group data are input into a neural network model to obtain target posture data of a target object, the neural network model is trained by machine learning using a plurality of groups of predetermined data, each group of data in the plurality of groups of predetermined data includes first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data, wherein the first historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EMG, the second historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EIT, and the third historical sensor group data is used to represent historical data of a plurality of sensors corresponding to FMG. Compared with the high cost and low accuracy of determining the human body posture in the prior art, the human body posture determination method of the present application obtains the first sensor group data, the second sensor group data and the third sensor group data, wherein the first sensor group data is used to represent the data of a plurality of sensors corresponding to EMG, the second sensor group data is used to represent the data of a plurality of sensors corresponding to EIT, and the third sensor group data is used to represent the data of a plurality of sensors corresponding to FMG, that is, the sensor data of three modes of electromyography, myodynamogram and electrical impedance tomography is obtained, and then the first sensor group data, the second sensor group data and the third sensor group data are subjected to the enhancement data processing, so that the first target sensor group data, the second target sensor group data and the third target sensor group data obtained by processing are more consistent with the actual posture of the target object, and then the first target sensor group data, the second target sensor group data and the third target sensor group data are input into the neural network model, so that the target posture data of the target object obtained by the neural network model is obtained by the data of a plurality of sensors corresponding to EMG, EIT and FMG, avoiding the high cost and low accuracy of determining the human body posture in the prior art, and ensuring high accuracy of the target posture data. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are part of this specification, are included to provide a further understanding of the present application, illustrate the preferred embodiments of the present application and explain the principles of the present application. In the drawings:

[0018] Figure 1 A flow chart of a method for determining a human pose is shown according to an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a device for determining a human pose is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] It should be noted that the embodiments and features of the embodiments in the present application can be combined if there is no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0021] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be understood that when an element (such as a layer, film, region, or substrate) is described as "on" another element, it can be directly on the other element, or there can be an intermediate element. Also, in the specification and claims, when an element is described as "connected to" another element, it can be "directly connected to" the other element, or "connected to" the other element through a third element.

[0024] As mentioned in the background, the cost of determining human posture in the prior art is high and the accuracy is low. In a typical embodiment of the present application, a human posture determination method, a human posture determination device, a computer readable storage medium, a processor and a human posture determination system are provided.

[0025] According to an embodiment of the present application, a human posture determination method is provided.

[0026] Figure 1 is a flowchart of a human posture determination method according to an embodiment of the present application. As shown in Figure 1 , the method comprises the following steps:

[0027] Step S101, obtaining first sensor group data, second sensor group data and third sensor group data, the first sensor group data being used to represent data of a plurality of sensors corresponding to EMG, the second sensor group data being used to represent data of a plurality of sensors corresponding to EIT, and the third sensor group data being used to represent data of a plurality of sensors corresponding to FMG;

[0028] Step S102, performing first predetermined processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data, wherein the first predetermined processing comprises data enhancement processing;

[0029] Step S103, inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of a target object, wherein the neural network model is trained by machine learning using a plurality of sets of predetermined data, each set of the predetermined data comprising first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data, wherein the first historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EMG, the second historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EIT, and the third historical sensor group data is used to represent historical data of a plurality of sensors corresponding to FMG.

[0030] The method for determining the human posture comprises the following steps: obtaining first sensor group data, second sensor group data and third sensor group data, wherein the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG; performing data enhancement processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data; and inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of a target object, wherein the neural network model is trained by machine learning using multiple sets of predetermined data, each set of the predetermined data comprises first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data, wherein the first historical sensor group data is used to represent the historical data of multiple sensors corresponding to EMG, the second historical sensor group data is used to represent the historical data of multiple sensors corresponding to EIT, and the third historical sensor group data is used to represent the historical data of multiple sensors corresponding to FMG. Compared with the prior art, the method for determining the human posture has the advantages that the first sensor group data, the second sensor group data and the third sensor group data are obtained, the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG, the data enhancement processing is performed on the first sensor group data, the second sensor group data and the third sensor group data, the first target sensor group data, the second target sensor group data and the third target sensor group data obtained by processing are more consistent with the actual posture of the target object, the first target sensor group data, the second target sensor group data and the third target sensor group data are input into the neural network model, the target posture data of the target object is obtained by the neural network model and the data of multiple sensors corresponding to EMG, EIT and FMG, the problem that the cost of determining the human posture is high and the accuracy is low in the prior art is avoided, and the accuracy of the target posture data is high.

[0031] In particular, the human posture includes determination of the hand gesture of the hand, and in the case of determining the hand gesture of the hand, the sensors corresponding to the EMG, the EIT and the FMG are worn on the hand, of course, the human posture can also be other positions of the human body, that is, the sensors corresponding to the EMG, the EIT and the FMG need to be worn on the corresponding positions, in addition, it can also be other biological bodies other than humans.

[0032] In a specific embodiment, the data enhancement processing includes Gaussian noise processing, and of course, can also include other data enhancement methods such as random flipping, cropping, scaling grayscale, etc.

[0033] At present, EIT is an imaging technology for detecting the internal structure impedance distribution of an object through an external electric excitation signal. EIT places a set of electrodes on the surface of the measured conductive object, applies high-frequency alternating current to each pair of electrodes as an excitation signal, and sequentially measures the electrical response signal on the other electrode pair to obtain the internal resistivity of the object. Due to its advantages of no radiation, no damage, low cost and simple structure, EIT has been widely used in damage detection, geological exploration and other fields. Today, the application of EIT in biomedical imaging and human-computer interaction has been widely studied. It measures the cross-sectional impedance distribution of the target. When the surface electrodes are realized around the target object, EIT sends a high-frequency signal from one electrode and measures the electrical response signal from other electrodes. The difference in signals can restore the impedance distribution of the internal structure. Various gestures will cause different movements of internal muscles and bones and cause changes in internal impedance distribution. Therefore, EIT can monitor gestures by reconstructing the changed impedance. Past gesture recognition using EIT methods show that EIT has high accuracy for gestures similar to muscle contractions. The main application of early EIT is in the medical field. It was first introduced in the early 1980s. However, the relatively bulky and expensive nature of EIT devices has limited this technology to the medical field. The latest technological advancements in sensors and artificial intelligence industries have made EIT a cheaper and more accessible technology. EIT-Kit (Electrical Impedance Tomography Kit) demonstrates rapid prototyping capabilities, can bring complex medical sensing, and demonstrates its potential in the medical field.

[0034] EMG refers to a series of muscle-related electrical signals generated during muscle contraction due to neural control, which is generally given by experimental methods and can represent the physiological characteristics of the muscle after amplification. Electromyography has become a growing field of research for detecting small changes in the human body. For example, the Thalic laboratory has previously developed the MYO armband to detect the millivolts transmitted in the human body. The 8-channel electromyography sensor is wrapped around the user's forearm, and the electromyography electrode can measure the electrical signals generated by muscle activation. When the brain sends a motor control signal through the nerves to activate the muscle, this process will produce ion exchange on the muscle membrane and generate a small current. The electrical signals from the current can be detected by the electromyography sensor. Researchers have examined that increasing the number of electromyography electrodes can improve the accuracy of gesture recognition. More specifically, a higher density of array-like electromyography electrodes (High Definition Surface Electro MyoGraphy, HD-SEMG) provides more comprehensive spatiotemporal features, which is crucial in applications such as gesture recognition. In addition, researchers have shown that deep learning can play an important role in reconstructing gestures. The combination of deep learning techniques can reduce the noise and complexity of signals from different fingers. It shows a promising and accessible way to use electromyography in real-world environments. Most electromyography sensors have insufficient sensor data to directly reconstruct a person's hands. For example, deceleration movements, pointing, and waving hands are considered difficult to be directly captured by low-cost, few-channel electromyography sensors. On the other hand, most multi-channel electromyography sensors are cumbersome, and each channel of electromyography requires a separate channel of amplifier circuitry.

[0035] In addition, FMG is a method of collecting motion signals by sensing changes in muscle volume. Its basic principle is that different muscle activities cause different actions. When the action occurs, the volume of the underlying tendon complex changes, causing changes in the distribution of surface mechanical forces. Different actions are encoded into different force images. By decoding these images, the original motion information can be obtained. Force sensors have become another popular way of wearable gesture recognition. When a user performs a fine motor movement with their hand, the contraction and relaxation of the muscle will cause local pressure changes. Pressure sensors on the wrist can detect the continuous changes in pressure between the device and the muscle. Researchers have shown that pressure-based sensing can help users recover key information about the wrist muscles. Through algorithms, this information can therefore predict the user's gestures. The difference from EMG is that FMG is relatively more stable, with lower variance of FMG signals. FMG also produces better separation patterns, and sweating or wet conditions will not affect its use.

[0036] Of course, when using one of EMG, FMG or EIT alone, only the recognition of a specific gesture can be achieved, and a continuous model prediction cannot be formed. In the multi-modal solution using FMG+EMG, two or more circuit and sensor systems are required, resulting in high deployment cost and poor user experience. In addition, the existing solutions based on biological sensors generally have less data and low dimensionality. As a result of using multi-layer deep learning neural networks, more traditional machine learning algorithms are used. In the process of determining the human posture described above, the data of the three sensors of EMG, FMG and EIT are combined, and a neural network model is used for deep learning, which ensures that the target posture data of the target object described above can be predicted simply and accurately.

[0037] According to one specific embodiment of the present application, the first target sensor group data, the second target sensor group data and the third target sensor group data are input into a neural network model to obtain target posture data of the target object, including: obtaining the neural network model, the neural network model including a first neural network model, a second neural network model, a third neural network model and a fourth neural network model, wherein the first neural network model is trained by machine learning according to at least a plurality of the first historical sensor group data, the second neural network model is trained by machine learning according to at least a plurality of the second historical sensor group data, the third neural network model is trained by machine learning according to at least a plurality of the third historical sensor group data, and the fourth neural network model is trained by machine learning according to at least the historical actual posture data; the first target sensor group data, the second target sensor group data and the third target sensor group data are input into the first neural network model, the second neural network model and the third neural network model one by one to obtain corresponding first posture data, second posture data and third posture data; the first posture data, the second posture data and the third posture data are subjected to a second predetermined processing to obtain corresponding first images, second images and third images, and the second predetermined processing includes a pooling processing; and the target posture data is determined according to at least the first images, the second images and the third images. By obtaining the neural network model including the first neural network model, the second neural network model, the third neural network model and the fourth neural network model, and then inputting the first target sensor group data into the first neural network model, the second target sensor group data into the second neural network model and the third target sensor group data into the third neural network model, the sensor data of EMG, EIT and FMG can be processed by three different neural network models respectively, so as to ensure that the first posture data, the second posture data and the third posture data obtained by processing represent posture data of three different types of sensors respectively, and then the first posture data, the second posture data and the third posture data are subjected to the pooling processing, so that the first images, the second images and the third images obtained by processing have the same size, which is convenient for subsequent processing, and ensures that the target posture data can be determined according to the first images, the second images and the third images relatively simply.

[0038] Specifically, the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into the first neural network model, the second neural network model, and the third neural network model, respectively, and a 64x64 image is formed through a pooling process. Of course, the size of the image can be changed according to actual conditions, as long as the sizes of the first image, the second image, and the third image are the same.

[0039] In a specific embodiment, an RNN (Recurrent Neural Network) or a Transformer can also be selected, where the Transformer is a model that uses an attention mechanism to improve model training accuracy. An MLP (Multilayer Perceptron) can also be selected. The neural network model is selected because it has a low frame rate, is easy to deploy, and has high accuracy.

[0040] To further ensure that the target posture data is highly accurate, according to another specific embodiment of the present application, the target posture data is determined based on at least the first image, the second image, and the third image, including: performing a third predetermined processing on the first image, the second image, and the third image to obtain a fourth image, the third predetermined processing including a Concat processing; and inputting the fourth image into the fourth neural network model to obtain the target posture data. By performing the Concat processing on the first image, the second image, and the third image, the fourth image obtained through processing includes information of the three images corresponding to the first image, the second image, and the third image, ensuring that the fourth image is determined based on image information corresponding to EMG, EIT, and FMG, and ensuring that the accuracy of the fourth image is high. By inputting the fourth image into the fourth neural network model, the accuracy of the target posture data obtained is further ensured to be high.

[0041] Specifically, the three 64x64 images obtained, i.e., the first image, the second image, and the third image, are processed through Concat processing (stacking) to form a 3x64x64 image.

[0042] According to another specific embodiment of the present application, the first sensor group data, the second sensor group data and the third sensor group data are obtained by obtaining the first sensor group data, the second sensor group data and the third sensor group data within a predetermined time, and the first sensor group data, the second sensor group data and the third sensor group data are data obtained after fourth predetermined processing, and the fourth predetermined processing includes high-pass filtering and / or low-pass filtering. By obtaining the first sensor group data, the second sensor group data and the third sensor group data within a predetermined time, and since the first sensor group data, the second sensor group data and the third sensor group data are obtained after high-pass filtering and / or low-pass filtering, it is ensured that the frequency of the first sensor group data, the second sensor group data and the third sensor group data obtained after processing is within the corresponding range, and it is ensured that the first sensor group data, the second sensor group data and the third sensor group data meet the actual requirements, so that the accuracy of the target attitude data determined according to the first sensor group data, the second sensor group data and the third sensor group data is higher, and the accuracy of the target attitude data obtained is further ensured to be higher.

[0043] In a specific embodiment, the EMG includes 16 pairs of double-stage EMG sensors, the EIT includes 16 pairs of double-end sensors, and the FMG includes 32 potential pressure point sensors. The three sensors are simultaneously deployed to the position to be determined of a person, and then real-time data collection is started. The EMG simultaneously collects data of 32 electrodes at a frequency of 300 Hz, the EIT simultaneously collects data of 16 pairs of electrodes at a frequency of 30 Hz, and the FMG collects pressure point data of 32 electrodes at a frequency of 60 Hz. During the collection process, we use a high-pass filter of 20 Hz for the EMG to filter noise in the electrical signal. Of course, a low-pass filter of a specific hertz can also be used for the EMG, and the same applies to the EIT and the FMG. The specific values of the high-pass filter and the low-pass filter are determined according to the actual situation. After the collection is completed, we collect 900 frames of EMG data, 180 frames of pressure data, and 90 frames of EIT data in the past, i.e., we collect data in the past 3 seconds. Of course, in the actual application process, the number of the above-mentioned sensors, the frequency of the collected data, the frequency of the filter and the time of the collected data can be changed according to the actual situation.

[0044] Since the body impedance of each target object and the wearing manner of the sensor are different, the same neural network model has a low accuracy of the calculation result for different target objects. In order to further ensure that the target posture data is high in accuracy, according to a specific embodiment of the present application, before the first sensor group data, the second sensor group data and the third sensor group data are acquired, the method further includes: acquiring a plurality of different predetermined actions of the target object to obtain first predetermined sensor group data, second predetermined sensor group data and third predetermined sensor group data, under the condition that the target object wears the first sensor group, the second sensor group and the third sensor group; acquiring a predetermined neural network model; and learning the predetermined neural network model according to at least the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data to obtain the neural network model. Under the condition that the target object wears the first sensor group, the second sensor group and the third sensor group, the plurality of different predetermined actions of the target object are acquired, that is, the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data corresponding to different predetermined actions are acquired, the predetermined neural network model is acquired, and the predetermined neural network model is learned according to at least the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data, so that the learned neural network model is more in line with the characteristics of different target objects, that is, each target object has its own neural network model, ensuring that the neural network model is high in accuracy, and further ensuring that the target posture data of the target object determined according to the neural network model is high in accuracy.

[0045] In a specific embodiment, the neural network model first outputs N-dimensional angle data, and is calibrated with actual information collected by multiple depth cameras at the same time and optimized by a back propagation algorithm. Through 100 learning processes, learning rates, optimization algorithms and more than 50,000 frames of data, the reconstruction of the posture of a specific part is realized.

[0046] Specifically, the transfer learning of the algorithm is realized in a data-driven manner. We collect tens of thousands of frames of data, classify them in an unsupervised manner, and reduce them to 10 dimensions. We extract the 10-dimensional data as the 10 gestures that are statistically most different. Before use, the system will require each user to wear a sensor and collect data on the 10 gestures for about half a minute and label them. Then, we send these data to our deep learning neural network for further retraining. After reducing the number of training and the learning rate, we limit the optimization space of the model within a certain range to avoid overfitting. That is, the model can better optimize the new target object wearing the multi-modal sensor through the half-minute newly collected learning data, and achieve higher accuracy of specific part gesture recognition. Of course, it is not fixed 10 gestures, but other number of gestures of specific parts, which are mainly determined by actual conditions, and the data collection time can also be extended or shortened according to actual needs.

[0047] According to another specific embodiment of the present application, the first neural network model, the second neural network model and the third neural network model each include M layers of convolutional layers, and the fourth neural network model includes N layers of the convolutional layers, where 0

[0048] In a specific embodiment, M is equal to 5 and N is equal to 18. Of course, M and N can be other numbers of layers, as long as 0

[0049] Specifically, the determination process of the human body posture can be applied to the fields of robot control, prosthetic control, rehabilitation training, medical detection, gesture control in VR (Virtual Reality), etc.

[0050] In addition, in the actual application process, other types of sensors can be added to supplement and further improve the process.

[0051] This application also provides a human posture determination device. It should be noted that the human posture determination device of this application can be used to execute the human posture determination method provided in this application. The human posture determination device provided in this application will be described below.

[0052] Figure 2 This is a schematic diagram of a human posture determination device according to an embodiment of this application. Figure 2 As shown, the device includes a first acquisition unit 10, a first processing unit 20, and an input unit 30. The first acquisition unit 10 acquires data from a first sensor group, a second sensor group, and a third sensor group. The first sensor group data represents data from multiple sensors corresponding to EMG, the second sensor group data represents data from multiple sensors corresponding to EIT, and the third sensor group data represents data from multiple sensors corresponding to FMG. The first processing unit 20 performs a first predetermined processing on the first sensor group data, the second sensor group data, and the third sensor group data to obtain first target sensor group data, second target sensor group data, and third target sensor group data, respectively. This includes data augmentation processing; the input unit 30 is used to input the data from the first target sensor group, the second target sensor group, and the third target sensor group into a neural network model to obtain the target pose data of the target object. The neural network model is trained using multiple sets of predetermined data through machine learning. Each set of predetermined data includes the first historical sensor group data, the second historical sensor group data, the third historical sensor group data, and historical actual pose data. The first historical sensor group data is used to represent the historical data of multiple sensors corresponding to the EMG, the second historical sensor group data is used to represent the historical data of multiple sensors corresponding to the EIT, and the third historical sensor group data is used to represent the historical data of multiple sensors corresponding to the FMG.

[0053] The human body posture determination device includes a first obtaining unit, a first processing unit, and an input unit. The first obtaining unit obtains first sensor group data, second sensor group data, and third sensor group data. The first sensor group data represents data of multiple sensors corresponding to EMG. The second sensor group data represents data of multiple sensors corresponding to EIT. The third sensor group data represents data of multiple sensors corresponding to FMG. The first processing unit performs first predetermined processing on the first sensor group data, the second sensor group data, and the third sensor group data to obtain first target sensor group data, second target sensor group data, and third target sensor group data. The first predetermined processing includes data enhancement processing. The input unit inputs the first target sensor group data, the second target sensor group data, and the third target sensor group data into a neural network model to obtain target posture data of a target object. The neural network model is trained by machine learning using multiple sets of predetermined data. Each set of the predetermined data includes first historical sensor group data, second historical sensor group data, third historical sensor group data, and historical actual posture data. The first historical sensor group data represents historical data of multiple sensors corresponding to EMG. The second historical sensor group data represents historical data of multiple sensors corresponding to EIT. The third historical sensor group data represents historical data of multiple sensors corresponding to FMG. Compared with the prior art, the human body posture determination device can obtain the first sensor group data, the second sensor group data, and the third sensor group data. The first sensor group data represents data of multiple sensors corresponding to EMG. The second sensor group data represents data of multiple sensors corresponding to EIT. The third sensor group data represents data of multiple sensors corresponding to FMG. The first target sensor group data, the second target sensor group data, and the third target sensor group data obtained by performing the data enhancement processing on the first sensor group data, the second sensor group data, and the third sensor group data are more consistent with the actual posture of the target object. The target posture data of the target object obtained by inputting the first target sensor group data, the second target sensor group data, and the third target sensor group data into the neural network model is obtained by the neural network model and the data of the multiple sensors corresponding to EMG, EIT, and FMG. The problem of high cost and low accuracy in determining the human body posture in the prior art is avoided, and the accuracy of the target posture data is ensured.

[0054] Specifically, the above-mentioned human posture includes determination of the gesture of the hand, and in the case where the gesture of the hand needs to be determined, the sensors corresponding to the EMG, EIT and FMG are worn on the hand, of course, the above-mentioned human posture can also be other positions of the human body, that is, the sensors corresponding to the EMG, EIT and FMG need to be worn on the corresponding positions, in addition, it can also be other biological bodies other than humans.

[0055] In a specific embodiment, the data enhancement processing includes Gaussian noise processing, of course, it can also include other data enhancement methods such as random flipping, cropping, scaling grayscale, etc.

[0056] At present, EIT is an imaging technology for detecting the internal structure impedance distribution of an object through an external electric excitation signal. EIT places a group of electrodes on the surface of the measured conductive object, applies high-frequency alternating current as an excitation signal to each pair of electrodes, and sequentially measures the electrical response signal on the other electrode pair, thereby obtaining the internal resistivity object. Due to its advantages of no radiation, no damage, low cost and simple structure, EIT has been widely used in damage detection, geological exploration and other fields. Today, the application of EIT in biomedical imaging and human-computer interaction has been widely studied. It measures the cross-sectional impedance distribution of the target object. When the surface electrodes are realized around the target object, EIT sends a high-frequency signal from one electrode and measures the electrical response signal from other electrodes. The difference in signals can restore the impedance distribution of the internal structure. Various gestures will cause different movements of internal muscles and bones and cause changes in internal impedance distribution. Therefore, EIT can monitor gestures by reconstructing the changed impedance. Past gesture recognition using EIT method shows that EIT has high accuracy for gestures similar to muscle contraction. The main application of early EIT is in the medical field. It was first introduced in the early 1980s. However, the relatively bulky and expensive nature of EIT devices has limited this technology to the medical field. The latest technological advances in sensor and artificial intelligence industries have made EIT a cheaper and more accessible technology. EIT-Kit (Electrical Impedance Tomography Toolkit) demonstrates rapid prototyping capabilities and can bring complex medical sensing and demonstrate its potential in the medical field.

[0057] EMG refers to a series of muscle-related electrical signals generated during muscle contraction due to neural control, which is generally given by experimental methods and can represent the physiological characteristics of the muscle after amplification. Electromyography has become a growing field of research for detecting small changes in the human body. For example, the Thalic laboratory has previously developed the MYO armband to detect the millivolts transmitted in the human body. The 8-channel electromyography sensor is wrapped around the user's forearm, and the electromyography electrode can measure the electrical signals generated by muscle activation. When the brain sends a motor control signal through the nerves to activate the muscle, this process will produce ion exchange on the muscle membrane and generate a small current. The electrical signals from the current can be detected by the electromyography sensor. Researchers have examined that increasing the number of electromyography electrodes can improve the accuracy of gesture recognition. More specifically, a higher density of array-like electromyography electrodes (High Definition Surface Electro MyoGraphy, HD-SEMG) provides more comprehensive spatiotemporal features, which is crucial in applications such as gesture recognition. In addition, researchers have shown that deep learning can play an important role in reconstructing gestures. The combination of deep learning techniques can reduce the noise and complexity of signals from different fingers. It shows a promising and accessible way to use electromyography in real-world environments. Most electromyography sensors have insufficient sensor data to directly reconstruct a person's hands. For example, deceleration movements, pointing, and waving hands are considered difficult to be directly captured by low-cost, few-channel electromyography sensors. On the other hand, most multi-channel electromyography sensors are cumbersome, and each channel of electromyography requires a separate channel of amplifier circuitry.

[0058] In addition, FMG is a method of collecting motion signals by sensing changes in muscle volume. Its basic principle is that different muscle activities cause different actions. When the action occurs, the volume of the underlying tendon complex changes, causing changes in the distribution of surface mechanical forces. Different actions are encoded into different force images. By decoding these images, the original motion information can be obtained. Force sensors have become another popular way of wearable gesture recognition. When a user performs a fine motor movement with their hand, the contraction and relaxation of the muscle will cause local pressure changes. Pressure sensors on the wrist can detect the continuous changes in pressure between the device and the muscle. Researchers have shown that pressure-based sensing can help users recover key information about the wrist muscles. Through algorithms, this information can therefore predict the user's gestures. The difference from EMG is that FMG is relatively more stable, with lower variance in FMG signals. FMG also produces better separation patterns, and sweating or wet conditions will not affect its use.

[0059] Of course, when using one of EMG, FMG or EIT alone, only the recognition of a specific gesture can be achieved, and a continuous model prediction cannot be formed. In the multi-modal solution using FMG+EMG, two or more circuit and sensor systems are required, resulting in high deployment cost and poor user experience. In addition, the existing solutions based on biological sensors generally have less data and low dimensionality. As a result of using multi-layer deep learning neural networks, more traditional machine learning algorithms are used. In the process of determining the human posture described above, the data of the three sensors of EMG, FMG and EIT are combined, and a neural network model is used for deep learning, which ensures that the target posture data of the target object described above can be predicted simply and accurately.

[0060] According to an embodiment of the present application, the input unit comprises a first obtaining module, an input module, a processing module and a determining module. The first obtaining module is configured to obtain the neural network model, wherein the neural network model comprises a first neural network model, a second neural network model, a third neural network model and a fourth neural network model. The first neural network model is trained by machine learning according to at least a plurality of sets of the first historical sensor group data. The second neural network model is trained by machine learning according to at least a plurality of sets of the second historical sensor group data. The third neural network model is trained by machine learning according to at least a plurality of sets of the third historical sensor group data. The fourth neural network model is trained by machine learning according to at least the historical actual posture data. The input module is configured to input the first target sensor group data, the second target sensor group data and the third target sensor group data into the first neural network model, the second neural network model and the third neural network model respectively, to obtain corresponding first posture data, second posture data and third posture data. The processing module is configured to perform a second predetermined processing on the first posture data, the second posture data and the third posture data, to obtain corresponding first image, second image and third image, wherein the second predetermined processing comprises a pooling processing. The determining module is configured to determine the target posture data according to at least the first image, the second image and the third image. By obtaining the neural network model comprising the first neural network model, the second neural network model, the third neural network model and the fourth neural network model, and then inputting the first target sensor group data into the first neural network model, the second target sensor group data into the second neural network model and the third target sensor group data into the third neural network model, the sensor data of EMG, EIT and FMG can be processed by three different neural network models respectively, so as to ensure that the first posture data, the second posture data and the third posture data obtained by processing represent the posture data of three different types of sensors respectively. Then, by performing the pooling processing on the first posture data, the second posture data and the third posture data, the first image, the second image and the third image obtained by processing have the same size, which is convenient for subsequent processing, and ensures that the target posture data can be determined according to the first image, the second image and the third image relatively simply.

[0061] Specifically, the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into the first neural network model, the second neural network model, and the third neural network model, respectively, and a 64x64 image is formed through a pooling process. Of course, the size of the image can be changed according to actual conditions, as long as the sizes of the first image, the second image, and the third image are the same.

[0062] In a specific embodiment, an RNN (Recurrent Neural Network) or a Transformer can also be selected, where the Transformer is a model that uses an attention mechanism to improve model training accuracy, and an MLP (Multilayer Perceptron) can also be selected. The neural network model is selected because it has a low frame rate after implementation, is easy to deploy, and has high accuracy.

[0063] To further ensure that the target posture data has high accuracy, according to another specific embodiment of the present application, the determination module includes a processing submodule and an input submodule, where the processing submodule is configured to perform third predetermined processing on the first image, the second image, and the third image to obtain a fourth image, and the third predetermined processing includes Concat processing; and the input submodule is configured to input the fourth image into the fourth neural network model to obtain the target posture data. By performing the Concat processing on the first image, the second image, and the third image, the fourth image obtained through processing includes information of the three images corresponding to the first image, the second image, and the third image, ensuring that the fourth image is determined based on image information corresponding to EMG, EIT, and FMG, and ensuring that the fourth image has high accuracy. By inputting the fourth image into the fourth neural network model, the accuracy of the target posture data obtained is further ensured to be high.

[0064] Specifically, the three 64x64 images, i.e., the first image, the second image, and the third image, are processed through Concat processing (stacking) to form a 3x64x64 image.

[0065] According to another specific embodiment of the present application, the first obtaining unit comprises a second obtaining module, which is configured to obtain the first sensor group data, the second sensor group data and the third sensor group data within a predetermined time, wherein the first sensor group data, the second sensor group data and the third sensor group data are data obtained after fourth predetermined processing, and the fourth predetermined processing comprises high-pass filtering and / or low-pass filtering. By obtaining the first sensor group data, the second sensor group data and the third sensor group data within a predetermined time, and since the first sensor group data, the second sensor group data and the third sensor group data are obtained after high-pass filtering and / or low-pass filtering, the frequencies of the first sensor group data, the second sensor group data and the third sensor group data obtained after processing are ensured to be within the corresponding ranges, the first sensor group data, the second sensor group data and the third sensor group data are ensured to meet the actual requirements, the accuracy of the target attitude data determined according to the first sensor group data, the second sensor group data and the third sensor group data is higher, and the accuracy of the target attitude data obtained is further ensured to be higher.

[0066] In a specific embodiment, the EMG comprises 16 pairs of double-stage EMG sensors, the EIT comprises 16 pairs of double-end sensors, the FMG comprises 32 potential pressure point sensors, and the three sensors are simultaneously deployed to the position to be determined of a person to start real-time data collection. The EMG simultaneously collects data of 32 electrodes at a frequency of 300 Hz, the EIT simultaneously collects data of 16 pairs of electrodes at a frequency of 30 Hz, and the FMG collects pressure point data of 32 electrodes at a frequency of 60 Hz. During the collection process, we use a high-pass filter of 20 Hz for the EMG to filter noise in the electrical signal. Of course, a low-pass filter of a specific hertz can also be used for the EMG, and the same applies to the EIT and the FMG. The values corresponding to the high-pass filter and the low-pass filter are determined according to the actual situation. After the collection is completed, we collect 900 frames of EMG data in the past, 180 frames of pressure data in the past, and 90 frames of EIT data in the past, i.e., we collect data in the past 3 seconds. Of course, in the actual application process, the number of the sensors, the frequency of the collected data, the frequency of the filter and the time of the collected data can be changed according to the actual situation.

[0067] Since the body impedance of each target object and the wearing manner of the sensor are different, the same neural network model has a low accuracy of the calculation result for different target objects. In order to further ensure that the target posture data has high accuracy, according to a specific embodiment of the present application, the device further comprises a second acquisition unit, a third acquisition unit and a second processing unit. The second acquisition unit is used to acquire a plurality of different predetermined actions of the target object to obtain first predetermined sensor group data, second predetermined sensor group data and third predetermined sensor group data before acquiring the first sensor group data, the second sensor group data and the third sensor group data, in the case that the target object wears the first sensor group, the second sensor group and the third sensor group. The third acquisition unit is used to acquire a predetermined neural network model. The second processing unit is used to learn the predetermined neural network model according to at least the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data to obtain the neural network model. In the case that the target object wears the first sensor group, the second sensor group and the third sensor group, a plurality of different predetermined actions of the target object are acquired, that is, different first predetermined sensor group data, second predetermined sensor group data and third predetermined sensor group data corresponding to different predetermined actions are acquired. Then, the predetermined neural network model is acquired, and the predetermined neural network model is learned according to at least the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data. The learned neural network model is more consistent with the characteristics of different target objects, that is, each target object has its own neural network model, which ensures that the neural network model has high accuracy, and further ensures that the target posture data of the target object determined according to the neural network model has high accuracy.

[0068] In a specific embodiment, the neural network model first outputs N-dimensional angle data, and is calibrated with actual information collected by multiple depth cameras at the same time and optimized by a back propagation algorithm. Through 100 learning processes, learning rates, optimization algorithms and more than 50,000 frames of data, the reconstruction of the posture of a specific part is realized.

[0069] Specifically, the transfer learning of the algorithm is realized in a data-driven manner. We collect tens of thousands of frames of data, classify them in an unsupervised manner, and reduce them to 10 dimensions. We extract the 10-dimensional data as the 10 gestures that are statistically most different. Before use, the system will require each user to wear a sensor and collect data on the 10 gestures for about half a minute and label them. Then, we send these data to our deep learning neural network for further retraining. After reducing the number of training and the learning rate, we limit the optimization space of the model within a certain range to avoid overfitting. That is, the model can better optimize the new target object wearing the multi-modal sensor through the half-minute newly collected learning data, and achieve higher accuracy of specific part gesture recognition. Of course, it is not fixed 10 gestures, but other number of gestures of specific parts, which are mainly determined by actual conditions, and the data collection time can also be extended or shortened according to actual needs.

[0070] According to another specific embodiment of the present application, the first neural network model, the second neural network model and the third neural network model each include M layers of convolutional layers, and the fourth neural network model includes N layers of the convolutional layers, where 0

[0071] In a specific embodiment, M is equal to 5 and N is equal to 18. Of course, M and N can be other numbers of layers, as long as 0

[0072] Specifically, the determination process of the human body posture can be applied to the fields of robot control, prosthetic control, rehabilitation training, medical detection, gesture control in VR (Virtual Reality), etc.

[0073] In addition, in the actual application process, other types of sensors can be added to supplement and further improve the process.

[0074] The human posture determination apparatus includes a processor and a memory, and the first acquisition unit, the first processing unit, and the input unit are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0075] The processor includes a core, and the core calls the corresponding program units in the memory.

[0076] The memory can include a non-permanent memory in a computer readable medium, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

[0077] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the human posture determination method.

[0078] The embodiment of the present application provides a processor, which is used for running a program, and the program is executed to realize the human posture determination method.

[0079] The embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor realizes at least the following steps when executing the program.

[0080] In step S101, first sensor group data, second sensor group data, and third sensor group data are acquired, the first sensor group data is used for representing data of a plurality of sensors corresponding to EMG, the second sensor group data is used for representing data of a plurality of sensors corresponding to EIT, and the third sensor group data is used for representing data of a plurality of sensors corresponding to FMG.

[0081] In step S102, first target sensor group data, second target sensor group data, and third target sensor group data are obtained by respectively performing first predetermined processing on the first sensor group data, the second sensor group data, and the third sensor group data, and the first predetermined processing includes data enhancement processing.

[0082] Step S103, inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of the target object, the neural network model being trained by machine learning using a plurality of sets of predetermined data, each set of the plurality of sets of predetermined data comprising first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data, wherein the first historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EMG, the second historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EIT, and the third historical sensor group data is used to represent historical data of a plurality of sensors corresponding to FMG.

[0083] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0084] The present application also provides a computer program product adapted to execute a program comprising at least the following method steps when executed on a data processing device:

[0085] Step S101, obtaining first sensor group data, second sensor group data and third sensor group data, the first sensor group data being used to represent data of a plurality of sensors corresponding to EMG, the second sensor group data being used to represent data of a plurality of sensors corresponding to EIT, and the third sensor group data being used to represent data of a plurality of sensors corresponding to FMG;

[0086] Step S102, performing first predetermined processing on the first sensor group data, the second sensor group data and the third sensor group data respectively to obtain first target sensor group data, second target sensor group data and third target sensor group data, wherein the first predetermined processing comprises data enhancement processing;

[0087] Step S103, inputting the first target sensor group data, the second target sensor group data and the third target sensor group data into a neural network model to obtain target posture data of the target object, the neural network model being trained by machine learning using a plurality of sets of predetermined data, each set of the plurality of sets of predetermined data comprising first historical sensor group data, second historical sensor group data, third historical sensor group data and historical actual posture data, wherein the first historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EMG, the second historical sensor group data is used to represent historical data of a plurality of sensors corresponding to EIT, and the third historical sensor group data is used to represent historical data of a plurality of sensors corresponding to FMG.

[0088] According to another typical embodiment of the present application, a human posture determination system is also provided, which comprises a controller, a first sensor group, a second sensor group and a third sensor group, wherein the controller is configured to execute any of the above methods; the first sensor group is configured to provide first sensor group data, which is used to represent data of a plurality of sensors corresponding to EMG; the second sensor group is configured to provide second sensor group data, which is used to represent data of a plurality of sensors corresponding to EIT; and the third sensor group is configured to provide third sensor group data, which is used to represent data of a plurality of sensors corresponding to FMG.

[0089] The human posture determination system comprises a controller, a first sensor group, a second sensor group and a third sensor group, wherein the controller is configured to execute any of the above methods; the first sensor group is configured to provide first sensor group data, which is used to represent data of a plurality of sensors corresponding to EMG; the second sensor group is configured to provide second sensor group data, which is used to represent data of a plurality of sensors corresponding to EIT; and the third sensor group is configured to provide third sensor group data, which is used to represent data of a plurality of sensors corresponding to FMG. Compared with the prior art, the human posture determination system of the present application can obtain the first sensor group data, the second sensor group data and the third sensor group data, wherein the first sensor group data is used to represent data of a plurality of sensors corresponding to EMG, the second sensor group data is used to represent data of a plurality of sensors corresponding to EIT, and the third sensor group data is used to represent data of a plurality of sensors corresponding to FMG, i.e. by obtaining sensor data of three modes of electromyography, myodynamic diagram and electrical impedance tomography, and by performing the above enhanced data processing on the first sensor group data, the second sensor group data and the third sensor group data, the first target sensor group data, the second target sensor group data and the third target sensor group data obtained by processing are more consistent with the actual posture of the target object, and then the first target sensor group data, the second target sensor group data and the third target sensor group data are input into the neural network model, so that the target posture data of the target object obtained is obtained by the neural network model and the data of a plurality of sensors corresponding to EMG, EIT and FMG, thereby avoiding the problem of high cost and low accuracy in determining the human posture in the prior art, and ensuring high accuracy of the target posture data.

[0090] In the above-mentioned embodiments of the present application, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0091] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented by other manners. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the above-mentioned units can be a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0092] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0093] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0094] The above-mentioned integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The above-mentioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0095] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

[0096] 1) In the above-mentioned human posture determination method of the present application, first, the first sensor group data, the second sensor group data, and the third sensor group data are obtained, the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG; then, the first sensor group data, the second sensor group data, and the third sensor group data are subjected to data enhancement processing respectively to obtain the first target sensor group data, the second target sensor group data, and the third target sensor group data; finally, the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into a neural network model to obtain target posture data of a target object, the neural network model is trained by machine learning using multiple sets of predetermined data, each set of data in the multiple sets of predetermined data includes first historical sensor group data, second historical sensor group data, third historical sensor group data, and historical actual posture data, wherein the first historical sensor group data is used to represent historical data of multiple sensors corresponding to EMG, the second historical sensor group data is used to represent historical data of multiple sensors corresponding to EIT, and the third historical sensor group data is used to represent historical data of multiple sensors corresponding to FMG. Compared with the problem of high cost and low accuracy in determining human posture in the prior art, the above-mentioned human posture determination method of the present application, by obtaining the first sensor group data, the second sensor group data, and the third sensor group data, wherein the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG, that is, by obtaining sensor data in three ways of electromyography, muscle strength diagram, and electrical impedance tomography, and then by performing the above-mentioned enhancement data processing on the first sensor group data, the second sensor group data, and the third sensor group data, the first target sensor group data, the second target sensor group data, and the third target sensor group data obtained by processing are more consistent with the actual posture of the target object, and then the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into the neural network model, so that the target posture data of the target object obtained is obtained by the neural network model and the data of multiple sensors corresponding to EMG, EIT, and FMG, avoiding the problem of high cost and low accuracy in determining human posture in the prior art, and ensuring high accuracy of the target posture data.

[0097] 2) In the human posture determination device of the present application, the first sensor group data, the second sensor group data, and the third sensor group data are obtained by the first acquisition unit, the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG; the first predetermined processing is performed on the first sensor group data, the second sensor group data, and the third sensor group data by the first processing unit to obtain the first target sensor group data, the second target sensor group data, and the third target sensor group data, wherein the first predetermined processing includes data enhancement processing; the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into the neural network model by the input unit to obtain the target posture data of the target object, and the neural network model is trained by machine learning using multiple sets of predetermined data, each set of data in the multiple sets of predetermined data includes first historical sensor group data, second historical sensor group data, third historical sensor group data, and historical actual posture data, wherein the first historical sensor group data is used to represent the historical data of multiple sensors corresponding to EMG, the second historical sensor group data is used to represent the historical data of multiple sensors corresponding to EIT, and the third historical sensor group data is used to represent the historical data of multiple sensors corresponding to FMG. Compared with the high cost and low accuracy of determining human posture in the prior art, the human posture determination device of the present application obtains the first sensor group data, the second sensor group data, and the third sensor group data, wherein the first sensor group data is used to represent the data of multiple sensors corresponding to EMG, the second sensor group data is used to represent the data of multiple sensors corresponding to EIT, and the third sensor group data is used to represent the data of multiple sensors corresponding to FMG, that is, the sensor data of three modes of electromyography, myodynamogram, and electrical impedance tomography is obtained, and the first sensor group data, the second sensor group data, and the third sensor group data are subjected to the enhancement data processing, so that the first target sensor group data, the second target sensor group data, and the third target sensor group data obtained by processing are more consistent with the actual posture of the target object, and the first target sensor group data, the second target sensor group data, and the third target sensor group data are input into the neural network model, so that the target posture data of the target object obtained by the neural network model is obtained by the data of multiple sensors corresponding to EMG, EIT, and FMG, avoiding the high cost and low accuracy of determining human posture in the prior art, and ensuring the high accuracy of the target posture data.

[0098] 3) The human posture determination system of this application includes a controller, a first sensor group, a second sensor group, and a third sensor group. The controller is used to execute any of the methods described above. The first sensor group provides first sensor group data, which characterizes data from multiple sensors corresponding to EMG. The second sensor group provides second sensor group data, which characterizes data from multiple sensors corresponding to EIT. The third sensor group provides third sensor group data, which characterizes data from multiple sensors corresponding to FMG. Compared to the high cost and low accuracy of human posture determination in the prior art, the human posture determination system of this application acquires the first sensor group data, the second sensor group data, and the third sensor group data. Specifically, the first sensor group data characterizes data from multiple sensors corresponding to EMG, the second sensor group data characterizes data from multiple sensors corresponding to EIT, and the third sensor group data characterizes data from multiple sensors corresponding to FMG. This is achieved by acquiring sensor data from three methods: electromyography (EMG), force mapping, and electrical impedance tomography (EIT). The system then analyzes the data from the first sensor group data, the second sensor group data, and the third sensor group data. The data from the sensor groups undergoes the aforementioned enhanced data processing, making the processed data from the first, second, and third target sensor groups more consistent with the actual posture of the target object. These data are then input into the aforementioned neural network model, resulting in target posture data for the target object obtained through the neural network model and data from multiple sensors corresponding to EMG, EIT, and FMG. This avoids the high cost and low accuracy issues in determining human posture in existing technologies, ensuring high accuracy of the target posture data.

[0099] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining human posture, characterized in that, The method includes: Acquire data from a first sensor group, a second sensor group, and a third sensor group. The first sensor group data is used to characterize the data from multiple sensors corresponding to EMG electromyography, the second sensor group data is used to characterize the data from multiple sensors corresponding to EIT electrical impedance imaging, and the third sensor group data is used to characterize the data from multiple sensors corresponding to FMG myometry. The data from the first sensor group, the second sensor group, and the third sensor group are subjected to a first predetermined processing to obtain the data from the first target sensor group, the second target sensor group, and the third target sensor group, respectively. The first predetermined processing includes data augmentation processing. The data from the first target sensor group, the second target sensor group, and the third target sensor group are input into a neural network model to obtain the target pose data of the target object. The neural network model is trained using multiple sets of predetermined data through machine learning. Each set of predetermined data includes first historical sensor group data, second historical sensor group data, third historical sensor group data, and historical actual pose data. Specifically, the first historical sensor group data represents the historical data from multiple sensors corresponding to EMG electromyography, the second historical sensor group data represents the historical data from multiple sensors corresponding to EIT electrical impedance tomography, and the third historical sensor group data represents the historical data from multiple sensors corresponding to FMG force mapping. The process of inputting the data from the first target sensor group, the second target sensor group, and the third target sensor group into a neural network model to obtain target pose data of the target object includes: acquiring the neural network model, which includes a first neural network model, a second neural network model, a third neural network model, and a fourth neural network model, wherein the first neural network model is trained by machine learning based on at least multiple sets of first historical sensor group data, the second neural network model is trained by machine learning based on at least multiple sets of second historical sensor group data, the third neural network model is trained by machine learning based on at least multiple sets of third historical sensor group data, and the fourth neural network model is trained by machine learning based on at least the historical actual pose data; inputting the data from the first target sensor group, the second target sensor group, and the third target sensor group into the first neural network model, the second neural network model, and the third neural network model one-to-one to obtain corresponding first pose data, second pose data, and third pose data; performing a second predetermined processing on the first pose data, the second pose data, and the third pose data to obtain corresponding first images, second images, and third images, wherein the second predetermined processing includes pooling processing; and determining the target pose data based at least on the first image, the second image, and the third image. Determining the target pose data based at least on the first image, the second image, and the third image includes: performing a third predetermined processing on the first image, the second image, and the third image to obtain a fourth image, wherein the third predetermined processing includes a concat processing; inputting the fourth image into the fourth neural network model to obtain the target pose data, wherein the first neural network model, the second neural network model, and the third neural network model each include M layers of convolutional layers, and the fourth neural network model includes N layers of the convolutional layers, wherein 0 < M < N, and M and N are integers.

2. The method according to claim 1, characterized in that, Acquire data from the first sensor group, the second sensor group, and the third sensor group, including: Acquire data from the first sensor group, the second sensor group, and the third sensor group within a predetermined time period. The data from the first sensor group, the second sensor group, and the third sensor group are obtained after undergoing a fourth predetermined processing, which includes high-pass filtering and / or low-pass filtering.

3. The method according to claim 1, characterized in that, Before acquiring data from the first sensor group, the second sensor group, and the third sensor group, the method further includes: When the target object is wearing the first sensor group, the second sensor group and the third sensor group, multiple different predetermined actions of the target object are acquired to obtain the first predetermined sensor group data, the second predetermined sensor group data and the third predetermined sensor group data. Obtain the pre-defined neural network model; The predetermined neural network model is obtained by learning from at least the data of the first predetermined sensor group, the data of the second predetermined sensor group, and the data of the third predetermined sensor group.

4. A device for determining human posture, characterized in that, The apparatus used in the method for determining human posture according to any one of claims 1 to 3, the apparatus comprising: The first acquisition unit is used to acquire data from a first sensor group, a second sensor group, and a third sensor group. The first sensor group data is used to characterize the data from multiple sensors corresponding to EMG electromyography, the second sensor group data is used to characterize the data from multiple sensors corresponding to EIT electrical impedance imaging, and the third sensor group data is used to characterize the data from multiple sensors corresponding to FMG muscle imaging. A first processing unit is configured to perform a first predetermined processing on the data from the first sensor group, the data from the second sensor group, and the data from the third sensor group, respectively, to obtain data from the first target sensor group, the data from the second target sensor group, and the data from the third target sensor group, wherein the first predetermined processing includes data augmentation processing. The input unit is used to input the data from the first target sensor group, the second target sensor group, and the third target sensor group into a neural network model to obtain the target pose data of the target object. The neural network model is trained by machine learning using multiple sets of predetermined data. Each set of predetermined data includes a first historical sensor group, a second historical sensor group, a third historical sensor group, and historical actual pose data. The first historical sensor group is used to represent the historical data of multiple sensors corresponding to EMG electromyography, the second historical sensor group is used to represent the historical data of multiple sensors corresponding to EIT electrical impedance imaging, and the third historical sensor group is used to represent the historical data of multiple sensors corresponding to FMG force mapping.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program performs the method according to any one of claims 1 to 3.

6. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 3 when it runs.

7. A system for determining human posture, characterized in that, include: A controller, the controller being configured to perform the method according to any one of claims 1 to 3; A first sensor group is used to provide first sensor group data, which is used to characterize the data of multiple sensors corresponding to EMG electromyography. The second sensor group is used to provide second sensor group data, which is used to characterize the data of multiple sensors corresponding to EIT electrical impedance imaging. The third sensor group is used to provide third sensor group data, which is used to characterize the data of multiple sensors corresponding to the FMG myograph.

Images