Action semantic information recognition method based on data enhancement and intelligent motion segmentation

Abstract

The present invention relates to a method and system for identifying action semantic information based on data enhancement and intelligent motion segmentation. The method comprises: using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to segment the motion data of a six-axis inertial sensor to obtain a plurality of six-axis inertial sensor data segments; performing data enhancement processing on the six-axis inertial sensor data segments based on a signal reconstruction algorithm to generate six-axis inertial sensor waveform data segments after data enhancement, performing trajectory resolution based on the data enhancement to generate a three-dimensional motion trajectory corresponding to the six-axis inertial sensor waveform data segments after data enhancement; performing upsampling processing on the six-axis inertial sensor waveform data segments after data enhancement and their three-dimensional motion trajectories, inputting the upsampled waveform data segments and their three-dimensional motion trajectories into a spatial graph neural network for semantic information recognition, obtaining recognition results of action semantic information, and realizing high-precision action semantic information recognition based on inertial sensors.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction control based on inertial sensors, and in particular to a method and system for identifying action semantic information based on data enhancement and intelligent motion segmentation. Background Art

[0002] MEMS (Micro-Electro-Mechanical System) inertial sensors have the advantages of small size, easy to wear, low power consumption, low cost, and easy mass production. The unit price of common inertial sensors on the market can be as low as 0.2-0.5 yuan. Therefore, human-computer interaction based on inertial sensors is not only more natural (compared with voice, image and other methods, it is not limited by distance and scene), but also has low requirements for hardware configuration, so it has huge market prospects. However, the accuracy of inertial sensors is poor, especially the errors of cheap inertial sensors are difficult to support high-precision posture solution, trajectory restoration, and motion segmentation, and thus it is difficult to achieve high-precision motion recognition, which is precisely the basis of human-computer interaction control. This contradiction is also the bottleneck of the current development of human-computer interaction based on inertial sensors.

[0003] Another difficulty of this technology is that the human-computer interaction control technology based on inertial sensors relies on the accurate extraction of "short-term motion data", that is, the intelligent and accurate segmentation of the motion process, and then the accurate recognition of the semantic information in the action. However, most of the existing motion segmentation technologies based on inertial sensors rely on the recognition of static states during the motion process, which requires users to make a lot of pauses during the motion process, which obviously limits the freedom of motion and seriously affects the naturalness of human-computer interaction control. In addition, the segmentation point of the motion should be determined by the task context, that is, the motion process should be intelligently segmented according to the user's segmentation intention and data needs, which is also a key technical issue faced by current research.

[0004] In addition, most existing time series analysis techniques rely on the characteristics of the signal in the time domain or frequency domain, which can be attributed to "waveform characteristics". In the task of semantic information recognition of actions, the difference in semantic information of different actions is essentially determined by the trajectory morphology corresponding to different actions, so the morphological information of the trajectory is extremely important. High-precision restoration of inertial sensor trajectories and integration of trajectory morphological information into downstream tasks based on inertial sensors, especially semantic information recognition tasks of gesture actions, will be a very valuable solution in the field of inertial sensors.

[0005] Most existing deep learning models can only receive structured data, while unstructured geometric morphological data is difficult to directly input into the neural network. Taking the trajectory data of the present invention as an example, if it is input into the model in the form of XYZ coordinates, this data structure loses the morphological characteristics of the trajectory and is difficult to fuse with the original 6-axis signal waveform characteristics of the inertial sensor. Therefore, information fusion of multi-source heterogeneous data based on machine learning models is an important technical issue.

[0006] There are some mature human activity recognition (HAR) solutions based on inertial sensors, such as walking, running, jumping, going up and down stairs, etc. However, these solutions are difficult to apply to the semantic information detection of short-term motion, and therefore cannot achieve human-computer interactive control. For the task of short-term action semantic information recognition, the acquisition of short-term action corresponding data depends on accurate motion segmentation technology, which is not yet mature. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for motion semantic information recognition based on data enhancement and intelligent motion segmentation, so as to realize high-precision motion semantic information recognition based on inertial sensors.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for identifying action semantic information based on data enhancement and intelligent motion segmentation, comprising:

[0010] Acquire six-axis inertial sensor motion data; the six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor;

[0011] The six-axis inertial sensor motion data is segmented by using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information;

[0012] Performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a data-enhanced six-axis inertial sensor waveform data segment;

[0013] Performing trajectory calculation based on the data-enhanced six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the data-enhanced six-axis inertial sensor waveform data segment;

[0014] Performing upsampling processing on the data-enhanced six-axis inertial sensor waveform data segment and its corresponding three-dimensional motion trajectory to obtain an upsampled waveform data segment and its corresponding three-dimensional motion trajectory;

[0015] The upsampled waveform data segment and its corresponding three-dimensional motion trajectory are input into a spatial graph neural network for semantic information recognition to obtain a recognition result of the action semantic information.

[0016] Optionally, the intelligent motion segmentation algorithm based on waveform-trajectory information fusion is used to segment the six-axis inertial sensor motion data to obtain a plurality of six-axis inertial sensor data segments containing complete semantic information, specifically including:

[0017] Performing segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, trajectory morphology information, and acceleration waveform information in a world coordinate system to obtain segmentation points detected based on trajectory morphology features;

[0018] Based on deep learning and sensor original signal waveform information, segmentation point detection is performed on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features;

[0019] Determining a final segmentation point based on AND gate logic according to the segmentation point detected based on the trajectory morphology feature and the segmentation point detected based on the waveform feature;

[0020] The six-axis inertial sensor motion data is segmented according to the final segmentation point to obtain a plurality of six-axis inertial sensor data segments containing complete semantic information.

[0021] Optionally, the performing segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, the trajectory morphology information and the acceleration waveform information in the world coordinate system to obtain the segmentation points detected based on the trajectory morphology features specifically includes:

[0022] De-averaging the six-axis inertial sensor motion data to obtain de-averaged six-axis motion data;

[0023] Performing variance division processing on the three-axis acceleration data in the de-averaged six-axis motion data to obtain six-axis motion data after variance division;

[0024] Determine a preliminary turning point position according to the three-axis acceleration data in the six-axis motion data after the variance is divided;

[0025] Multiplying the six-axis motion data after the preliminary inflection point position by an attenuated gain function to obtain the six-axis motion data after gain addition;

[0026] Performing trajectory calculation on the six-axis motion data after adding the gain, to obtain a distorted motion trajectory with obvious inflection points;

[0027] The morphological features of the trajectory are used to set a multi-scale spline function, and segmentation points are detected on the distorted motion trajectory with obvious inflection points based on the Fréchet distance to obtain segmentation points detected based on the trajectory morphological features.

[0028] Optionally, the performing segmentation point detection on the six-axis inertial sensor motion data based on deep learning and sensor original signal waveform information to obtain segmentation points detected based on waveform features specifically includes:

[0029] Using information captured by the optical sensor, segmentation points in the six-axis inertial sensor motion data are marked to obtain segmentation labels in the inertial sensor data;

[0030] Training a 1D-CNN model based on the six-axis inertial sensor motion data and the segmentation labels to obtain a trained segmentation point detection model based on waveform features;

[0031] The segmentation point detection model based on waveform features is used to perform segmentation point detection on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features.

[0032] Optionally, before performing data enhancement processing on the six-axis inertial sensor data segment based on the signal reconstruction algorithm to generate the data enhanced six-axis inertial sensor waveform data segment, the method further includes:

[0033] Acquire a plurality of six-axis raw signal samples X collected by the inertial sensor;

[0034] Perform model-driven signal reconstruction on multiple six-axis original signal samples to generate multiple sets of data-wavelet pattern matching relationships {[x i ,y j ],β k}; where x i is the i-th axis signal of the six-axis original signal sample X; j is the jth IMF component of the i-th axis signal; β k for y j The corresponding optimal wavelet transform mode;

[0035] According to the data-wavelet pattern matching relationship {[x i ,y j ],β k Generate a training sample for the 1DCNN model; the training sample includes the input data [X, y j ] and the ideal output data β of the 1DCNN model k ;

[0036] The 1DCNN model is trained using the training samples to generate a trained data enhancement model.

[0037] Optionally, performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement specifically includes:

[0038] For each of the six-axis raw signal samples X in the six-axis inertial sensor data segment, the i-th axis signal x i Perform empirical mode decomposition to obtain the x i The jth IMF component y j ;

[0039] The IMF component y j Combined with the six-axis original signal sample X, the input data [X, y j ];

[0040] The input data [X, y j ] Input the trained data enhancement model and output the IMF component y j The corresponding optimal wavelet transform mode;

[0041] The IMF component y is transformed by using the optimal wavelet transform mode. j Perform wavelet transform to generate new IMF components;

[0042] The six-axis original signal sample X is empirically reconstructed according to the new IMF component to generate data-enhanced six-axis inertial sensor data; a plurality of the data-enhanced six-axis inertial sensor data constitute the data-enhanced six-axis inertial sensor waveform data segment.

[0043] Optionally, the step of inputting the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into a spatial graph neural network for semantic information recognition to obtain a recognition result of the action semantic information specifically includes:

[0044] The upsampled waveform data segment and its corresponding three-dimensional motion trajectory are input into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information; the mathematical model of the spatial graph neural network is Where A is the adjacency matrix represented by the upsampled three-dimensional motion trajectory, A ij is the element in the i-th row and j-th column of the adjacency matrix A; S i ,S j Respectively represent the position vectors of the i-th and j-th positions in the upsampled three-dimensional motion trajectory; Dist(S i ,S j) represents two position vectors S i ,S j The Euclidean distance between them; DIM is the position vector S i ,S j Dimensions; Represent the position vector S i , S j The dim-th dimension of the waveform data segment after upsampling; X is the waveform data in the waveform data segment after upsampling; W (n-1) is the n-1th fully connected matrix; H (n) is the nth feature layer in the spatial graph neural network; sigmoid() represents the sigmoid function; softmax() represents the softmax function; Y is the recognition result of the action semantic information.

[0045] An action semantic information recognition system based on data enhancement and intelligent motion segmentation, comprising:

[0046] A motion data acquisition module, used to acquire motion data of a six-axis inertial sensor; the six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor;

[0047] An intelligent motion segmentation module, used to segment the six-axis inertial sensor motion data using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information;

[0048] A data enhancement module, used for performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement;

[0049] A trajectory calculation module, used for performing trajectory calculation based on the six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the six-axis inertial sensor waveform data segment;

[0050] An upsampling module, used for upsampling the six-axis inertial sensor waveform data segment and its corresponding three-dimensional motion trajectory to obtain the upsampled waveform data segment and its corresponding three-dimensional motion trajectory;

[0051] The semantic information recognition module is used to input the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information.

[0052] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0053] The present invention provides a method and system for identifying action semantic information based on data enhancement and intelligent motion segmentation. The method comprises: acquiring six-axis inertial sensor motion data; the six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor; segmenting the six-axis inertial sensor motion data using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information; performing data enhancement processing on the six-axis inertial sensor data segments based on a signal reconstruction algorithm to generate six-axis inertial sensor waveform data segments after data enhancement; performing trajectory calculation based on the six-axis inertial sensor waveform data segments to generate a three-dimensional motion trajectory corresponding to the six-axis inertial sensor waveform data segments; performing upsampling processing on the six-axis inertial sensor waveform data segments and their corresponding three-dimensional motion trajectories to obtain upsampled waveform data segments and their corresponding three-dimensional motion trajectories; inputting the upsampled waveform data segments and their corresponding three-dimensional motion trajectories into a spatial graph neural network for semantic information recognition to obtain recognition results of action semantic information. The method of the present invention proposes an inertial sensor data enhancement method based on deep learning and signal reconstruction, which can significantly improve the accuracy of posture solution and trajectory restoration based on inertial sensors; the proposed intelligent motion segmentation algorithm based on inertial sensors does not rely on the static state during the motion process, so it can achieve high-precision segmentation and data extraction for any motion, and the algorithm can intelligently determine the segmentation point according to the different problems and the needs of the task; in order to be able to fuse signal waveform information and trajectory morphology information, the method of the present invention also proposes a spatial graph neural network (Spatial-GNN), which realizes high-precision recognition of action semantic information based on inertial sensors, and is currently the most accurate action recognition model based on MEMS inertial sensors, so it can support the realization of high-precision human-computer interaction control. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 A flowchart of a method for identifying action semantic information based on data enhancement and intelligent motion segmentation according to the present invention;

[0056] Figure 2 A schematic diagram of optical-inertial trajectory and raw data provided by an embodiment of the present invention;

[0057] Figure 3A schematic diagram of the actual trajectory of an object's motion captured by an optical sensor provided in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the unprocessed world coordinate system acceleration (Z axis) and the corresponding trajectory provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the de-averaged world coordinate system acceleration and the corresponding trajectory provided by an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of the acceleration of the world coordinate system and the corresponding trajectory after the variance is divided provided by an embodiment of the present invention;

[0061] Figure 7 A schematic diagram of the world coordinate system acceleration and corresponding trajectory after adding gain provided by an embodiment of the present invention;

[0062] Figure 8 A schematic diagram of the sliding process of a multi-scale spline function on a distorted trajectory provided by an embodiment of the present invention;

[0063] Fig. 9 A schematic diagram of a process of labeling an inertial sensor using an optical sensor provided in an embodiment of the present invention;

[0064] Fig.10 A schematic diagram of the segmentation results of inertial motion data including 62 gesture actions provided by an embodiment of the present invention;

[0065] Fig.11 A schematic diagram of a model-driven signal reconstruction method provided by an embodiment of the present invention;

[0066] Fig.12 A schematic diagram of a sensor posture change measurement platform provided in an embodiment of the present invention;

[0067] Fig.13 A schematic diagram of the training process of a wavelet pattern matcher provided in an embodiment of the present invention;

[0068] Fig.14 A schematic diagram of a model-driven signal reconstruction process provided by an embodiment of the present invention;

[0069] Fig.15 A schematic diagram of the structure of a graph neural network (Spatial-GNN) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The purpose of the present invention is to provide a method and system for motion semantic information recognition based on data enhancement and intelligent motion segmentation, so as to realize high-precision motion semantic information recognition based on inertial sensors.

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The essence of achieving natural and accurate human-computer interaction control tasks based on inertial sensors is to extract and identify the data containing "specific semantic information" in the inertial sensors, and accurate extraction is a prerequisite for accurate identification.

[0074] Among them, the data extraction task, i.e. extracting inertial sensor data, is to accurately segment the inertial sensor data according to the semantic information contained in the user's actions.

[0075] The core of the semantic information recognition task is to accurately identify the specific trajectory of the user's movement in the air based on the inertial sensor (such as numbers, letters or other special symbols), so the trajectory information in the action is very critical. In order to accurately restore the motion trajectory in the action, the present invention designs a data enhancement scheme based on the "signal reconstruction algorithm". In order to utilize the trajectory information and the original 6-axis (three-axis acceleration, three-axis angular velocity) waveform information collected by the inertial sensor, the present invention designs a spatial graph neural network (Spatial-GNN) to achieve more accurate semantic information recognition and human-computer interaction control.

[0076] In summary, the implementation process of the method of the present invention can be divided into three parts: intelligent motion segmentation based on waveform-trajectory information fusion; data enhancement based on "signal reconstruction algorithm"; and semantic information recognition based on spatial graph neural network (Spatial-GNN).

[0077] Figure 1 This is a flow chart of a method for identifying action semantic information based on data enhancement and intelligent motion segmentation. Figure 1 The present invention provides a method for identifying action semantic information based on data enhancement and intelligent motion segmentation, which specifically includes:

[0078] Step 101: Acquire six-axis inertial sensor motion data.

[0079] First, the six-axis inertial sensor motion data is obtained, which is the waveform data of the three-axis acceleration and three-axis angular velocity collected by the inertial sensor. Then, the intelligent motion segmentation algorithm is used to segment the input six-axis inertial sensor motion data to obtain several six-axis inertial sensor data segments containing complete semantic information.

[0080] Step 102: Segment the six-axis inertial sensor motion data using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information.

[0081] Most of the existing inertial sensor motion segmentation schemes rely on the static state or some other special actions during the motion process. However, this reconstruction scheme destroys the continuity during the motion process, which makes the human-computer interaction process very unnatural. Therefore, the intelligent segmentation technology of motion becomes particularly critical. Based on the morphological characteristics of the distorted trajectory calculated by the inertial sensor, the present invention proposes an intelligent motion segmentation algorithm with the help of multi-resolution Fréchet distance and deep learning model. The segmentation algorithm does not rely on the static state during the motion process, but detects the segmentation point by relying on the drastic change of direction during the motion process, so there is no need to restrict the motion. In addition, the deep learning model can learn the segmentation intention of the annotator through the pre-annotated segmentation points in the data set, which enables the segmentation algorithm to adapt to different task needs and usage scenarios. The intelligent motion segmentation algorithm consists of two parallel parts: segmentation point detection based on Fréchet distance, trajectory morphological information and acceleration waveform information in the world coordinate system, and segmentation point detection based on deep learning and sensor original signal waveform information.

[0082] Therefore, the step 102 uses an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to segment the six-axis inertial sensor motion data to obtain a number of six-axis inertial sensor data segments containing complete semantic information, specifically including:

[0083] Step 2.1: Perform segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, trajectory morphology information, and acceleration waveform information in the world coordinate system to obtain segmentation points detected based on trajectory morphology features.

[0084] In general time series research, it is a common practice to extract the waveform characteristics of the signal. However, due to the serious noise interference in the inertial sensor signal, the hysteresis effect in the accelerometer signal, and the time drift in the gyroscope signal, the position where the direction changes dramatically during the actual movement is often not the extreme point of the original signal, such as Figure 2As shown in the figure, the dots represent the motion segmentation points (direction sudden change points) determined by means of optical sensors, and the curves between the dots are the data and trajectory segments obtained therefrom.

[0085] Depend on Figure 2 It can be found that whether it is the original 6-axis sensor data or the vector sum data of acceleration and angular velocity, the extreme points of the waveform and the actual motion segmentation points ( Figure 2 The dots in the middle will not completely overlap, and may even be far apart. Therefore, it is difficult to achieve accurate frame-by-frame motion segmentation based only on the waveform features of the original signal. Figure 2 It can be found that in the distorted trajectory calculated from the inertial sensor data, the motion segmentation point just corresponds to the inflection point of the trajectory. The reason is that the quadratic integration operation when calculating the trajectory amplifies the inertial sensor error, but also amplifies the fluctuation caused by the drastic change in the motion direction, and this fluctuation often causes obvious inflection points in the calculated distorted trajectory. This phenomenon provides a good feature for the motion segmentation task.

[0086] However, due to the influence of the accumulated error and time drift of the inertial sensor, the inflection point caused by the sudden change of the motion direction is not always present in the trajectory solution. One reason is that the three-axis acceleration in the world coordinate system calculated from the original signal may deviate from the coordinate axis. For example, using an optical sensor to collect the real trajectory of an object, such as Figure 3 As shown in the figure, it can be seen that the direction of the trajectory changes dramatically when it moves along the Z axis. In the original Z axis acceleration signal, its value is generally below the 0 coordinate axis, as shown in Figure 4 As shown in the left figure, this means that the object always moves along the negative direction of the Z axis. Even if a part of it is above the coordinate axis, the role of this part of the data is only to slow down the object appropriately in the negative direction, but not enough to cause the object to move in the positive direction. The direction of movement remains unchanged, and there will naturally be no turning point in the trajectory, such as Figure 4 As shown in the right figure. Similarly, the same problem will occur in the acceleration of the XY direction. Therefore, in order to present the inflection point in the distorted trajectory, the original signal needs to be processed. It is worth noting that the purpose of this part of data processing is to highlight the inflection point in the trajectory, so as to achieve frame-accurate motion segmentation, so it does not pay attention to the degree of distortion of the trajectory.

[0087] Therefore, the step 2.1 performs segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, trajectory morphology information, and acceleration waveform information in the world coordinate system to obtain segmentation points detected based on trajectory morphology features, specifically including:

[0088] Step 2.1.1: De-averaging the six-axis inertial sensor motion data to obtain de-averaged six-axis motion data;

[0089] Specifically, the mean value of each axis signal of the six-axis inertial sensor motion data is subtracted, that is, it is translated to the vicinity of the coordinate axis, such as Figure 5 As shown in the left figure. This de-averaging operation changes the distortion trajectory from a straight line with an almost constant motion direction to an arc with a slowly changing motion direction, as shown in Figure 5 As shown in the right figure. In addition, the de-averaging operation can also make subsequent operations more stable.

[0090] Step 2.1.2: performing variance division processing on the three-axis acceleration data in the de-averaged six-axis motion data to obtain the six-axis motion data after variance division;

[0091] However, the change in the direction of movement in the trajectory is not drastic enough, and the slowly changing trajectory is not sufficient to provide clear inflection point information. This problem manifests itself in the acceleration data as follows: the acceleration value at the inflection point does not have an obvious advantage, so it is impossible to quickly change the previously accumulated movement speed, that is, to reverse the direction of movement in a short period of time. Therefore, it is necessary to enhance the signal strength near the split point and appropriately weaken the non-inflection point signal. It can be found that the frequency of the data waveform at the inflection point is lower, so the variance is smaller. Utilizing this feature, the present invention sets a sliding window and divides the acceleration data in the window by the corresponding variance to achieve the amplification of the data at the inflection point and the suppression of the rest of the data, such as Figure 6 shown.

[0092] from Figure 6 It can be seen from the acceleration of the world coordinate system and the corresponding trajectory that since the data at the inflection point has an obvious numerical advantage, the change of the trajectory is almost completely determined by the data at the inflection point. This feature ensures that the operation of the data at the inflection point can maximize the impact on the trajectory.

[0093] Step 2.1.3: Determine the initial turning point position according to the three-axis acceleration data in the six-axis motion data after the variance is divided;

[0094] Since the change in the direction of motion at the inflection point needs to be more drastic, the data in the second half of the inflection point needs to be appropriately amplified. First, it is necessary to detect the approximate position of the inflection point, that is, to determine the preliminary inflection point position. After the variance division process in the previous step, the values ​​of the three-axis acceleration at the inflection point are amplified, and the values ​​at the non-inflection point are reduced. Therefore, the approximate interval of the inflection point can be determined by setting a suitable threshold. Since the signal has been "de-averaged" before, there will be a zero-crossing point in this interval for each of the three-axis accelerations, and the mean Q of the three zero-crossing points is determined as the approximate position of the inflection point, that is, the preliminary inflection point position.

[0095] Step 2.1.4: multiplying the six-axis motion data after the initial inflection point position by an attenuated gain function to obtain the six-axis motion data after gain addition;

[0096] The data before the Q point is not processed, and the data after it is multiplied by an attenuated gain function f(t):

[0097]

[0098] Where t is the corresponding frame of the time axis, σ is an artificially set hyperparameter greater than zero, which can be adjusted as needed. The smaller σ is, the stronger the gain of the signal.

[0099] This gain function can enhance the drastic change of the motion direction and make the turning points in the trajectory more obvious.

[0100] Step 2.1.5: performing trajectory calculation on the six-axis motion data after adding the gain, and obtaining a distorted motion trajectory with obvious inflection points;

[0101] The trajectory of the six-axis motion data after adding the gain is solved, and finally, a motion trajectory with obvious inflection points is obtained, such as Figure 7 shown.

[0102] Step 2.1.6: Using the morphological features of the trajectory, a multi-scale spline function is set, and segmentation points are detected on the distorted motion trajectory with obvious inflection points based on the Fréchet distance to obtain segmentation points detected based on the trajectory morphological features.

[0103] After obtaining the distorted motion trajectory with obvious inflection points, the inflection points can be detected using the morphological features of the trajectory to obtain relatively accurate segmentation points. Figure 8 As shown, the present invention sets a multi-scale spline function ( Figure 8 The thick line shows the trajectory ( Figure 8 Each slide can be regarded as a fitting of the spline function to the trajectory in the window, and the Fréchet distance is used to measure the fitting effect or closeness between the spline function and the trajectory segment in the window. In general, the spline function can fit the corresponding trajectory segment very well, and the Fréchet value is small at this time. However, when it slides to a position where the direction of movement changes drastically, it is difficult for the spline function to fit the distorted trajectory at the inflection point, and the Fréchet value will increase sharply, thereby obtaining the position of the segmentation point. Since spline functions of different scales have different sensitivities to different trajectory shapes, the present invention sets a multi-scale spline function to achieve accurate detection of the segmentation point, and obtains the segmentation point detected based on the trajectory morphological characteristics.

[0104] Step 2.2: Perform segmentation point detection on the six-axis inertial sensor motion data based on deep learning and sensor original signal waveform information to obtain segmentation points detected based on waveform features.

[0105] Although the segmentation point detection model based on the distorted trajectory described in step 2.1 has achieved good segmentation effects, the model is not learnable, so it is difficult to adaptively adjust the severity of the segmentation according to different requirements. For example, in some action trajectories (such as handwritten numbers and letters), even if there is a drastic change in direction, it is not desired that the movement be segmented by the model. Similarly, between two actions that contain complete semantic information, even if the drastic change in the direction of the action is not high, it is still desired that the model can segment them. This requires pre-annotating a large number of segmentation points according to human needs and training them with the help of a deep learning model.

[0106] Therefore, the step 2.2 performs segmentation point detection on the six-axis inertial sensor motion data based on deep learning and the sensor original signal waveform information to obtain segmentation points detected based on waveform features, specifically including:

[0107] Step 2.2.1: Marking the segmentation points in the six-axis inertial sensor motion data with the help of information captured by the optical sensor to obtain segmentation labels in the inertial sensor data;

[0108] The present invention uses the information captured by the optical sensor to mark the segmentation points in the original waveform of the inertial sensor. The experimental process is as follows: Fig. 9 As shown, the segmentation labels in the inertial sensor data are obtained.

[0109] Step 2.2.2: Train the 1D-CNN model based on the six-axis inertial sensor motion data and the segmentation label to obtain a trained segmentation point detection model based on waveform features.

[0110] After obtaining the segmentation labels in the inertial sensor data, the present invention trains the 1D-CNN model to implement the segmentation point detection task based on the 6-axis sensor data waveform. The specific method is: set a window that slides on the original data, and input the 6-axis data in the window into the 1D-CNN model. The model will predict the potential segmentation points in the window. By comparing the prediction results with the pre-labeled segmentation labels, the prediction error can be obtained. As the window slides, a large amount of errors are back-propagated back to the 1D-CNN model to train it. Finally, the fully trained segmentation point detection model has the ability to predict potential segmentation points based on the input waveform data.

[0111] Step 2.2.3: Use the segmentation point detection model based on waveform features to perform segmentation point detection on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features.

[0112] In practical applications, segmentation points detected based on waveform features can be obtained by using the segmentation point detection model based on waveform features to detect segmentation points of the six-axis inertial sensor motion data.

[0113] Step 2.3: Determine the final segmentation point based on AND gate logic according to the segmentation point detected based on the trajectory morphology feature and the segmentation point detected based on the waveform feature.

[0114] In order to combine the two types of potential segmentation points detected based on waveform features and trajectory morphology features, the present invention sets an "AND" gate. When there is a segmentation point detected based on waveform features near the segmentation point detected based on trajectory morphology features, the midpoint of the two is determined as the final segmentation point; when there is no segmentation point detected based on waveform features near the segmentation point detected based on trajectory morphology features, the point is deleted. The "AND" gate logic combines the two features of morphology and waveform, and incorporates the artificial segmentation intention, which ensures that the final segmentation point can adapt to different needs while accurately detecting the motion segmentation point.

[0115] Step 2.4: Segment the six-axis inertial sensor motion data according to the final segmentation point to obtain a plurality of six-axis inertial sensor data segments containing complete semantic information.

[0116] In order to verify the actual effect of the intelligent motion segmentation model proposed in this invention, the user held the inertial sensor and performed continuous hand movements at a speed of 0.5-1.5m / s. During the movement process of about 165 seconds (32887 frames, sampling frequency of 200Hz), a total of 62 gestures containing complete semantic information (26 uppercase and lowercase letters and numbers 0-9) were performed. The intelligent motion segmentation algorithm proposed in this invention detected 124 segmentation points from 32887 frames of data, such as Fig.10 As shown in the figure, the dots are the segmentation points detected by the model, and the curves between the segmentation points are the inertial motion data corresponding to different gesture actions. These inertial motion data constitute a six-axis inertial sensor data segment containing complete semantic information.

[0117] By comparing the data segmentation points marked by the optical sensor, it can be found that the intelligent motion segmentation model of the present invention not only accurately identifies 62 gestures containing complete semantic information, but also can accurately detect the location of each action start and end frame to the single digit. The intelligence of this intelligent motion segmentation algorithm lies in that even if these 62 gestures contain a large number of locations where the direction of motion changes dramatically, the segmentation model will not judge them as segmentation points, thus ensuring the integrity of the semantic information in the motion data and adapting well to human needs. It is worth noting that the segmentation point positions detected by the model are indeed not extreme points in the waveform data, and the model can accurately detect motion segmentation sites from monotonous or flat waveforms without obvious features, which proves that the introduction of trajectory morphological features by the intelligent segmentation model is successful, which is also consistent with previous analysis.

[0118] Step 103: performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement.

[0119] In order to accurately restore the motion trajectory in the action, the present invention designs a data enhancement method based on the "signal reconstruction algorithm". This method uses empirical mode decomposition, wavelet transform and one-dimensional convolutional neural network to achieve inertial sensor signal reconstruction. The reconstruction process can be divided into two steps: model-driven signal reconstruction and data-driven signal reconstruction. The former provides the latter with the data set required for training.

[0120] The model-driven signal reconstruction process is as follows:

[0121] Model-driven signal reconstruction is the process of making data sets and data annotation. The specific methods are as follows Fig.11 As shown in the figure, the inertial sensor is placed on a turntable and rotated multiple times. At the end of each movement, the turntable is used to record the changes in the sensor's posture during the movement, as shown in the figure. Fig.12 Assume that during a certain movement, the 6-axis original signal samples collected by the inertial sensor are X, where the i-th axis signal is represented by x i (x i ∈X,i=1,2,3…6), then the model-driven reconstruction process of signal X is as follows:

[0122] (1) For signal x i (x i ∈X, i=1,2,3…6) to perform empirical mode decomposition and obtain several intrinsic mode function (IMF) components y j , where X is a raw data sample of a 6-axis inertial sensor;

[0123] (2) For each IMF component y j 15 wavelet transform modes are performed respectively (different wavelet bases are regarded as one wavelet transform mode. The present invention sets 15 different wavelet basis functions, which are: 'bior1.1', 'bior1.3', 'bior1.5', 'bior2.2', 'bior2.4', 'bior2.6', 'bior2.8', 'bior3.1', 'bior3.3', 'bior3.5', 'bior3.7', 'bior3.9', 'bior4.4', 'bior5.5', 'bior6.8'). At this time, an IMF component yj (yj∈Y xi ,j=1,2,3…M) of 15 wavelet transform results, among which Y xi For signal x i The set of all decomposed IMF components, M is the number of IMF components in the set;

[0124] (3) These 15 wavelet transform results are used to replace the original IMF components yj, and then the replaced IMF components yj are used to calculate the j The original signal X was reconstructed respectively, and 15 signal reconstruction results of the original signal were obtained;

[0125] (4) Using the 15 original signal reconstruction results, perform attitude calculations respectively (there are mature algorithms for attitude calculations, see related inertial guidance theories), and obtain 15 attitude calculation results. These 15 attitude calculation results are compared with the actual attitude changes recorded during the measurement of the data, and the attitude calculation errors of the 15 signals are obtained.

[0126] (5) The wavelet transform mode β corresponding to the minimum attitude error k (β k ∈{'bior1.1','bior1.3','bior1.5','bior2.2','bior2.4','bior2.6','bior2.8','bior3.1','bior3.3','bior3.5','bior3.7','bior3.9','bior4.4','bior5.5','bior6.8'}, k=1,2,3…15) and replace the original IMF components with the processing results of the IMF components by this wavelet;

[0127] (6) Continue to operate on the next IMF component. Repeat steps (2) to (6) until all 6-axis original signal samples X are processed.

[0128] By adopting model-driven signal reconstruction, different original signals x are obtained.i Different IMF components y j The optimal wavelet transform mode β k (β k ∈{'bior1.1','bior1.3','bior1.5','bior2.2','bior2.4','bior2.6','bior2.8','bior3.1','bior3.3','bior3.5','bior3.7','bior3.9','bior4.4','bior5.5','bior6.8'}, k=1,2,3…15), therefore, for each input signal sample X and its decomposed IMF component yj, a "data-wavelet pattern matching relationship" can be obtained: {[x i ,y j ],β k}, and finally 6841 sets of such “data-wavelet pattern matching relationships” were obtained.

[0129] Among them, the data-driven signal reconstruction process is as follows:

[0130] Through a large number of repeated comparisons and calculations, model-driven data reconstruction finally obtained 6841 sets of "data-wavelet pattern matching relationships", which will be used as training sets to train the one-dimensional convolutional neural network so that it can intelligently select the appropriate wavelet processing mode according to the input signal. This process is data-driven signal reconstruction. First, a wavelet pattern matcher needs to be trained based on the 6841 sets of "data-wavelet pattern matching relationships", that is, Fig.13 The 1DCNN module in the example has the following specific operation procedures:

[0131] (1) Input data of 1DCNN model: [X,y in “data-wavelet pattern matching relationship” j ] is the input of the 1DCNN model, which combines the original signal X with its decomposed IMF component y j Connect in parallel to form a set of 7×N input data (N is the signal length);

[0132] (2) Output data of 1DCNN model: The parallel input data is input into the one-dimensional convolutional neural network (1DCNN), and the output is the IMF component y j Corresponding wavelet transform mode

[0133] (3) Training of 1DCNN model: The wavelet transform mode of the model output The ideal wavelet transform pattern β in the pre-obtained “data-wavelet pattern matching relationship” kCompare, calculate the error, and update the 1DCNN model according to the error (the training process is an existing technology). Repeat steps (1)-(3) until the error is within the threshold range, and obtain a trained data enhancement model.

[0134] Therefore, before performing data enhancement processing on the six-axis inertial sensor data segment based on the signal reconstruction algorithm in step 103 to generate the six-axis inertial sensor waveform data segment after data enhancement, the method further includes:

[0135] Acquire a plurality of six-axis raw signal samples X collected by the inertial sensor;

[0136] Perform model-driven signal reconstruction on multiple six-axis original signal samples to generate multiple sets of data-wavelet pattern matching relationships {[x i ,y j ],β k}; where x i is the i-th axis signal of the six-axis original signal sample X; j is the jth IMF component of the i-th axis signal; β k for y j The corresponding optimal wavelet transform mode;

[0137] According to the data-wavelet pattern matching relationship {[x i ,y j ],β k Generate a training sample for the 1DCNN model; the training sample includes the input data [X, y j ] and the ideal output data β of the 1DCNN model k ;

[0138] The 1DCNN model is trained using the training samples to generate a trained data enhancement model.

[0139] The trained 1DCNN model can be based on the input original signal X and the IMF component y decomposed from it. j Determine the most suitable wavelet transform method for the IMF component, and use the result of the wavelet transform to replace the original IMF component, and finally complete the signal reconstruction. The complete data-driven signal reconstruction process is as follows: Fig.14 The specific operation process is as follows:

[0140] (1) For each axis signal x of the 6-axis inertial sensor raw signal X i (x i ∈X, i=1,2,3…6) perform empirical mode decomposition respectively to obtain several IMF components;

[0141] (2) Each IMF component is combined with the original signal to form 7×N input data;

[0142] (3) Input the data into the trained 1DCNN model to obtain the wavelet transform mode suitable for the IMF;

[0143] (4) After performing corresponding wavelet transform on the IMF component, the original IMF component is replaced;

[0144] (5) After replacing all IMF components, the original signal is reconstructed using empirical mode reconstruction to obtain new 6-axis inertial sensor data, completing data enhancement.

[0145] That is, the step 103 performs data enhancement processing on the six-axis inertial sensor data segment based on the signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement, specifically including:

[0146] For each of the six-axis raw signal samples X in the six-axis inertial sensor data segment, the i-th axis signal x i Perform empirical mode decomposition to obtain the x i The jth IMF component y j ;

[0147] The IMF component y j Combined with the six-axis original signal sample X, the input data [X, y j ];

[0148] The input data [X, y j ] Input the trained data enhancement model and output the IMF component y j The corresponding optimal wavelet transform mode;

[0149] The IMF component y is transformed by using the optimal wavelet transform mode. j Perform wavelet transform to generate new IMF components;

[0150] The six-axis original signal sample X is empirically reconstructed according to the new IMF component to generate data-enhanced six-axis inertial sensor data; a plurality of the data-enhanced six-axis inertial sensor data constitute the data-enhanced six-axis inertial sensor waveform data segment.

[0151] Step 104: performing trajectory calculation based on the data-enhanced six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the data-enhanced six-axis inertial sensor waveform data segment.

[0152] The method described in step 103 is used to enhance the data of each segment of the 6-axis inertial sensor data, and then trajectory calculation is performed separately (existing technology). At this time, for any segment of 6-axis sensor waveform data containing complete semantic information There is a three-dimensional motion trajectory. K is the signal length of the data / track, and the signal length K corresponding to different motion segments is generally different.

[0153] Step 105: performing up-sampling processing on the data-enhanced six-axis inertial sensor waveform data segment and its corresponding three-dimensional motion trajectory to obtain an up-sampled waveform data segment and its corresponding three-dimensional motion trajectory.

[0154] For each segment of 6-axis inertial sensor waveform data and its corresponding three-dimensional trajectory Upsampling is performed (using the cubic spline interpolation method, the prior art) to obtain the upsampled new waveform data X (N × 6) and its corresponding trajectory S (N × 3), where N is the upsampled signal length (N>K). At this time, the inertial sensor waveform data and its trajectory with different semantic information have the same signal length.

[0155] Step 106: Input the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into a spatial graph neural network for semantic information recognition to obtain a recognition result of the action semantic information.

[0156] In the task of semantic information recognition of gestures, the core and intuitive information is the trajectory corresponding to different gestures. Different actions are defined by the geometric shapes of different trajectories, rather than by the original acceleration and angular velocity waveforms measured by inertial sensors. Therefore, in order to achieve high-precision semantic information recognition of actions, it is far from enough to simply input the waveform information of the original signal into the classifier. Combining the trajectory information of the movement with the original waveform information measured by the sensor is the standard paradigm for future semantic recognition of inertial sensor actions and human-computer interaction control.

[0157] However, the motion trajectory is not a standard structured data. If the trajectory is only input into the classifier in the form of X, Y, and Z coordinates, the classifier still does not receive the morphological information of the trajectory. Therefore, it is a key technology to construct a classifier suitable for this task that can simultaneously receive structured and unstructured data. To this end, the present invention designs a spatial graph neural network (Spatial-GNN), whose specific structure is as follows Fig.15As shown in the figure, the core of the structure is to convert the input inertial sensor waveform signal into a spatial trajectory with morphological information, and then use the adjacency matrix to represent the spatial trajectory. The adjacency matrix A that stores the trajectory morphological information will participate in each round of forward propagation calculation until the final classification result is obtained. It is worth noting that the original inertial sensor waveform signal is input into the spatial graph neural network in the first round of calculation. Therefore, the model actually uses both the waveform characteristics of the original signal and the morphological characteristics of the corresponding trajectory, which is consistent with the original design intention of the present invention.

[0158] Each waveform signal obtained in step 105 and its corresponding three-dimensional trajectory are input into the spatial graph neural network structure designed by the present invention to perform semantic information recognition. The specific operation is as follows:

[0159] 1) The three-dimensional space trajectory is represented in the form of an adjacency matrix; the weight of the edge between two nodes is represented as the inverse square of the Euclidean distance Dist+1 between the two points in space:

[0160]

[0161] Among them A ij is the element in the i-th row and j-th column of the adjacency matrix A; S i ,S j Respectively represent the spatial position coordinates of the i-th and j-th points in the trajectory, also known as the position vectors of the i-th and j-th positions in the trajectory sequence; Dist(S i ,S j ) represents two position vectors S i ,S j The Euclidean distance between them is calculated as follows:

[0162]

[0163] Where DIM is the position vector S i , S j Dimension, here DIM=3. Represent the two position vectors S i , S j The dimth dimension of . At this time, the adjacency matrix A(N×N) represented by the trajectory is obtained.

[0164] 2) Multiply the adjacency matrix A (N×N) by the input signal X (N×6) and then by a fully connected matrix W to be trained (0) (6×M), we can get an N×M matrix. After applying the sigmoid function to each element of the matrix, the new N×M matrix is ​​named the feature layer H (1) ;

[0165]

[0166] 3) Multiply the adjacency matrix A(N×N) by the feature layer H obtained in the previous step (1) (N×M) is then multiplied by a fully connected matrix W to be trained (1) (M×M), we can get an N×M matrix. After applying the sigmoid function to each element of the matrix, the new N×M matrix is ​​named the feature layer H (2) ;

[0167] 4) Perform step 3) n-1 times, where n is a pre-set hyperparameter, depending on the problem; at this point, the feature layer H can be obtained (n) (N×M);

[0168] 5) H (n) (N×M) inputs the softmax function to get the output layer Y(1×M);

[0169] Steps 1)-5) can be expressed by mathematical model as follows:

[0170]

[0171] The one-dimensional vector Y(1×M) is the recognition result of the action semantic information.

[0172] Therefore, the step 106 inputs the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into the spatial graph neural network for semantic information recognition, and obtains the recognition result of the action semantic information, which specifically includes:

[0173] The upsampled waveform data segment and its corresponding three-dimensional motion trajectory are input into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information; the mathematical model of the spatial graph neural network is Where A is the adjacency matrix represented by the upsampled three-dimensional motion trajectory, A ij is the element in the i-th row and j-th column of the adjacency matrix A; S i ,S j Respectively represent the position vectors of the i-th and j-th positions in the upsampled three-dimensional motion trajectory; Dist(S i ,S j ) represents two position vectors S i ,S j The Euclidean distance between them; DIM is the position vector S i ,S j Dimensions; Represent the position vector S i , S j The dim-th dimension of the waveform data segment after upsampling; X is the waveform data in the waveform data segment after upsampling; W(n-1) is the n-1th fully connected matrix; H (n) is the nth feature layer in the spatial graph neural network; sigmoid() represents the sigmoid function; softmax() represents the softmax function; Y is the recognition result of the action semantic information.

[0174] In actual tests, M is 62, representing 62 actions containing different semantic information (26 uppercase and lowercase letters and 0 to 9, a total of 62 gestures). After testing, the spatial graph neural network (Spatial-GNN) provided by the present invention can achieve an accuracy rate of more than 98% for the recognition of 62 gestures. If actions with very similar semantic information such as the number "0" and the letter "O" are excluded, the recognition rate of the gestures by the method of the present invention can almost reach 99.5%.

[0175] In summary, the present invention aims at the four key technical problems mentioned in the background technology and proposes a method for identifying action semantic information based on data enhancement and intelligent motion segmentation. Its main innovations and technical effects can be summarized as follows:

[0176] 1. The present invention proposes an inertial sensor data enhancement method based on deep learning and signal reconstruction, which can significantly improve the accuracy of attitude solution and trajectory restoration based on inertial sensors;

[0177] 2. The present invention proposes an intelligent motion segmentation algorithm based on inertial sensors. The algorithm does not rely on the static state during the motion process, so it can achieve high-precision segmentation and data extraction for any motion. In addition, the algorithm can intelligently determine the segmentation point according to the different problems and the needs of the task.

[0178] 3. The method of the present invention proposes a general research paradigm in the field of inertial sensor research, namely:

[0179] Data enhancement → trajectory restoration → signal waveform and trajectory morphological feature fusion → downstream tasks

[0180] The traditional research approach is:

[0181] Waveform signal features → downstream tasks

[0182] The method of the present invention introduces the attention to the trajectory morphology information, and the trajectory information is often the core of the downstream tasks. For example, in the downstream task of action semantic information recognition, the difference in semantic information mainly comes from the difference in the trajectory morphology of different actions. Therefore, the research paradigm proposed by the present invention has certain promotion value.

[0183] 4. In order to be able to fuse signal waveform information and trajectory morphology information, the present invention provides a spatial graph neural network (Spatial-GNN). This network model realizes high-precision recognition of action semantic information based on inertial sensors, and can achieve a recognition accuracy of more than 98% for 62 actions. It is currently the most accurate action recognition model based on MEMS inertial sensors, and can therefore support high-precision human-computer interaction control.

[0184] It should be noted that the four parts of the technical solution of the present invention are highly dependent on each other. For example, the data enhancement algorithm is a prerequisite for high-precision restoration of the trajectory, and high-precision trajectory information is a prerequisite for high-precision semantic information recognition. Similarly, the intelligent motion segmentation algorithm is also a prerequisite for accurately extracting action data, realizing semantic information recognition and human-computer interaction control. Therefore, the four parts constitute a complete technical solution, which together realizes high-precision action semantic information recognition based on inertial sensors.

[0185] Based on the method provided by the present invention, the present invention also provides an action semantic information recognition system based on data enhancement and intelligent motion segmentation, comprising:

[0186] A motion data acquisition module, used to acquire motion data of a six-axis inertial sensor; the six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor;

[0187] An intelligent motion segmentation module, used to segment the six-axis inertial sensor motion data by using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information;

[0188] A data enhancement module, used for performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement;

[0189] A trajectory calculation module, used for performing trajectory calculation based on the data-enhanced six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the data-enhanced six-axis inertial sensor waveform data segment;

[0190] An upsampling module is used to perform upsampling processing on the six-axis inertial sensor waveform data segment after data enhancement and its corresponding three-dimensional motion trajectory to obtain the upsampled waveform data segment and its corresponding three-dimensional motion trajectory;

[0191] The semantic information recognition module is used to input the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information.

[0192] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0193] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for identifying action semantic information based on data enhancement and intelligent motion segmentation, characterized in that: include: Acquire six-axis inertial sensor motion data; the six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor; The six-axis inertial sensor motion data is segmented by using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information; The intelligent motion segmentation algorithm based on waveform-trajectory information fusion is used to segment the six-axis inertial sensor motion data to obtain a number of six-axis inertial sensor data segments containing complete semantic information, specifically including: Performing segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, trajectory morphology information, and acceleration waveform information in a world coordinate system to obtain segmentation points detected based on trajectory morphology features; Based on deep learning and sensor original signal waveform information, segmentation point detection is performed on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features; Determining a final segmentation point based on AND gate logic according to the segmentation point detected based on the trajectory morphology feature and the segmentation point detected based on the waveform feature; Segmenting the six-axis inertial sensor motion data according to the final segmentation point to obtain a plurality of six-axis inertial sensor data segments containing complete semantic information; Performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a data-enhanced six-axis inertial sensor waveform data segment; Performing trajectory calculation based on the data-enhanced six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the data-enhanced six-axis inertial sensor waveform data segment; Performing upsampling processing on the data-enhanced six-axis inertial sensor waveform data segment and its corresponding three-dimensional motion trajectory to obtain an upsampled waveform data segment and its corresponding three-dimensional motion trajectory; The upsampled waveform data segment and its corresponding three-dimensional motion trajectory are input into a spatial graph neural network for semantic information recognition to obtain a recognition result of the action semantic information.

2. The method according to claim 1, characterized in that The segmentation point detection is performed on the six-axis inertial sensor motion data based on the Fréchet distance, the trajectory morphology information and the acceleration waveform information in the world coordinate system to obtain the segmentation points detected based on the trajectory morphology features, specifically including: De-averaging the six-axis inertial sensor motion data to obtain de-averaged six-axis motion data; Performing variance division processing on the three-axis acceleration data in the de-averaged six-axis motion data to obtain six-axis motion data after variance division; Determine a preliminary turning point position according to the three-axis acceleration data in the six-axis motion data after the variance is divided; Multiplying the six-axis motion data after the preliminary inflection point position by an attenuated gain function to obtain the six-axis motion data after gain addition; Performing trajectory calculation on the six-axis motion data after adding the gain, to obtain a distorted motion trajectory with obvious inflection points; The morphological features of the trajectory are used to set a multi-scale spline function, and segmentation points are detected on the distorted motion trajectory with obvious inflection points based on the Fréchet distance to obtain segmentation points detected based on the trajectory morphological features.

3. The method according to claim 1, characterized in that The segmentation point detection of the six-axis inertial sensor motion data based on deep learning and sensor original signal waveform information to obtain the segmentation points detected based on waveform features specifically includes: Using information captured by the optical sensor, segmentation points in the six-axis inertial sensor motion data are marked to obtain segmentation labels in the inertial sensor data; Training a 1D-CNN model based on the six-axis inertial sensor motion data and the segmentation labels to obtain a trained segmentation point detection model based on waveform features; The segmentation point detection model based on waveform features is used to perform segmentation point detection on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features.

4. The method according to claim 1, characterized in that: Before performing data enhancement processing on the six-axis inertial sensor data segment based on the signal reconstruction algorithm to generate the data enhanced six-axis inertial sensor waveform data segment, the method further includes: Acquire a plurality of six-axis raw signal samples X collected by the inertial sensor; Perform model-driven signal reconstruction on multiple six-axis original signal samples to generate multiple sets of data-wavelet pattern matching relationships {[x i ,y j ],β k }; where x i is the i-th axis signal of the six-axis original signal sample X; j is the jth IMF component of the i-th axis signal; β k for y j The corresponding optimal wavelet transform mode; According to the data-wavelet pattern matching relationship {[x i ,y j ],β k Generate a training sample for the 1DCNN model; the training sample includes the input data [X, y j ] and the ideal output data β of the 1DCNN model k ; The 1DCNN model is trained using the training samples to generate a trained data enhancement model.

5. The method according to claim 4, characterized in that The performing data enhancement processing on the six-axis inertial sensor data segment based on the signal reconstruction algorithm to generate the six-axis inertial sensor waveform data segment after data enhancement specifically includes: For each of the six-axis raw signal samples X in the six-axis inertial sensor data segment, the i-th axis signal x i Perform empirical mode decomposition to obtain the x i The jth IMF component y j ; The IMF component y j Combined with the six-axis original signal sample X, the input data [X, y j ]; The input data [X, y j ] Input the trained data enhancement model and output the IMF component y j The corresponding optimal wavelet transform mode; The IMF component y is transformed by using the optimal wavelet transform mode. j Perform wavelet transform to generate new IMF components; The six-axis original signal sample X is empirically reconstructed according to the new IMF component to generate data-enhanced six-axis inertial sensor data; a plurality of the data-enhanced six-axis inertial sensor data constitute the data-enhanced six-axis inertial sensor waveform data segment.

6. The method according to claim 1, characterized in that The step of inputting the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into a spatial graph neural network for semantic information recognition to obtain a recognition result of the action semantic information specifically includes: The upsampled waveform data segment and its corresponding three-dimensional motion trajectory are input into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information; the mathematical model of the spatial graph neural network is Where A is the adjacency matrix represented by the upsampled three-dimensional motion trajectory, A ij is the element in the i-th row and j-th column of the adjacency matrix A; S i ,S j Respectively represent the position vectors of the i-th and j-th positions in the upsampled three-dimensional motion trajectory; Dist(S i ,S j ) represents two position vectors S i ,S j The Euclidean distance between them; DIM is the position vector S i ,S j Dimensions; Represent the position vector S i , S j The dim-th dimension of the waveform data segment after upsampling; X is the waveform data in the waveform data segment after upsampling; W (n-1) is the n-1th fully connected matrix; H (n) is the nth feature layer in the spatial graph neural network; sigmoid() represents the sigmoid function; softmax() represents the softmax function; Y is the recognition result of the action semantic information.

7. An action semantic information recognition system based on data enhancement and intelligent motion segmentation, characterized in that: include: Motion data acquisition module, used to acquire motion data of six-axis inertial sensor; The six-axis inertial sensor motion data is waveform data of three-axis acceleration and three-axis angular velocity collected by the inertial sensor; An intelligent motion segmentation module, used to segment the six-axis inertial sensor motion data using an intelligent motion segmentation algorithm based on waveform-trajectory information fusion to obtain a number of six-axis inertial sensor data segments containing complete semantic information; The intelligent motion segmentation algorithm based on waveform-trajectory information fusion is used to segment the six-axis inertial sensor motion data to obtain a number of six-axis inertial sensor data segments containing complete semantic information, specifically including: Performing segmentation point detection on the six-axis inertial sensor motion data based on the Fréchet distance, trajectory morphology information, and acceleration waveform information in a world coordinate system to obtain segmentation points detected based on trajectory morphology features; Based on deep learning and sensor original signal waveform information, segmentation point detection is performed on the six-axis inertial sensor motion data to obtain segmentation points detected based on waveform features; Determining a final segmentation point based on AND gate logic according to the segmentation point detected based on the trajectory morphology feature and the segmentation point detected based on the waveform feature; Segmenting the six-axis inertial sensor motion data according to the final segmentation point to obtain a plurality of six-axis inertial sensor data segments containing complete semantic information; A data enhancement module, used for performing data enhancement processing on the six-axis inertial sensor data segment based on a signal reconstruction algorithm to generate a six-axis inertial sensor waveform data segment after data enhancement; A trajectory calculation module, used for performing trajectory calculation based on the data-enhanced six-axis inertial sensor waveform data segment to generate a three-dimensional motion trajectory corresponding to the data-enhanced six-axis inertial sensor waveform data segment; An upsampling module is used to perform upsampling processing on the six-axis inertial sensor waveform data segment after data enhancement and its corresponding three-dimensional motion trajectory to obtain the upsampled waveform data segment and its corresponding three-dimensional motion trajectory; The semantic information recognition module is used to input the upsampled waveform data segment and its corresponding three-dimensional motion trajectory into the spatial graph neural network for semantic information recognition to obtain the recognition result of the action semantic information.

Images