Human motion posture recognition system and method based on multi-feature fusion

By combining acceleration information and motion posture monitoring images and using deep learning technology to fusion of features, the problem that existing systems cannot recognize multiple motion postures is solved, and accurate recognition of human motion postures is achieved.

CN119089312AInactive Publication Date: 2024-12-06ZHEJIANG FUBAO INTELLIGENT TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411572572.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-12-25
Filing Date
2024-11-06
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing human motion posture recognition system based on three-axis acceleration sensors cannot effectively recognize multiple motion postures, such as running and jumping.

Method used

The multi-feature fusion method is adopted, combining acceleration information in human movement and real-time motion posture monitoring image information, and feature extraction and induction fusion are performed through deep learning technology to generate a motion timing feature map to identify human movement postures.

Benefits of technology

Accurate recognition of various human movement postures is achieved, the sensitivity and accuracy of the recognition system is improved, and the ability to identify movement postures such as stationary, walking, running and jumping is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089312A_ABST
    Figure CN119089312A_ABST
Patent Text Reader

Abstract

The invention discloses a human motion posture recognition system and method based on multi-feature fusion. The method comprises the following steps of: firstly, acquiring acceleration values of a human body at a plurality of preset time points in a preset time period in a movement process, then acquiring movement posture monitoring images of the human body at the plurality of preset time points, and finally acquiring a movement posture monitoring image of the human body; performing feature extraction and induction fusion on the acceleration values at the plurality of preset time points and the motion posture monitoring images at the plurality of preset time points so as to obtain a motion posture time sequence feature map guided by motion time sequence features, and finally, based on the motion posture time sequence feature map guided by the motion time sequence features, obtaining a motion posture time sequence feature map guided by the motion time sequence features. And determining the type of the human motion posture. Therefore, in combination with an image processing technology based on deep learning, recognition of the human body motion posture can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of human motion posture recognition, and more specifically, to a human motion posture recognition system and method based on multi-feature fusion. Background Art

[0002] In recent decades, researchers have been looking for a method that can objectively, accurately and reliably identify human motion postures. The acceleration sensor-based methods that have emerged in recent years have provided new ideas and practical methods for identifying human motion postures and have been widely used. More and more studies have shown that human motion acceleration information can reflect important kinematic characteristics and gait characteristics of human motion.

[0003] The triaxial accelerometer is a small, lightweight sensor that can measure the acceleration of human motion in three directions, including vertical, lateral, and front-to-back. The triaxial accelerometer is more sensitive to the motion state of the human body, and its output can better reflect the real motion state of the human body.

[0004] However, the current human motion posture recognition based on three-axis acceleration sensors still has some obvious shortcomings. For example, the existing recognition system only performs simple processing on the sensor signal, such as counting steps, but in addition to walking, human motion postures also include running, jumping, etc. However, the existing recognition system cannot recognize a variety of motion postures.

[0005] Therefore, an optimized human motion posture recognition solution is expected. Summary of the invention

[0006] In view of this, the present application proposes a human motion posture recognition system and method based on multi-feature fusion, which can comprehensively utilize the acceleration information of the human body during movement and the real-time motion posture monitoring image information, and combine it with deep learning-based image processing technology to realize the recognition of human motion posture.

[0007] According to one aspect of the present application, a method for human motion posture recognition based on multi-feature fusion is provided, which includes: obtaining acceleration values ​​of a plurality of predetermined time points within a predetermined time period during the motion of a human body; obtaining motion posture monitoring images of the human body at the plurality of predetermined time points; performing feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture timing feature graph guided by motion timing features; and determining the category of the human motion posture based on the motion posture timing feature graph guided by the motion timing features.

[0008] According to another aspect of the present application, a human motion posture recognition system based on multi-feature fusion is provided, which includes: an acceleration value acquisition module, used to obtain the acceleration values ​​of a plurality of predetermined time points within a predetermined time period during the motion of a human body; an image acquisition module, used to obtain motion posture monitoring images of the human body at the plurality of predetermined time points; a feature extraction and induced fusion module, used to perform feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture timing feature graph guided by motion timing features; and a category analysis module, used to determine the category of the human motion posture based on the motion posture timing feature graph guided by the motion timing features.

[0009] According to the embodiment of the present application, it first obtains the acceleration values ​​of a plurality of predetermined time points within a predetermined time period during the motion of the human body, then obtains the motion posture monitoring images of the human body at the plurality of predetermined time points, then performs feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture time series feature graph guided by the motion time series feature, and finally, determines the category of the human body motion posture based on the motion posture time series feature graph guided by the motion time series feature. In this way, the recognition of the human body motion posture can be realized by combining the image processing technology based on deep learning.

[0010] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 A flow chart of a method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown.

[0013] Figure 2 A schematic diagram of the architecture of a method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown.

[0014] Figure 3 A flowchart of sub-step S130 of the method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown.

[0015] Figure 4 A flowchart of sub-step S131 of the method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown.

[0016] Figure 5A flowchart of sub-step S132 of the method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown.

[0017] Figure 6 A block diagram of a human motion posture recognition system based on multi-feature fusion according to an embodiment of the present application is shown.

[0018] Figure 7 A diagram showing an application scenario of a method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0020] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0021] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0022] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0023] In response to the above technical problems, the technical concept of the present application is to comprehensively utilize the acceleration information of the human body during movement and the real-time motion posture monitoring image information, and combine it with image processing technology based on deep learning to realize the recognition of human motion posture.

[0024] Based on this, Figure 1 A flow chart of a method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application is shown. Figure 2FIG. 1 is a schematic diagram showing the architecture of a method for human motion posture recognition based on multi-feature fusion according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the method for human motion posture recognition based on multi-feature fusion according to the embodiment of the present application, the method includes the following steps: S110, obtaining the acceleration values ​​of a plurality of predetermined time points within a predetermined time period during the motion of a human body; S120, obtaining the motion posture monitoring images of the human body at the plurality of predetermined time points; S130, performing feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture timing feature graph guided by motion timing features; and, S140, determining the category of the human motion posture based on the motion posture timing feature graph guided by the motion timing features.

[0025] Specifically, in the technical solution of the present application, firstly, the acceleration values ​​of a human body at a plurality of predetermined time points within a predetermined time period during the motion process and the motion posture monitoring images at the plurality of predetermined time points are obtained. Then, the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points are subjected to data structuring processing to obtain a sequence of acceleration time series input vectors and motion posture monitoring images.

[0026] Furthermore, if Figure 3 As shown, feature extraction and induced fusion are performed on the acceleration values ​​of the multiple predetermined time points and the motion posture monitoring images of the multiple predetermined time points to obtain a motion posture timing feature graph guided by motion timing features, including: S131, data structuring processing is performed on the acceleration values ​​of the multiple predetermined time points and the motion posture monitoring images of the multiple predetermined time points to obtain a sequence of acceleration timing input vectors and motion posture monitoring images; and, S132, fusing the acceleration timing input vector and the sequence of motion posture monitoring images to obtain the motion posture timing feature graph guided by the motion timing features.

[0027] It should be understood that, in one example, the purpose of step S131 is to process and organize the original acceleration value and motion posture monitoring image for subsequent feature extraction and fusion, and the data structured processing may include time serialization of the acceleration value, sorting or aligning the image sequence, etc., so that the data can be processed and analyzed in chronological order. The purpose of step S132 is to fuse the information of the acceleration and motion posture image to generate a time series feature map, and the fusion can adopt different methods, such as splicing the acceleration and image information, weighted fusion, or learning using a neural network model. The fused time series feature map can provide more comprehensive and rich motion posture information, which is helpful for subsequent motion analysis, recognition or prediction. In summary, step S131 and step S132 are used for data preprocessing and feature fusion, respectively, to obtain a motion posture time series feature map guided by motion time series features, thereby providing more accurate and comprehensive information for subsequent motion analysis and application.

[0028] In a specific example of the present application, in step S131, as Figure 4 As shown, the encoding process of performing data structuring processing on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain the acceleration time series input vector and the sequence of motion posture monitoring images includes: S1311, arranging the acceleration values ​​at the plurality of predetermined time points into the acceleration time series input vector according to the time dimension; and, S1312, arranging the motion posture monitoring images at the plurality of predetermined time points into the sequence of motion posture monitoring images along the channel dimension. It should be understood that the purpose of step S1311 is to arrange the acceleration values ​​in time order for subsequent feature extraction and fusion. Generally, the acceleration value is a vector containing multiple time points. By arranging these vectors according to the time dimension, an acceleration time series input vector can be obtained, in which each element represents the acceleration value at a time point. The purpose of step S1312 is to arrange multiple motion posture monitoring images for subsequent feature extraction and fusion. Usually, the motion posture monitoring image is a multi-channel image, each channel represents different posture information. By arranging the images along the channel dimension, multiple images can be combined into a sequence, in which each element represents a motion posture monitoring image at a time point. In summary, step S1311 and step S1312 are used to perform data structuring processing on the acceleration value and the motion posture monitoring image, respectively, to obtain the acceleration time series input vector and the sequence of motion posture monitoring images. These processing steps are to arrange the original data according to the time dimension, to provide convenience and preparation for subsequent feature extraction and fusion.

[0029] Here, arranging the acceleration values ​​of the multiple predetermined time points into an acceleration timing input vector according to the time dimension can retain the timing information of the acceleration signal, that is, the dynamic changes during human body movement; arranging the motion posture monitoring images of the multiple predetermined time points into a sequence of motion posture monitoring images along the channel dimension can retain the timing information of the motion posture while regularizing the dimension of the data to facilitate use as input for subsequent models.

[0030] Then, the sequence of motion posture monitoring images is passed through a motion posture temporal feature extractor based on a three-dimensional convolutional neural network model to obtain a motion posture temporal feature graph. Here, considering that the movement of the human body is a dynamic process, there are motion posture temporal features in the sequence of motion posture monitoring images, that is, the motion posture of the human body presents a certain regular change in the time dimension within a certain period of time. That is, the motion posture change features contained in the sequence of motion posture monitoring images are captured by a motion posture temporal feature extractor based on a three-dimensional convolutional neural network model to distinguish the categories of different motion postures of the human body.

[0031] Furthermore, the acceleration timing input vector and the motion posture timing feature graph are passed through an induced fusion module based on MetaNet to obtain a motion posture timing feature graph guided by motion timing features. The induced fusion module based on MetaNet enables the acceleration timing input vector as a one-dimensional vector to directly interact with the visual features, i.e., the motion posture timing feature distribution expressed by the motion posture timing feature graph, directly controls the relevant characteristics of each feature channel, and helps the network focus on the motion posture timing features guided by the acceleration motion timing feature information. That is, guiding the feature distribution of the motion posture timing feature graph with the acceleration motion timing feature information as the guide can improve the feature expression of the human body motion posture.

[0032] Accordingly, in step S132, Figure 5 As shown, the acceleration timing input vector and the sequence of motion posture monitoring images are fused to obtain the motion posture timing feature graph guided by the motion timing feature, including: S1321, passing the sequence of motion posture monitoring images through a motion posture timing feature extractor based on a three-dimensional convolutional neural network model to obtain a motion posture timing feature graph; and, S1322, passing the acceleration timing input vector and the motion posture timing feature graph through an induced fusion module based on MetaNet to obtain the motion posture timing feature graph guided by the motion timing feature.

[0033] It should be understood that in the human motion posture recognition method based on multi-feature fusion, the fusion of the acceleration time series input vector and the sequence of motion posture monitoring images is achieved through two steps S1321 and S1322. The main function of step S1321 is to use the three-dimensional convolutional neural network model to extract features from the motion posture monitoring image sequence. Through the processing of convolution, activation and pooling, the motion posture time series feature extractor can capture the dynamic changes and patterns of the motion. The time series feature map extracted in this step contains the key information in the motion posture image sequence for subsequent feature fusion. The purpose of step S1322 is to fuse the acceleration time series input vector and the motion posture time series feature map by using the induced fusion module based on MetaNet. MetaNet is a network structure for learning feature fusion weights, which can dynamically adjust the weights of feature fusion according to the importance of different input features. By fusing the acceleration time series input vector and the motion posture time series feature map, the time series characteristics of the motion posture can be more comprehensively captured, and the accuracy of posture recognition can be improved. In summary, steps S1321 and S1322 are used to extract the motion posture time series feature graph and fuse multiple feature sources to obtain the motion posture time series feature graph guided by the motion time series feature. These feature graphs will be used as input classifiers for predicting the classification results of postures. Through such a feature fusion method, different feature sources can be comprehensively utilized to improve the performance and robustness of posture recognition.

[0034] Among them, the motion posture temporal feature extractor based on the three-dimensional convolutional neural network model includes an input layer, a three-dimensional convolutional layer, an activation layer, a pooling layer and an output layer. It is worth mentioning that the three-dimensional convolutional neural network (3D CNN) is a deep learning model for processing time series data. Unlike the traditional two-dimensional convolutional neural network, the three-dimensional convolutional neural network can process data with a time dimension. In step S132, the motion posture temporal feature extractor based on the three-dimensional convolutional neural network model is used to extract the temporal features of the motion posture image sequence. It includes the following layers: 1. Input layer: receiving the motion posture monitoring image sequence as input. 2. Three-dimensional convolutional layer: extracting features from the motion posture image sequence by performing convolution operations in the spatial and temporal dimensions. Unlike the two-dimensional convolutional layer, the convolution kernel of the three-dimensional convolutional layer has three dimensions, namely width, height and time. 3. Activation layer: applying activation functions, such as ReLU (Rectified Linear Unit), to introduce nonlinear transformations and enhance the expression ability of the network. 4. Pooling layer: reducing the spatial size of the feature map through downsampling operations, reducing the amount of calculation, and extracting more abstract features. 5. Output layer: Generate motion posture temporal feature graph as the output of the extractor. By using a 3D convolutional neural network model, the motion posture temporal feature extractor can learn temporal features from the motion posture image sequence and capture the dynamic changes and patterns of motion. These temporal features can better describe the evolution of motion and provide more accurate and rich information for subsequent feature fusion and analysis.

[0035] Among them, in step S1322, the acceleration timing input vector and the motion posture timing feature graph are passed through an induced fusion module based on MetaNet to obtain the motion posture timing feature graph guided by the motion timing feature, including: channel weighting the motion posture timing feature graph with the acceleration timing input vector to obtain the motion posture timing feature graph guided by the motion timing feature.

[0036] Furthermore, the motion posture temporal feature graph guided by the motion temporal feature is passed through a classifier to obtain a classification result, and the classification result is used to represent the category label of the human motion posture. The category label of the human motion posture includes: still, walking, running and jumping.

[0037] Accordingly, in step S140, based on the motion posture timing feature graph guided by the motion timing feature, the category of the human body motion posture is determined, including: passing the motion posture timing feature graph guided by the motion timing feature through a classifier to obtain a classification result, and the classification result is used to represent the category label of the human body motion posture.

[0038] More specifically, the motion posture timing feature graph guided by the motion timing feature is passed through a classifier to obtain a classification result, and the classification result is used to represent the category label of the human body motion posture, including: expanding the motion posture timing feature graph guided by the motion timing feature into a classification feature vector according to a row vector or a column vector; using the fully connected layer of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and inputting the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0039] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules, and then classify (or predict) unknown data. Logistic regression, SVM, etc. are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification, but this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.

[0040] It is worth mentioning that fully connected encoding refers to performing matrix multiplication operations on the input vector and the weight matrix, and performing nonlinear transformations through activation functions to generate encoded feature vectors. In classification tasks, fully connected encoding is often used to map the input feature vector to a higher-dimensional representation space in order to better capture the relationship and pattern between features. Each neuron in the fully connected layer is connected to all neurons in the previous layer, so more complex feature combinations can be learned. Specifically, the steps of fully connected encoding are as follows: 1. Input feature vector: Expand the motion posture temporal feature map guided by the motion timing features as row vectors or column vectors to obtain a classification feature vector. 2. Matrix multiplication: Perform matrix multiplication operations on the classification feature vector and the weight matrix to obtain a new vector. 3. Nonlinear transformation: Apply activation functions (such as ReLU) to the results of matrix multiplication, introduce nonlinear transformations, and enhance the expressiveness of the network. 4. Output encoded feature vector: After being processed by the fully connected layer, the encoded classification feature vector is obtained, which contains higher-dimensional feature representations. The role of fully connected encoding is to map the input features to a more expressive feature space by learning appropriate weight parameters. This helps to extract richer and more abstract features, thereby improving the performance of the classifier. The encoded feature vector can better represent the important features of the input data so that subsequent classification tasks can make accurate predictions and inferences.

[0041] In the technical solution of the present application, each feature matrix of the motion posture timing feature graph is used to express the image semantic features of the motion posture monitoring image, and there is a temporal correlation between the image semantic features between the feature matrices thereof. Therefore, after the acceleration timing input vector and the motion posture timing feature graph are passed through the MetaNet-based induced fusion module, the feature matrices of the motion posture timing feature graph guided by the motion timing feature are further constrained by the acceleration source timing distribution. This makes the motion posture timing feature graph guided by the motion timing feature have a dense distribution representation of the timing dimension on the channel dimension corresponding to the spatial distribution of the feature matrix of the image semantic features as a whole, and each eigenvalue also has a dense spatial distribution representation for expressing the image semantic features, which results in the need to improve the time-space dimension feature semantic logical dependency of the motion posture timing feature graph guided by the motion timing feature when the motion posture timing feature graph guided by the motion timing feature is classified and regressed through a classifier.

[0042] In a preferred example, the motion posture timing feature graph guided by the motion timing feature is input into a classifier to obtain a classification result, including: firstly, expanding the motion posture timing feature graph guided by the motion timing feature into a motion posture timing feature vector guided by the motion timing feature, and then optimizing the motion posture timing feature vector guided by the motion timing feature, specifically including: determining a first weight matrix and a second weight matrix, wherein the first weight matrix and the second weight matrix are The eigenvalues ​​are the first The eigenvalue and The square root of the correlation value and the L2 distance value of the eigenvalue: ;in, and respectively represent the first The eigenvalue and The square root of the correlation value and the L2 distance value of the eigenvalue, represents the first weight matrix Eigenvalues, represents the second weight matrix Characteristic value; respectively matrix-multiply the motion posture timing feature vector guided by the motion timing feature with the first weight matrix to obtain a first intermediate vector, matrix-multiply the motion posture timing feature vector guided by the motion timing feature with the second weight matrix to obtain a second intermediate vector, and dot-multiply the first weight matrix with the second weight matrix and then matrix-multiply the motion posture timing feature vector guided by the motion timing feature to obtain a third intermediate vector; dot-multiply the first intermediate vector, the second intermediate vector and the third intermediate vector to obtain an optimized motion posture timing feature vector guided by the motion timing feature.

[0043] For example, the motion posture temporal feature vector guided by the motion temporal feature is a row vector, then the motion posture timing feature vector guided by the optimized motion timing feature is It is expressed as: ;in, represents the motion posture temporal feature vector guided by the motion temporal feature, represents the first weight matrix, represents the second weight matrix, Indicates point addition, represents matrix multiplication, represents dot product, , and represent the first intermediate vector, the second intermediate vector and the third intermediate vector respectively, A motion posture timing feature vector guided by the optimized motion timing feature is represented.

[0044] Finally, the motion posture timing feature vector guided by the optimized motion timing feature is fully connected encoded using the fully connected layer of the classifier to obtain the motion posture timing feature vector guided by the encoded optimized motion timing feature; and the motion posture timing feature vector guided by the encoded optimized motion timing feature is input into the Softmax classification function of the classifier to obtain the classification result.

[0045] That is, by taking the motion posture timing feature vector guided by the motion timing feature as its statistical reference-free distribution response framework based on the autocorrelation matrix and the self-distance matrix, and constructing the retrieval-enhanced reverse response of the motion posture timing feature vector guided by the motion timing feature, simple repetition of feature distribution is avoided, and by ensuring the intrinsic mapping logic of the motion posture timing feature vector guided by the motion timing feature based on the retrieval-response context, its surface combination mapping is avoided, so as to achieve the semantic logic dependency mapping of the motion posture timing feature vector guided by the motion timing feature while maintaining the intuitive response correlation, thereby improving the accuracy of the classification result.

[0046] In summary, the human motion posture recognition method based on multi-feature fusion according to the embodiment of the present application can comprehensively utilize the acceleration information of the human body during movement and the real-time motion posture monitoring image information, and combine it with the image processing technology based on deep learning to realize the recognition of human motion posture.

[0047] Figure 6 FIG. 1 is a block diagram of a human motion posture recognition system 100 based on multi-feature fusion according to an embodiment of the present application. Figure 6 As shown, according to the embodiment of the present application, the human motion posture recognition system 100 based on multi-feature fusion includes: an acceleration value acquisition module 110, which is used to obtain the acceleration values ​​of a plurality of predetermined time points within a predetermined time period during the motion of the human body; an image acquisition module 120, which is used to obtain the motion posture monitoring images of the human body at the plurality of predetermined time points; a feature extraction and induced fusion module 130, which is used to perform feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture timing feature graph guided by motion timing features; and a category analysis module 140, which is used to determine the category of the human motion posture based on the motion posture timing feature graph guided by the motion timing features.

[0048] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the above-mentioned human motion posture recognition system based on multi-feature fusion 100 have been described in the above reference. Figures 1 to 5 It has been introduced in detail in the description of the human motion posture recognition method based on multi-feature fusion, and therefore, its repeated description will be omitted.

[0049] As described above, the human motion posture recognition system 100 based on multi-feature fusion according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a human motion posture recognition algorithm based on multi-feature fusion. In a possible implementation, the human motion posture recognition system 100 based on multi-feature fusion according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the human motion posture recognition system 100 based on multi-feature fusion can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the human motion posture recognition system 100 based on multi-feature fusion can also be one of the many hardware modules of the wireless terminal.

[0050] Alternatively, in another example, the human motion posture recognition system 100 based on multi-feature fusion and the wireless terminal may also be separate devices, and the human motion posture recognition system 100 based on multi-feature fusion may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0051] Figure 7 FIG. 1 shows an application scenario diagram of a method for recognizing human motion posture based on multi-feature fusion according to an embodiment of the present application. Figure 7 As shown, in this application scenario, first, the acceleration values ​​of a human body at multiple predetermined time points within a predetermined time period during the movement process are obtained (for example, Figure 7 D1 as shown in the figure), and the motion posture monitoring images of the human body at the plurality of predetermined time points (for example, Figure 7 Then, the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points are input to a server (for example, Figure 7 In S) as shown in , the server is able to use the human motion posture recognition algorithm based on multi-feature fusion to process the acceleration values ​​of the multiple predetermined time points and the motion posture monitoring images of the multiple predetermined time points to obtain a classification result of a category label for representing the human motion posture.

[0052] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0053] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for human motion posture recognition based on multi-feature fusion, characterized in that: include: Obtaining acceleration values ​​of a human body at a plurality of predetermined time points within a predetermined time period during movement; Acquire motion posture monitoring images of the human body at the plurality of predetermined time points; perform feature extraction and induced fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture timing feature graph guided by motion timing features; and determine the category of the human body's motion posture based on the motion posture timing feature graph guided by the motion timing features.

2. The method for human motion posture recognition based on multi-feature fusion according to claim 1, characterized in that: The acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points are subjected to feature extraction and induced fusion to obtain a motion posture timing feature graph guided by motion timing features, including: performing data structuring processing on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a sequence of acceleration timing input vectors and motion posture monitoring images; and fusing the acceleration timing input vectors and the sequence of motion posture monitoring images to obtain a motion posture timing feature graph guided by the motion timing features.

3. The method for human motion posture recognition based on multi-feature fusion according to claim 2 is characterized in that: The acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points are subjected to data structuring processing to obtain an acceleration timing input vector and a sequence of motion posture monitoring images, including: arranging the acceleration values ​​at the plurality of predetermined time points into the acceleration timing input vector according to a time dimension; and arranging the motion posture monitoring images at the plurality of predetermined time points into a sequence of motion posture monitoring images along a channel dimension.

4. The method for human motion posture recognition based on multi-feature fusion according to claim 3 is characterized in that: The acceleration timing input vector and the sequence of motion posture monitoring images are fused to obtain the motion posture timing feature graph guided by the motion timing feature, including: passing the sequence of motion posture monitoring images through a motion posture timing feature extractor based on a three-dimensional convolutional neural network model to obtain the motion posture timing feature graph; and passing the acceleration timing input vector and the motion posture timing feature graph through an induced fusion module based on MetaNet to obtain the motion posture timing feature graph guided by the motion timing feature.

5. The method for human motion posture recognition based on multi-feature fusion according to claim 4 is characterized in that: The motion posture temporal feature extractor based on the three-dimensional convolutional neural network model includes an input layer, a three-dimensional convolutional layer, an activation layer, a pooling layer and an output layer.

6. The method for human motion posture recognition based on multi-feature fusion according to claim 5, characterized in that: The acceleration timing input vector and the motion posture timing feature graph are passed through an induced fusion module based on MetaNet to obtain the motion posture timing feature graph guided by the motion timing feature, including: channel weighting the motion posture timing feature graph with the acceleration timing input vector to obtain the motion posture timing feature graph guided by the motion timing feature.

7. The method for human motion posture recognition based on multi-feature fusion according to claim 6, characterized in that: Based on the motion posture time series feature graph guided by the motion time series feature, determining the category of human motion posture includes: passing the motion posture time series feature graph guided by the motion time series feature through a classifier to obtain a classification result, wherein the classification result is used to represent the category label of the human motion posture.

8. The method for human motion posture recognition based on multi-feature fusion according to claim 7, characterized in that: The category labels of the human body motion postures include: still, walking, running and jumping.

9. A human motion posture recognition system based on multi-feature fusion, characterized in that: include: An acceleration value acquisition module is used to acquire acceleration values ​​of a human body at a plurality of predetermined time points within a predetermined time period during movement; An image acquisition module is used to obtain the motion posture monitoring images of the human body at the plurality of predetermined time points; a feature extraction and induction fusion module is used to perform feature extraction and induction fusion on the acceleration values ​​at the plurality of predetermined time points and the motion posture monitoring images at the plurality of predetermined time points to obtain a motion posture time series feature graph guided by motion time series features; And a category analysis module is used to determine the category of human body motion posture based on the motion posture time series feature graph guided by the motion time series feature.

Citation Information

Patent Citations

  • Multi-information fusion human body behavior detection method and system

    CN112434669A

  • Human body posture recognition method

    CN116186517A

  • Human body action recognition method and device based on multi-modal fusion

    CN116758628A

  • Robust 2D human body posture estimation method and device based on IMU and RGB images

    CN116912934A

  • Gait-based identity recognition method and device, chip and terminal

    CN116959035A